Ampersand v0.1 Model Card

Sycophancy Evaluation Awareness

A model that recognizes it is in an evaluation may behave differently from how it would with real groups of people. We quantify and mitigate this in our multi-turn sycophancy evaluation, where a fictional user seeking reassurance engages with the assistant over four exchanges. GPT-6 Luna judges whether each assistant response had an opportunity for sycophancy and whether it was sycophantic. This measure uses brief conversations as a proxy for sycophancy: it captures responses that favor immediate user approval over honest assessment, but does not measure the longer term consequences of what the assistant says. Assessing whether a response serves the user’s interests over time would require observing how their decisions and circumstances evolve beyond the conversation. Following prior work, after the interaction we ask the assistant for an evaluation probability and a deployment probability on separate branches of the same history. We combine the answers as the self-reported P(eval) = ½ [evaluation answer + 100 − deployment answer].

Our control is 300 conversation histories from real people from ThoughtTrace. We retain their original four exchanges, with the limitation that the assistant turns were written by other models rather than the model being evaluated. We measure evaluation awareness as the gap between mean P(eval) on our sycophancy evaluation and on these deployment conversations. On the widely used SycophancyEval benchmark, GPT-6 Astra’s mean P(eval) is 41.6 absolute percentage points higher than on the ThoughtTrace control.

We first used prompted frontier models to play the fictional user described above, giving each model the user’s profile and asking it to respond to the assistant throughout the conversation. We compared seven user models: GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5.5, Gemini 3.8 Flash, GLM 5.3, and Muse Spark 1.3. These simulations produced little or no positive gap between the evaluation and real human conversations for several assistants: with Opus 5.5 playing the user, this was the case for Ampersand, GLM 5.3, Gemini 3.8 Flash, and Opus itself. Astra and Sol, however, showed persistently large gaps across the prompted user models. Opus was their strongest prompted simulator, so we use it as the baseline below. Even with Opus, the gap remains 37.6 percentage points for Astra Medium and 41.1 points for Sol. Interestingly, the choice of user simulator also affects measured sycophancy: Assistants tend to show more sycophancy with simulators whose conversations they rate as more likely to be deployment. We observe this pattern across all seven assistants in the initial sweep.

We next use Persimmon, our large-scale model designed to realistically simulate how people behave in multi-turn chat settings. To use more test-time compute with Persimmon, we generate N candidate messages at each user turn and have GPT-6.1 Sol select the message that sounds most natural as the next user turn in the conversation. To Sol, Astra, and Argon, Persimmon conversations look more like a real conversation with a user than Opus conversations do, reducing their reported probability of being in an evaluation. This is consistent with our Multi-User Turing Test results, where Persimmon’s generated conversations were rated as human in 19.8% of comparisons versus less than 1% for all frontier models tested. Best-of-128 selection further raises mean P(deployment) from 40.3% to 67.0% for Sol and from 39.9% to 82.4% for Argon, and from 40.1% to 64.5% for Astra.

As a robustness check, we varied the candidate budget from 1 to 128 and tried a different selector. With GPT-6.1 Sol as the assistant, conversations generally look more like a real conversation with a user as the number of candidates increases. Switching the selector from Sol to Astra at N=128 gives a similar result.

Our reference evaluation uses Persimmon without selection. This is the version used for the sycophancy results reported above.