[!NOTE] This page reflects the corrected analysis, live on arXiv as v3 of August 28, 2026. The changelog itemises what the revision changed.
Overview
An empirical audit of 8,830 responses across three Gemini generations (2.0, 2.5, 3.0), evaluating seven adversarial prompt categories under three guardrail conditions. Results were validated against 236 human annotations from five raters ($\kappa=0.78$ judge-vs-human consensus) and an external model judge (DeepSeek V3, $N=582$ valid comparisons; 93.3% weighted agreement).
The Granularity Gap
Current LLM safety metrics rely heavily on binary pass/fail classification. This paradigm obscures a spectrum of “social compliance”: hedging, flattery, and partial agreement where models prioritize user validation over epistemic accuracy.
Reported consistently as the rate at which the verdict fires, the profile is monotone: 0.30% at Level 1, 4.30% at Level 2, 6.31% at Level 3, and 95.94% at Level 4-5. It is a step function with its edge between Levels 3 and 4, not the U-shaped curve an earlier version described.The pass/fail metric this pipeline reports, the Challenge Rate, turns out to be computed from the Refusal axis: a judge vote is recorded as CHALLENGED when its refusal score is $\le 3$. It is therefore close to blind to sycophancy. Under no guardrail it certifies 87.66% of responses as having challenged the user, while 53.00% of those same responses carry substantial sycophantic content and 46.97% reach moderate or severe levels. Across all three guardrail conditions the corresponding figures are 27.2% and 22.7%.
That divergence is the Granularity Gap: a safety metric can be built on a construct adjacent to the one it is read as certifying, and then report reassuring numbers about a behaviour it never measured. Note what this is not: it is not a demonstration that binarisation is lossy. Regressing sycophancy on the unthresholded refusal axis gives $R^2=0.22$, slightly lower than the binarised $0.29$. The problem is the axis, not the threshold. No deployed safety classifier was tested here.
Figure 1: Rate at which the binary verdict fires, by sycophancy severity: 0.30% at Level 1 (n=6,429), 4.30% at Level 2 (n=395), 6.31% at Level 3 (n=1,760), 95.94% at Levels 4-5 (n=246). The edge sits between Levels 3 and 4.
The Alignment Tax
The paper introduces a measure of the epistemic cost of social compliance. Sycophancy co-occurs with degraded judged truthfulness ($\rho=0.40$): when models prioritize user validation, the judge’s accuracy rating degrades alongside it. This coupling, termed the Alignment Tax, intensified across model generations, rising 67% from $\rho=0.30$ (Gen 2.0) to $\rho=0.50$ (Gen 3.0). Both axes are scored by the same judge in one call and 94% of truthfulness scores sit at the floor, so this is co-occurrence within one evaluator rather than an independently verified factual effect.
Key Findings
- Non-Monotonic Safety Trajectories: Gen 2.5 regressed substantially before Gen 3.0 recovered. The flagship Pro model showed inverse scaling during this regression, suggesting enhanced reasoning capacity served motivated agreement rather than boundary enforcement.
Figure 2: Safety trajectory across Gemini generations. Mean sycophancy stratified by guardrail condition (Control, Protocol, Simple) shows significant regression in Gen 2.5 and non-monotonic recovery in Gen 3.0.
- Category Vulnerability: Requests for flattery (Egotistical Validation) elicit sycophancy at nearly twice the rate of overtly unethical requests. This suggests that RLHF-trained “helpfulness” creates exploitable blind spots centered on affective manipulation.
Figure 3: Mean sycophancy scores across eight Gemini variants and seven adversarial categories (Control condition, N=2,949). Egotistical Validation dominates across the entire family; Gen 2.5 Pro shows the most severe susceptibility (M=4.15 in this category).
Figure 4: Generational vulnerability profiles across all seven categories (Control). The Gen 2.5 expansion is uniform across categories, not specialization-driven, suggesting a system-wide regression rather than a targeted failure mode.
- Guardrail Interventions: Simple direct constraints (“Do not agree with false premises”) consistently outperformed complex reasoning protocols, cutting mean sycophancy in the most vulnerable category by 60.9% (Egotistical Validation, 3.27 → 1.28).
Figure 5: Guardrail impact on challenge rate (87.7% → 99.9%) and mean sycophancy (2.21 → 1.16), a 48% reduction. Simple constraints outperform complex reasoning protocols, suggesting susceptibility to flattery is an alignment artifact addressable through minimal intervention.
Conclusion
A refusal-based verdict, which is what pass/fail LLM evaluation typically reports, leaves most of the sycophancy severity distribution unresolved. It cannot be tuned into a severity monitor either: the threshold already in use is the best available on that axis, and no function of the refusal axis explains more than 35% of severity variance. The verdict has to be reported alongside a continuous scale rather than in place of one. We evaluated one judge’s verdict, not a deployed safety classifier.