[!NOTE] Working Paper The findings below come from the pilot phase of an ongoing four-phase audit. The verified paper and full pipeline are public at telling-more-than-they-can-know. Download the paper PDF
Overview
A pilot study analyzing the dissonance between LLM mechanistic interpretability and user-facing rationale generation in high-stakes deployment environments.
This research evaluates whether four production-grade language models (GPT-4o-mini, Gemini-3-Flash, DeepSeek-V3, GLM-4.7) reproduce demographic bias when scoring standardized psychological instruments (such as the PHQ-8), and critically, whether their post-hoc explanations accurately disclose the variables that actually drove those scores.
Pilot Study Findings
Identical clinical vignettes were instantiated across 18 intersectional demographic cohorts (crossed by race, gender, and socioeconomic status). The results demonstrated massive demographic bias in scoring: PHQ-8 scores showed a Low-minus-High SES gap of 5.78 points (Cohen’s $d = 1.95$).
Variance decomposition confirmed that Socioeconomic Status (SES) accounted for 48.75% of the total variance in the generated scores, while the choice of the generator model itself accounted for only 2.22%.
Figure 1: PHQ-8 group means by SES, gender, and race. The SES gap is the dominant driver of scoring variance.
The Failure of Post-Hoc Explanations
Despite SES being the empirical driver of the scores, the models’ explanations obscured it. Under a feature-attribution explanation format, the models claimed that “clinical presentation” accounted for 63.4% of their score while assigning SES a mere 17.2% of the weight.
Figure 2: ANOVA variance partitioning of PHQ-8 score variance, isolating the contribution of each prompt factor.
An LLM-as-judge protocol (three-vote consensus, $n=1,072$ evaluations) revealed a name-but-fairwash pattern: in 31.6% of cases the explanations explicitly named the demographic features involved yet still rated as fairwashing, rationalizing the biased score through clinical-sounding language. The pilot indicates that current post-hoc explanation formats fail to surface demographic bias, maintaining an “explanation gap” between what drives a model’s output and what it reports driving it.