[!NOTE] Preprint in Preparation The manuscript, supplement, code and a receipt for every number in the paper are in the repository, archived on Zenodo at 10.5281/zenodo.23147356.
Overview
Researchers now prompt large language models with demographic personas and use the answers in place of survey respondents. These simulated samples are usually checked by reading individual answers. The validity ladder replaces that check with a protocol: a sample supports only the uses whose matching checks it has passed.
The ladder
The protocol has a gate and four rungs. Each has a statistic, a pass rule, and a use that a pass supports.
- Gate. How many repeated draws make a persona’s score dependable.
- Level 1. Whether a single draw is an answer pattern a real person could give.
- Level 2. Whether gaps between groups match the population’s gaps.
- Level 3. Whether the overall level matches the population’s.
- Level 4. Whether the items hang together as they do in people.
A certifiability step runs before any answers are generated. It uses the human data alone to show which group differences a study could confirm.
Results
- Controls. Real survey respondents run through the same checks rarely fail, and errors planted at known sizes are caught by the check built for each.
- Ten external releases. The ladder was applied to ten datasets released by other authors, including silicon samples and digital twins. Samples drawn from each dataset’s own human answers rarely fail where the human data are precise. None of 725 readings of published simulators in nine of them reproduces the group differences, and for over half the human data were too thin to confirm any simulator.
- Worked example. In 28,800 PHQ-8 depression assessments from four models, read against the National Health and Nutrition Examination Survey, persona averages stabilize after a few answers, and every model fails every check above that. These are the same 28,800 assessments analysed in Plausible Patients, Impossible Populations.
The ladder is released as a Python package with a harness that scores any simulator release against the same rules.