How to evaluate a synthetic research vendor: eight validity tests

Headline accuracy numbers mislead because they leave four things open: what was measured, against which people, on which questions, and who ran the test. Edoardo Chidichimo and Felix Wallis start from that gap in Artificial Societies’ validity framework, Distribution Matching Is a Bad Way to Evaluate Simulations, published on 9 September 2026. In the framework’s reading, the usual answer to “accurate at what?” is distributional fit: how far simulated and human answer distributions sit apart, question by question, measured with total variation distance (TVD), Jensen–Shannon divergence (JSD) or Spearman’s rank correlation.

Distributional fit is easy to satisfy without modelling anyone. The framework’s first figure sets a human audience beside a synthetic one with identical answer distributions for “I intend to buy this” and “It is worth the price”, so every per-question metric scores zero distance. Their joint distributions, which record who gave which combination of answers, differ by a TVD of 0.4: 40% of one audience would have to change its combination of answers to match the other. Among the humans one attitude drives both answers, and among the synthetic respondents the two are unrelated.

Bisbee and colleagues found the same gap in real data (Political Analysis, May 2024 (opens in a new tab)). Their ChatGPT personas matched the averages of the American National Election Studies (ANES), with less variation than real respondents. Yet 48% of the regression coefficients, which measure how a trait such as age or party relates to an answer, differed significantly from the human estimates. A vendor whose evidence stops at matched averages has shown you the part of the result that Bisbee’s personas got right.

Synthetic research borrows three kinds of validity from social science, and each asks a separate question. Internal validity asks whether the stimulus caused the response shift, rather than noise, a quirk of the model or sycophancy, a language model’s habit of siding with whatever the prompt appears to want. Construct validity asks whether the questions measure the trait they are meant to measure, such as economic ideology or purchase intent. External validity asks whether the results hold for real populations, settings and times.

Each failure has a recognisable form in the framework. Without internal validity, a message can seem to persuade personas that were only being agreeable. Without construct validity, a synthetic audience rates a new product high on quality, value, trust and recommendation at once, a halo that real consumer data does not show. Without external validity, a coherent population can still be the wrong one, which is why the framework asks for effect sizes from human experiments.

Artificial Societies’ September 2026 framework groups the eight tests below and marks repeated questions and distributional fit as the current industry standard. The last column is the evidence to request from any vendor, including us.

Source: Chidichimo and Wallis, Distribution Matching Is a Bad Way to Evaluate Simulations, Artificial Societies, September 2026. In the framework’s figure, tests 6 to 8 are the ones that require human study data.

For message testing, the perturbation test matters most. Language model personas agree with confident framing more readily than people do, so a message can appear to persuade a synthetic audience that has only been agreeable. The framework (Chidichimo and Wallis, September 2026) therefore asks for checks in both directions: cosmetic changes should leave results where they are, while changing the messenger or removing the evidence behind a claim should move them.

Held-out data is human data a model could not have seen during training, so agreement with it tests the model rather than its memory. The framework sets a strict bar: major public survey programmes sit in language model training data, so only unpublished or newly fielded studies qualify. A match against ANES, the General Social Survey (GSS) or Pew may show that the model has read the published results. It says less about how the model will handle your question.

The American Association for Public Opinion Research (AAPOR) takes a compatible position in its May 2026 task force report, Responsible AI Integration in Survey Research (opens in a new tab). It suggests checking synthetic responses for leakage or memorisation, such as unusually specific strings reproduced from training sources, and recommends keeping at least some human respondents in a study so that simulated and human answers can be compared.

Timing gives the cleanest held-out design, because a prediction logged before an event cannot draw on an outcome that did not yet exist. Artificial Societies used it in its September 2026 earnings call study, preregistering the questions its simulated sell-side analysts would ask at the NVIDIA, Marvell and Credo calls on the Open Science Framework (OSF), a time-stamped research registry, before each call. Ask any vendor for the fielding date of every benchmark and the training cut-off of the model that produced the answers.

Artificial Societies’ evidence sits in three of the sources this article draws on. The January 2026 report predates the framework, so the mapping below is ours.

Source: Artificial Societies, method and evaluation page, misinformation study and earnings call study, read 28 September 2026.

Artificial Societies hashed and time-stamped the misinformation study’s design before the first survey call, which rules out tuning the design after seeing results. It cannot rule out a model having read the published papers, which is why the table does not count the replication as held out. Artificial Societies’ framework states the commitment as “We hold ourselves to all eight.” As of 28 September 2026, four of the eight have no published Artificial Societies result: repeated questions, perturbed questions, trait separation and structure match. Ask us for that evidence as you would any other vendor.

ESOMAR already asks suppliers how they measure and assess validity in its 20 Questions to Help Buyers of AI-Based Services for Market Research and Insights (opens in a new tab), issued in March 2024. For tools that use synthetic data, it also asks what validation compares synthetic outputs with primary research or real-world results. The questions below apply that request to the eight tests:

1. 1.

   What exactly does your accuracy figure measure, against which human data, on which questions, and who ran the evaluation?

2. 2.

   Which of your benchmark studies were unpublished, or fielded after the model’s training cut-off?

3. 3.

   Can you show joint distributions or subgroup cross-tabulations as well as per-question totals?

4. 4.

   What happens to results when answer options are reordered, the scale is reversed or the messenger changes?

5. 5.

   Have you reproduced the effect size of a human experiment, and where does your estimate fall against the human confidence interval?

6. 6.

   Have you logged any prediction before the outcome was known, and where is the time-stamped record?

7. 7.

   How many human respondents does a study include, and how are synthetic responses labelled in the deliverable?

8. 8.

   When the underlying model changes, which validations do you repeat?

Questions 7 and 8 follow AAPOR’s May 2026 advice on disclosure and on repeating validation when a model changes. Our guide to AI market research tools sets out what named vendors publish about their methods and evidence, and our explainer on silicon sampling covers the academic studies behind the method.

Validity tests leave two problems open. Human data, the standard they grade against, is itself unstable: answers shift with wording, timing and the audience a respondent imagines, and stated intentions predict behaviour weakly, the intention–behaviour gap every survey inherits. Subgroups are the other gap. A preprint by Florian Kutzner and colleagues, including Artificial Societies researchers, posted to arXiv on 23 September 2026, argues that ad hoc comparisons with human surveys test the wrong thing when decision makers need to anticipate consequential behaviour. It asks for validity claims by subgroup, since aggregate accuracy can hide the groups a decision affects most. A simulation that passes all eight tests is still graded against self-reports, which makes it, in the framework’s phrase, an imitation of an imitation.