What It Takes To Trust Artificial Societies

Suppose a competitor tells you their synthetic audience is 98% accurate. Accurate at what? Against which humans, on which questions, measured how? Currently, there is no published methodology for evaluating synthetic research architectures. For a technology informing the world’s most consequential decisions, this isn’t good enough. At Artificial Societies, we simulate audiences of high-value decision makers so enterprise customers can test strategy and messaging before committing to it in the real world. The stakes of their consequential decisions are necessarily the stakes of our evaluation.

Large language models (LLMs) write conspicuously plausible text. A persona enacted by an LLM can sound exactly like the person it is meant to represent while holding no stable beliefs at all, and a synthetic population can reproduce a survey’s headline distribution while the relationships between its respondents’ answers remain incoherent. Since consequential decisions have little margin for error, we’ve become obsessed with evaluating these synthetic samples. How do we ensure consistent outputs when working with stochasticInvolving randomness. LLMs are stochastic: the same prompt can produce different outputs on different runs, which is why consistency has to be measured rather than assumed. models? Or that persona responses reflect human belief systems? And when we test a message, does the observed effect reflect a real mechanism of human behaviour that will generalise to new settings and unseen populations?

Given their exceptional language understanding and fluency, LLMs can capture patterns of cultural knowledge, linguistic idiosyncrasies, common-sense reasoning, and even human behaviour (Liu et al., 2022; Park et al., 2022, 2023; Horton, 2023; Ashokkumar et al., 2026). With their similarly impressive capabilities in classifying and generating text, industry and academia have used these models to generate both survey (Argyle et al., 2023) and behavioural responses (Aher et al., 2022). Prompt engineering has enabled LLMs to adopt personas (with specified characteristics such as age, gender, ethnicity, and political orientation) and generate representative response distributions that capture the complex relationship between human subpopulations’ beliefs, attitudes, and cultural influences (Argyle et al., 2023). Our work shows that this performance extends to large populations of LLM agents that spontaneously reproduce homophily, a core feature of human community formation (He et al., 2024). In any case, synthetic surveys have become an increasingly popular way to learn about society and interrogate the hardest-to-reach populations that previously would have been impossible to research.^{1 (footnote)} The field, as we see it, has now reached an evaluation bottleneck.

Beyond Superficial Benchmarks

Current industry standards for evaluating synthetic research are underwhelming. Naive metrics that measure how well simulated response distributions match held-outHuman data the model has never been exposed to, so that agreement with it counts as a real test. Because major public survey programmes appear in LLM training data, only unpublished or newly fielded studies are truly held out. distributions of real human responses (i.e., Total Variation DistanceA measure of how far apart two distributions are, from 0 (identical) to 1 (no overlap). A TVD of 0.1 means you would need to move 10% of respondents between answer categories to turn one distribution into the other., Jensen-Shannon divergenceAnother measure of the distance between two distributions, also bounded between 0 and 1. It weights disagreements differently from TVD but answers the same question: how similar are these two sets of answers?, Spearman’s ρA correlation measure for rankings. If you order a question’s answer options from most to least popular for both humans and the simulation, Spearman’s ρ tells you how well the two orderings agree. on rankings of a survey question’s options) are reported as proof of validity. In fact, these metrics are easily satisfied when a simulation reproduces the (correct) marginal distributionHow answers to a single question are spread across the response options (e.g., 20% strongly agree, 35% agree…). Matching marginals means getting each question’s headline result right, considered on its own. of every question in a survey, even when the relationship between them is instead incorrect. In other words, a simulation can reproduce how many people gave each answer without reproducing who gave which combination of them. Marginal fit is necessary but, alone, it provides no information about whether the population’s underlying response structure has been captured (Figure 1).

A more demanding test, and one we report ourselves, is producing a simulation that is as accurate as human self-replicationThe rate at which a respondent gives the same answer when asked the same question twice. Humans reproduce their own answers roughly 91% of the time; a simulation should land near that figure, not above it.. When humans answer the same question twice, they only reproduce the same answer roughly 91% of the time (Park et al., 2026). Our own architecture reproduces human answers 86% of the time across 1,000 diverse surveys, achieving up to 95% of the human ceiling. That said, we shouldn't aim to reproduce human answers with perfect consistency. People are reliably inconsistent, with their response distributions shifting when questions are reworded. Our goal is therefore to produce simulation outputs with human-calibrated consistency (Figure 2).

Three Kinds of Validity

To move beyond superficial survey matching and assess how well a system models human behaviour, we engage social science and psychology. These disciplines proffer empirically validated theories about how groups of people behave, form worldviews, and are influenced by those around them, while equipping us with a repertoire of psychometric tools to measure various underlying (or “latentAn underlying trait that cannot be observed directly but shapes many observable answers, such as political ideology, risk tolerance, or trust in institutions. Survey questions are attempts to measure these indirectly.”) traits and assess their observable implications. We ground our evaluation architecture in three foundational tenets of social science (Campbell, 1957; Cronbach & Meehl, 1955): internal validityWhether an observed effect was genuinely caused by the thing being tested, rather than by noise, artefacts, or quirks of the system. (did the stimulus cause the response shift?), construct validityWhether the questions asked actually measure the trait they are meant to measure. (do the questions measure the trait we think they measure?), and external validityWhether findings generalise beyond the specific test to real populations, settings, and time periods. (do the results hold for real populations?). Figure 3 lays out the eight tests that fall under them.

Internal Validity

In classical behavioural science, internal validityWhether an observed effect was genuinely caused by the thing being tested, rather than by noise, artefacts, or quirks of the system. asks whether an observed effect is genuinely caused by the stimulus rather than system noise or artefacts. In other words, in a simulation, a shift in persona opinion must be strictly attributable to the tested stimulus, not to model hallucination or stochasticInvolving randomness. LLMs are stochastic: the same prompt can produce different outputs on different runs, which is why consistency has to be measured rather than assumed. noise. Take sycophancyThe tendency of language models to agree with whatever a prompt implies. Humans do this too, but LLM personas do it far more, so a confidently framed message can shift responses without any real persuasion taking place.: humans are prone to agreeing with whatever a question might imply, but LLM personas do so far more often. In simulation, a confidently framed stimulus can shift responses because the persona is being agreeable (an artefact of its training) rather than because it was persuaded.

We test for internal validityWhether an observed effect was genuinely caused by the thing being tested, rather than by noise, artefacts, or quirks of the system. at an individual level by checking that our personas’ responses are anchored to a stable belief system. To mimic humans, these responses should be generated by a small set of unobserved (“latentAn underlying trait that cannot be observed directly but shapes many observable answers, such as political ideology, risk tolerance, or trust in institutions. Survey questions are attempts to measure these indirectly.”) constructs, such as political ideology, risk tolerance, and institutional trust. Personas should hold these traits and conduct themselves accordingly. To test for this, we ask semantically equivalent versions of the same question, and measure agreement with Cohen’s kappaA measure of agreement between two sets of answers that corrects for agreement you would expect by chance. Used here to check whether a persona gives the same answer to two differently worded versions of the same question.. We also define sets of conceptually linked questions with valid response chains and measure the rate at which a persona violates them. Both metrics are scored against established human test–retestThe rate at which a respondent gives the same answer when asked the same question twice. Humans reproduce their own answers roughly 91% of the time; a simulation should land near that figure, not above it. bands, so a persona whose answers do not shift under reframing fails the same test as one whose answers move at random.

The same logic extends to the survey instrument itself. A simulation architecture is coherent and robust if it produces the same results when we change meaningless features of a survey. For example, reordering response options should not shift a distribution; reversing a Likert scale should only reverse the answer; and changing question order should not affect a marginalHow answers to a single question are spread across the response options (e.g., 20% strongly agree, 35% agree…). Matching marginals means getting each question’s headline result right, considered on its own.. Conversely, increasing the magnitude of a claim (a stimulus promising a 5% vs 25% saving) should shift responses further and monotonically across that range. Likewise, changing the messenger should change the reception, and the removal of substantiating evidence from a narrative should weaken scores. Thus, evaluation must strike a balance between a volatile simulation that reports significant differences in response to menial changes, and an inert one that stays the same even when we perturbA deliberate change to a survey to test robustness. Cosmetic perturbations (reordering options, reversing a scale) should leave results unchanged; substantive perturbations (changing a claim from 5% to 25%) should move them predictably. the elements that matter.

Construct Validity

As the science of measuring latent traitsAn underlying trait that cannot be observed directly but shapes many observable answers, such as political ideology, risk tolerance, or trust in institutions. Survey questions are attempts to measure these indirectly. and attitudes, psychometrics powers construct validityWhether the questions asked actually measure the trait they are meant to measure.. In human data, latent constructs leave a signature in the covarianceThe tendency of two measures to move together. If people who rate a product as overpriced also tend to report weak renewal intent, those two items covary. Human beliefs leave a characteristic covariance signature that a simulation should reproduce. among survey questions. The logic here is that related survey questions should load on the same latent traits, generating correlations in humans’ responses to them (Cronbach & Meehl, 1955). To ensure that our personas exhibit the same correlations, we administer sets of items known from human studies to measure a single construct (e.g., items on redistribution, regulation, and public spending that together measure economic ideology), and then assess their covariance with Cronbach’s alpha and McDonald’s omegaMeasures of internal consistency: how strongly a set of questions designed to measure the same construct actually hang together. Human instruments typically score between 0.7 and 0.9. Scores near 1.0 are a warning sign, indicating respondents are repeating themselves rather than reasoning.. Human instruments typically land between 0.7 and 0.9, and we hold our simulations to that same band.

Psychometrics also helps us distinguish convergent from discriminant validityMeasures of different traits should be distinguishable from one another (purchase intent and brand recall should be related, but not identical). Synthetic audiences often pass convergent tests and fail discriminant ones, rating everything about a product uniformly high. (Campbell & Fiske, 1959). Convergent validityDifferent measures of the same trait should agree with one another (stated purchase intent should track willingness to pay). requires that measures of the same underlying trait correlate strongly (e.g., stated purchase intent should track willingness to pay for the same product), whilst discriminant validity requires that measures of distinct traits correlate more weakly. For example, purchase intent and unaided brand recall should be related, but far less tightly than two measures of intent.

Synthetic populations frequently satisfy convergent validityDifferent measures of the same trait should agree with one another (stated purchase intent should track willingness to pay)., but often fail to discriminate: a bad LLM persona shown a new product will rate it high on quality, value, trustworthiness, and likelihood of recommending it, even though these judgements come apart in real consumer data (Lukauskas & Šarkauskaitė, 2026). Together, the internal and construct validityWhether the questions asked actually measure the trait they are meant to measure. tests establish what we call multi-level coherence: personas must maintain logically stable belief systems across paraphrased questions (persona level) and exhibit authentic latentAn underlying trait that cannot be observed directly but shapes many observable answers, such as political ideology, risk tolerance, or trust in institutions. Survey questions are attempts to measure these indirectly. psychometric covarianceThe tendency of two measures to move together. If people who rate a product as overpriced also tend to report weak renewal intent, those two items covary. Human beliefs leave a characteristic covariance signature that a simulation should reproduce. across the population (audience level).

External Validity

External validityWhether findings generalise beyond the specific test to real populations, settings, and time periods. asks whether findings generalise across populations, settings, and timeframes (Campbell, 1957). A simulation is externally valid if its outputs accurately predict human responses beyond a single isolated prompt. At its most basic, this means matching real-world response distributions across granular population sub-samples (Argyle et al., 2023; Santurkar et al., 2023). Beyond that, it means demonstrating that when we test a stimulus (e.g., a corporate narrative), the effect sizeThe magnitude of a change caused by a stimulus, not just whether a change occurred. A simulation with external validity should reproduce not only the direction but the size of effects observed in human experiments. produced by the synthetic audience matches the effect size produced in real-world human experiments (Ashokkumar et al., 2026; Aher et al., 2022). Replicating treatment effects provides evidence that the simulation has a causal understanding of the target population, increasing the chance that it will reliably generalise to unseen settings.

A third external test is structure match. Here, exploratory factor analysisA statistical method that looks at the pattern of covariance among many questions and identifies a smaller number of underlying factors that explain it. Used to check whether the simulation’s hidden structure resembles the structure found in human data. can help decompose the shared variance among items into unlabelled factors, allowing us to model each response as a linear combination of those factors plus a unique error term (Spearman, 1904; Fabrigar et al., 1999). Two outputs should follow. First, the recovered factors should correspond to constructs we can name (Messick, 1995): a factor loadingHow strongly a given question relates to a given underlying factor. If a persona’s age predicts how it loads on a factor capturing support for state intervention, the simulation is reproducing a relationship found in real populations. on nationalisation, tax-funded services, and business regulation is recognisably “support for state intervention”. Second, persona attributes — age, gender, occupation, etc. — should predict scores on those factors as they do in human data (Bisbee et al., 2024). In this way, convincing marginal distributionsHow answers to a single question are spread across the response options (e.g., 20% strongly agree, 35% agree…). Matching marginals means getting each question’s headline result right, considered on its own. with no recoverable factor structure indicate a hollow simulation that’s unlikely to generalise. In other words, a simulation architecture with no meaningful response structure will not generalise to truly held-outHuman data the model has never been exposed to, so that agreement with it counts as a real test. Because major public survey programmes appear in LLM training data, only unpublished or newly fielded studies are truly held out. samples or unseen questions, no matter its distributional accuracy against ANES, GSS, and Pew (ibid.).

An Imitation of an Imitation

A harder problem brews beneath all this. Though we argue that matching ground truth (viz. human-answered surveys) is important, whether by external criteria (matching distributions and effect sizesThe magnitude of a change caused by a stimulus, not just whether a change occurred. A simulation with external validity should reproduce not only the direction but the size of effects observed in human experiments.) or internal ones (coherence and psychometric structure), human survey responses are themselves messy and inherently unstable. People answer differently depending on how you ask, when you ask, and who they think is listening. And, of course, what people say is often a weak predictor of what they will do, the age-old intention–behaviour gapThe well-documented difference between what people say they will do and what they actually do. Every survey inherits it; synthetic research inherits it, too. that is inevitably inherited in synthetic research. Evaluating a synthetic population purely against human self-reports therefore means building an imitation of an imitation, and grading it against that very imitation.

To be clear, this is not an argument against synthetic research. Every survey ever fielded has run on the same proxy, and market research has spent decades managing the gap between stated and revealed preference. However, the critical benefit for social simulations is their capacity for interrogation in ways that are impossible for human respondents. We can gauge whether the persona’s answers come from a stable belief system, whether that structure matches the structure in human data, and whether it moves under a stimulus the way real people move.

Synthetic audiences will never be a true substitute for the people they seek to represent. But we can test them more thoroughly than any panel of respondents. Do personas keep their beliefs when a question is reworded, do those beliefs hang together across the audience as human beliefs do, does the audience move for substantive changes and stay still for cosmetic ones, and does it reproduce effects from experiments the model has never seen?

Of the eight tests in Figure 3, the two marked with an asterisk are the industry standard today. We hold ourselves to all eight. Internally, we make a commitment to consistently evaluating our methodology so that alongside contributing to the frontier of synthetic research, others can check our numbers and compare where we fall short. The decisions our customers bring to us deserve nothing less than results they can act on with confidence.