How accurate are AI personas? 2026 evaluation

AI persona accuracy needs a named measurement. Artificial Societies’ method and evaluation page reports four, with a human comparator for each. The model baseline is a biography-prompted large language model (LLM): a model instructed to answer as a person with an invented biography, which is one kind of synthetic persona. Each row below asks a different question about the resulting responses.

Source: Artificial Societies, Survey Evaluation Report, January 2026, presented on our evaluation page. Interpretation follows our September 2026 validity framework.

The distinction matters when you read a persona’s explanation. A vocabulary-rich answer earns a language score; it still needs tests of the beliefs behind it. Likewise, the hallucination figure counts self-contradiction within the simulation. Reading it as a factual-error rate would assign the result a meaning the test never measured.

Cronbach’s alpha addresses a separate issue: whether responses to questions about an underlying attitude move together. The coherence result sits within the human-panel range. For your study, that is evidence about the relationships between answers, while population accuracy asks whether those answers resemble the people you intend to understand. We keep the measurements separate because combining them into a single percentage would obscure the reason to trust each one.

Artificial Societies’ benchmark results of 1 October 2026 compare it with named models: on the Artificial Societies Benchmark, Artificial Societies had lower error than all nine off-the-shelf LLMs tested on every measure reported, scored on 68,280 answers from seven survey sources with the same respondents, items and scoring code for every system. Artificial Societies’ answer distributions needed an average adjustment of 8 percentage points to match the human responses, against 31 for the best LLM, Qwen3.8. In the Albertson survey experiment, which asks the same question in two wordings, Artificial Societies reproduced the resulting shift in answers to within 1.4 percentage points, against 6.2 for the best LLM. The benchmark reports error, where lower is better, so its figures sit beside the four measures above rather than replacing them; its tests are defined in Artificial Societies Benchmark: A Validation Framework for Synthetic Research (opens in a new tab) (arXiv, September 2026).

Independent studies of synthetic respondents report results that range from close replication to weak correlation, and the spread follows what each study measured. The five below cover the main designs: agents built from one person’s own answers, digital twins, persona-prompted polling, forecasts of experiment results and consumer ratings. We read each on its publisher’s page on 28 September 2026.

Source: each study’s publisher page, read 28 September 2026.

Three patterns matter for a buyer. Averages are easier than relationships: Bisbee’s averages matched while nearly half the regression coefficients did not. That is the problem our September 2026 validity framework raises, and one the persona-prompting method known as silicon sampling has to answer. Direction is easier than size: Ashokkumar’s forecasts tracked the relative size of treatment effects but overstated their magnitude. And the grounding data decides a great deal, since Park’s demographics-only agents trailed interview-and-survey agents by 12 points on the same test.

Peng and colleagues name five distortions in digital twins: insufficient individuation, stereotyping, representation bias, ideological bias and hyper-rationality. Our article on AI persona bias covers the representational ones, and how to evaluate a synthetic research vendor turns them into questions to ask. Park’s 86% and Artificial Societies’ 86% share a number, not a measure: the first measures how well an agent reproduces one person relative to that person’s repeatability, the second how closely a simulated population’s answers overlap with a human one.

Artificial Societies tests its simulations against two kinds of real evidence: published experiments with human participants, and events that had not happened when the predictions were logged. In Simulating Misinformation, Fact-Checking, and Prebunking, published on 23 September 2026, Artificial Societies reproduced four published UK experiments on its UK General Population society and compared the results with evidence from more than 25,000 real Britons. The design, instruments, code and panel were hashed and time-stamped before the first survey call.

In the fact-check experiment (Carey and colleagues, 2022), fact-checks cut belief in four false COVID-19 claims by 46% against the control group in the synthetic panel, where the paper reported 45%. The same replication recorded a miss: in the vaccine-misinformation experiment, factual posts moved the synthetic control group about four points towards vaccination, while 1,000 Britons showed no significant shift, which the post reads as a small bias towards vaccine-positive content. Prebunking vs debunking covers what the experiments show about each intervention.

The second test removes any chance that the answer sat in a model’s training data. In Testing Simulations Against Reality: Earnings Calls, published on 21 September 2026 by Dr Linda Wei and Edoardo Chidichimo, Artificial Societies preregistered on the Open Science Framework the questions its simulated sell-side analysts would ask before the NVIDIA, Marvell and Credo calls, then compared them with the transcripts. For Marvell and Credo, detailed analyst personas came closer than basic profiles to the real questions in substance and phrasing. The limits are in the same post: for Credo the detailed personas did not improve prediction of the broad topics, and analysts’ own past questions remained a strong baseline for wording and identity.

Artificial Societies uses networks of AI personas to simulate high-value audiences. Y Combinator’s company profile (opens in a new tab) describes our work as “We build networks of AIs to simulate high-value audiences”. The network is part of the method: we model individual belief systems and connect personas through real relationships and patterns of influence.

Our method page begins with observations, including published research, online commentary, reviews and anonymised social data. Public records of voting and investment also contribute. Psychometric methods reconstruct how individuals reason, and we check each persona against audience benchmarks and prior behaviour before it joins a society. The resulting population therefore has an observational basis you can discuss when defining the audience for your study.

Society creation groups personas by shared characteristics, then connects them through relationships and influence. Artificial Societies’ method write-up specifies 12 to 3,500 personas per society. The Simulation Engine exposes those personas to stimuli and scenarios, with results available at individual, segment and population level. That lets you examine the people behind an aggregate response. The scope still depends on the observations: an accuracy result for an observed population cannot establish accuracy for an audience with no diverse evidence to build from.

Distribution matching tests the balance of answers to a question. Our September 2026 article, Distribution Matching Is a Bad Way to Evaluate Simulations, explains why a matching balance can coexist with the wrong combinations of beliefs. You can reproduce the number of people choosing each answer without reproducing which people chose those answers together.

The article demonstrates this with simulated audiences whose answer distributions match question by question but whose joint distributions differ. For a communications study, that distinction belongs in the evaluation before you interpret a segment’s reaction. A population total cannot, by itself, tell you whether the relationships among attitudes are credible. We use coherence tests alongside distribution overlap to examine that separate requirement.

Human responses also set the appropriate consistency target. Artificial Societies’ January 2026 Survey Evaluation Report uses a 91% human self-replication ceiling: people repeat their answer to the same question roughly that proportion of the time. Our September framework explains that people’s answers depend on how and when you ask, and who they think is listening. A simulation designed to give an identical answer on every repeat would discard that human variation. We would judge a persona against human response patterns before rewarding it for perfect repeatability.

Artificial Societies’ September 2026 validity framework, by Edoardo Chidichimo and Felix Wallis, sets out eight tests across three kinds of validity. Internal validity asks whether the stimulus caused the response shift. Construct validity asks whether the questions measure the intended attitude. External validity asks whether the findings hold for real populations. These questions turn an accuracy discussion into an examination of the research design.

At persona level, we test agreement between equivalent phrasings and check whether answers to linked questions contradict one another. At questionnaire level, the framework calls for changes that should leave the distribution stable: reorder answer options, reverse a rating scale or change question order. It also calls for changes that should affect the result. Removing evidence behind a claim should weaken its reception; changing the messenger should change how the message lands.

Those tests address sycophancy, the tendency of LLM personas to agree with a question’s implied view. A favourable reaction to confident wording could reflect agreement with the prompt rather than persuasion. Our framework makes that distinction testable instead of treating a convincing explanation as sufficient evidence.

Construct tests examine whether related questions behave as human research would lead us to expect. External tests compare granular audience groups with human responses and assess whether simulations reproduce the size of changes found in human experiments. They also examine the underlying pattern of relationships among answers. Artificial Societies’ commitment in the framework is explicit: “We hold ourselves to all eight.”

One peer-reviewed paper underpins the group-behaviour modelling. He, Wallis, Gvirtz and Rathje’s Artificial intelligence chatbots mimic human collective behaviour (opens in a new tab), published online in the British Journal of Psychology in December 2024, studied 33,299 chatbots. Communities formed around a shared language; within the 17,746 predominantly English-using chatbots, communities formed around similar posted content, according to the paper.

The finding concerns homophily: people’s tendency to form communities with similar others. It supports the group-behaviour modelling behind artificial societies. For a buyer interested in how opinion moves through connected audiences, that is relevant evidence about why we model a society rather than stop at individual responses.

The paper and the survey evaluation answer distinct questions. The December 2024 study validates the reproduction of that collective behaviour. Artificial Societies’ survey figures come from its own January 2026 Survey Evaluation Report and are not peer-reviewed findings from the British Journal of Psychology paper. You can inspect both sources without treating the paper’s publication as certification of the later accuracy figures.

We would use Artificial Societies to test how a proposed strategy or message shifts opinion in an audience with diverse observations behind it. Public evidence and client-provided data can provide that grounding. When the audience lacks those observations, we would start by obtaining them; the method depends on evidence about the people you want to understand.

The decision also determines the evidence you need. Our September 2026 validity framework describes the intention–behaviour gap: what people say can be a weak predictor of what they subsequently do. Synthetic research inherits that limit from surveys. For strategic communications, testing opinion is a defined research task. If your decision requires proof of actual behaviour, we would commission research that observes it and use simulation for its defined task of testing opinion.