Skip to content

Method & Evaluation

How Artificial Societies accurately models human opinion

Artificial Societies uses networks of AI personas to simulate high-value audiences, used by Fortune 100 organisations to test how strategies shift opinions that matter. Here’s how AS models human opinion, and how it performs against human panels and simple synthetic personas.

From market intelligence to market simulation.

Each persona is grounded in a real individual, then connected into a realistic network of influence.

The standard approach of creating a synthetic persona is by prompting an LLM with an invented biography (e.g. “you are a 34-year-old teacher from Ohio”), which produces generic, stereotyped answers.

Artificial Societies is different. We employ a unique and proprietary approach, developed over years of research originating from the first large-scale AI society paper. This approach combines real-world intelligence gathering, the mapping of individual belief systems, and a unique network science modelling method.

  1. Data sources

    Start from real people.

    Published research, online commentary, reviews, anonymised social data, and public records of voting and investment show how people communicate and behave in context.

    LinkedIn, X, Reddit, Facebook, YouTube, Instagram, Amazon and more

    First-party data (Optional)

    Your proprietary data can be layered into the persona set. First-party data is segregated per engagement, held in the EU, and never used for training our models.

  2. Persona construction

    Model their belief systems.

    Psychometric methods reconstruct how individuals reason, allowing us to predict reactions to new information. Every persona is validated before entering a society.

    Personas validated against audience benchmarks and prior behaviour

  3. Society creation

    Simulate how opinion moves.

    Personas are clustered by shared characteristics and connected through real relationships and patterns of influence. Results can be analysed at market, segment or persona level.

    A society drawn as a network: each dot is a persona, coloured by the segment it clusters with, and each line a relationship between two of them.

    12 to 3,500 personas per society

Achieving individual nuance, at scale.

Artificial Societies combines the population accuracy of synthetic populations with the individual richness of digital twins.

Individual nuance →

  • Digital twins

    one individual

  • Artificial Societies

    networks of personas

  • Synthetic populations

    population averages

Large samples →

  • Digital twins

    one individual at a time

    • Strength: Detailed individual nuance with interpretable reasoning
    • Limitation: No reliable population-level distributions and difficult to scale

  • Artificial Societies

    networks of enriched personas

    • Strength: Accurate population distributions with rich individual nuance
    • Strength: Reliable segments and crosstabs across high-value audiences

  • Synthetic populations

    invented population averages

    • Strength: Accurate top-line averages
    • Limitation: Invented individuals with unreliable segmentation and crosstabs

How it performs

Artificial Societies accurately captures human opinion distributions, and every individual response is interpretable with why they answered as they did.

Example 1 of 4

Is ‘everyone should be treated fairly’ an American value?

95% distribution accuracy

Human responses

78% agree · 12% disagree · 10% not sure

Artificial Societies

73% agree · 15% disagree · 12% not sure

  • Agree
  • Disagree
  • Not sure

  • Community Organizer

    Director, neighborhood nonprofit

    Advocating for underserved families.

    Atlanta, United States · Female · Millennial · Director · Nonprofit

    ResponseAgree

    “Fairness is the one thing this country is supposed to stand for. If we don’t believe that, what are we even doing?”

  • Hedge Fund Analyst

    Analyst, long/short equities

    Earning those outcomes.

    New York, United States · Male · Gen Z · Mid-level · Finance

    ResponseDisagree

    “This place rewards the people who show up and work harder than everyone else. That’s the real American value.”

Tested against 1,000 real-world surveys.

Artificial Societies benchmarks better than LLMs on overall distribution accuracy and deeper measures of realism across populations and opinion topics.

  • Distribution accuracy

    86%

    Overlap between simulated and human opinion distributions.

    • Artificial Societies86%
    • Biography-prompted LLMs67%
    • Human self-replication ceiling91%

    What this means: Reliably test how a decision shifts opinion.

  • Hallucination rate

    <2%

    How often simulated personas contradict themselves. Lower is better.

    • Artificial Societies<2%
    • Human panels~9%
    • Biography-prompted LLMsup to 35%

    What this means: Higher-quality signal on high-value audiences.

  • Internal coherence

    89%

    Alignment between questions measuring the same underlying attitudes (Cronbach’s α).

    070% threshold100%
    • Artificial Societies
    • Human range 60–95%

    What this means: Valid relationships between opinions, needed for critical strategies.

  • Open-response quality

    93%

    Distinct-word richness benchmarked against 120,000 human social-media posts.

    • Artificial Societies93%
    • Biography-prompted LLMs61%
    • Human social media91%

    What this means: Open-ended answers with human-level nuance.

Source: Artificial Societies Survey Evaluation Report, (PDF).

Holding your own data to the same standard? Run the free data quality check on any survey dataset.