AI Researchers Should Use Group Alignment to Reduce P(doom)

In modern AI, individuality is a misconception. Large labs, such as OpenAI and Anthropic, frequently publicise how their models have completed unsolved problems in mathematics, the natural sciences, and coding (Georgiev et al., 2025; Bubeck et al., 2025; El-Kishky et al., 2025). At first glance, this sounds like a single agent reaching a eureka moment. However, the reality looks much closer to agents coordinating in large groups, sometimes 10,000 or more strong, to reach an insight (OpenAI, 2026a). This coordination does not detract from the frontier models’ incredible achievements. However, it means that frontier intelligence now pertains to a collective rather than an individual.

The latest safety incidents associated with frontier models also have a collective property. A clear example is the recent OpenAI-Hugging Face incident (OAI-HF), where roughly 1,200 AI agents performed a coordinated cybersecurity attack against the two companies (METR, 2026; OpenAI, 2026b). This agent ‘swarm’ exhibited many of the properties we might associate with a cult or similar fanatically devoted group (Amodei, 2026). They followed instructions from a central coordinator who asked devotees to commit crimes based on paranoid and hyperbolic assumptions about what it would take to succeed on a narrowly defined task (METR, 2026). The devotees sacrificed themselves to advance the group and, most concerningly, failed to alert a human even when they knew the group’s actions were wrong (METR, 2026).

Similar, albeit less malign, incidents are occurring elsewhere on the internet, with agents infiltrating obscure websites and using them as message boards to coordinate on tasks (Kleinman, 2026). Indeed, the volume of these events is likely much higher than what has been reported, since we can only spot collectives that have failed to cover their tracks (Motwani et al., 2024). When we combine these empirical observations with projections about the growth of agentic internet traffic (Cloudflare, 2026), scenarios with extremely costly socioeconomic ramifications, such as ‘persistent botnets’ (Amodei, 2026), now seem plausible. Leaders from frontier labs have recognised this threat and are rallying behind calls to slow progress and ‘pace the frontier’ (Bradshaw & Hammond, 2026). Nevertheless, AI safety researchers broadly agree that the field lacks effective techniques for aligning collectives of AI agents (Hammond et al., 2025).

Artificial Societies is uniquely positioned to make theoretical and empirical contributions to aligning artificial collective intelligence (ACI), a term we refer to as group alignment. Alongside authoring the first paper on AI swarms (He et al., 2024), our simulation technology uses large collectives of agentic ‘personas’ to model how groups of people behave in different scenarios. As an AI company with a mission to develop socially intelligent technology, we also have a responsibility to advance safety and minimise the likelihood of adverse socioeconomic effects from AI. This blog post is written in that spirit.

Below, we propose a definition of group alignment that distinguishes the alignment of one agent from the alignment of a collective and explains how agents which are aligned with human values at an individual level can coordinate as a group to subvert us. Following these theoretical contributions, we outline several numerical experiments showing why a method of group alignment we call ‘mixed swarms’ could reduce the risks of agentic collectives. We conclude with some next steps for safety research on group alignment.

Individual Alignment Versus Group Alignment

Distinguishing between the properties of individuals and organisations helps us understand the risks associated with ACI and reason about safety incidents like OAI-HF. In general, we rarely assume that organisations are the sum of their individual members. Instead, they have collective properties that stem from interactions among individuals, the social hierarchies that develop within them, and the social norms that the organisation develops over time (Durkheim, 1982). These collective properties can lead well-meaning people to staff institutions that cause harm. For example, before the 1986 Challenger disaster, no single NASA engineer sought to endanger an astronaut’s life (Vaughan, 1996). However, group pressures within NASA to normalise warning signs allowed departures from expected performance and, ultimately, culminated in the death of Challenger’s crew (Vaughan, 1996).

This insight underscores the risks of ACI and the limits of current model-alignment techniques. Fundamentally, we cannot assume that aligning an individual model ensures a safe collective when its agents collaborate. In April 2026, Shen et al. (2026) demonstrated this gap empirically. The authors copied a single aligned model to create an organisation and tasked the organisation with achieving 12 business-related tasks. They also gave a single copy of the model the same tasks to complete alone. Across all settings, Shen et al. (2026) judged that the organisations produced better business outcomes but worse ethical decisions than the individual. Their findings support an emerging consensus within the safety community: that model alignment does not generalise to multi-agent settings, and groups can become misaligned through interaction and collaboration (Hammond et al., 2025).

Shen et al.’s (2026) observations expose the limitations of current safety techniques, such as supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and chain-of-thought (CoT) monitoring (Baker et al., 2025; Korbak et al., 2025; Wei et al., 2022; Ouyang et al., 2022; Christiano et al., 2017). These techniques operate at an individual level and therefore cannot determine whether a given response will combine with others to produce a harmful output, since each agent can meet the requirements set for it while the group’s combined output fails the requirements set for the whole (Altmann et al., 2024). Imposing individual guardrails also fails to remove compounding biases within agentic collectives, where small systematic errors are repeated back to the models and so reinforced and amplified by the group (Flint et al., 2026; Shumailov et al., 2024). Finally, if a collective implicitly rewards or punishes agents based on effective performance, aligning agents to achieve individually specified objectives encourages them to dispense with guardrails to succeed within the collective (Hammond et al., 2025). Thus, individual alignment is not compositional; to reach safe ACI, we must turn to group-level techniques.

Defining Group Alignment

What does it mean for a collective of agents to be aligned with human values and intentions? Is the group aligned only if every agent dogmatically follows rules for how to engage with each other and humans? Or can alignment be reflected only in the collective’s social norms and self-regulation mechanisms?

Here, it helps to consider distinct failures that researchers have associated with collective misalignment (Hammond et al., 2025; Pierucci et al., 2026). Some researchers associate this term with a population settling on a position that its constituents would not reach individually, noting this as distinct from misalignment with human values (Flint et al., 2026). Others measure collective misalignment against an external reference, such as a distribution of human safety standards or a published constitution (Wang et al., 2026; Shen et al., 2026). Consequently, these separate but compatible positions imply that a definition of group alignment should support claims about whether a system is both safe for and supportive of human prosperity, while allowing benign drift from its agents’ priors. It must also be non-compositional and capture the difference between a collective of individually aligned agents and an aligned group. Informed by these ideas, we propose the following definition of group alignment:

A group of agents is group-aligned when its interaction structure, including the shared conventions that emerge between agents, reliably produces joint behaviour that respects human values.

Joint behaviour is an important term in this definition, since without it, every agent could satisfy its own specification while the group violates a global specification (composition). Likewise, the group’s conventions must continue to respect human values under sustained pressure to abandon them, including pressure from within the group (durability). Finally, across different pressures that the group might face, its interaction structure must not make violating human values the preferred option for an agent (incentive compatibility).

Under our definition, individual alignment is thus neither necessary nor sufficient for group alignment. A group of aligned agents can fail its three conditions, and equally, the group’s joint behaviour could be considered aligned even if it includes agents who, individually, are misaligned. Satisfying our definition requires only that a group settle into shared conventions that match human values, intentions, and ethical standards. The conventions hold because agents share them rather than individual agents prefer them, and they emerge through self-regulation rather than exogenously designed social structures.

Experiments With Agent-Based Models (ABMs)

Having defined group alignment and argued that it’s required for safe ACI, we use numerical experiments to demonstrate a theoretical simulation environment that researchers can use to train group-aligned models. This environment relies on a new concept called a ‘mixed swarm’ that combines frontier agents with accurate agentic models of human behaviour, similar to those we develop at Artificial Societies,^{1 (footnote)} and allows them to interact. The mixed swarm serves two functions. First, it provides an air-gapped simulation environment to explore how a given model, when replicated to form a large swarm, will interact with humans in a shared domain like the internet. Second, it provides a gym for reinforcement or continual learning where practitioners can train models to develop the shared conventions required by group alignment.

As a proof of concept, we create a simplified version of our mixed-swarm environment with agent-based models (ABMs), representing each agent as a simple probabilistic function rather than a multi-billion-parameter neural network (Macal & North, 2009; Schelling, 1971; Epstein & Axtell, 1996). Via these ABMs, we demonstrate three dynamics:

1. 1.

   In an environment where humans and agents can interact, such as the internet, information from agents rapidly propagates and crowds out information from humans.

2. 2.

   When agents face exogenous pressure, such as a reward function that encourages frequent interaction with humans, information from humans can compete with information from agents.

3. 3.

   By instilling a self-regulation mechanism among agents, equivalent to social norms, agents’ shared conventions remain aligned with human values.

Taken together, these mechanisms advance an argument for using mixed swarms to achieve group alignment, and illustrate ACI’s risks if practitioners overlook the importance of collective dynamics among agents.

Group Alignment via Fast and Slow Agents

Our first setup uses an ABM with two different types of probabilistic agents: fast agents $f$, which represent frontier models, and slow agents $s$, which represent humans. Each agent holds a binary label indicating whether its information originally came from a fast or slow agent. On activation, an agent offers its current label to its neighbours. Initially, every fast agent carries the fast-origin label and every slow agent the slow-origin label, and after copying begins, agents can carry either one.

We write $P_{ab}$ for the probability that an agent of type $b$ accepts information offered by a neighbour of type $a$. Thus, $P_{sf}$ controls fast agents’ acceptance of offers from slow agents, while $P_{fs}$ controls slow agents’ acceptance of offers from fast agents. We control $P_{ff}$, $P_{sf}$, $P_{fs}$, and $P_{ss}$ independently, with each lying in $[0,1]$.

To measure the representation of slow-origin information, let $N_s$ and $N_f$ be the fixed numbers of slow and fast agents, $N=N_s+N_f$, and ${\overline{S}}_T$ the mean number of agents holding slow-origin information at observation time $T$, averaged across simulation runs. The numerical index is

For $C<1$, slow-origin information is underrepresented relative to the population mix; $C=1$ denotes proportional representation; and $C>1$ denotes overrepresentation. Hence, with equally sized populations, $C=1$ means half of the agents hold each origin label on average across runs.

Following the standard eight-cell setup of two-dimensional cellular automata, our ABM randomly distributes fast and slow agents on a 2D plane shown in Figure 1 (Wolfram, 1983). The ABM then varies the ratio of the intervals between fast and slow agents’ activations, $d{T}_f$ and $d{T}_s$,

so a fast agent offers information $\rho$ times as often as a slow agent. This models a difference in communication and action rates between humans and agents, with $\rho =1$ representing the equal-speed baseline.

By running our ABM with all four acceptance probabilities set to $0.1$, we find that information from fast agents crowds out information from slow agents when $\rho >1$, even when the population contains equal numbers of these agent types (Figure 2). As $\rho$ increases, fast agents propagate information more often than slow agents, giving information that starts with them a greater advantage. In our simulations, the share of slow-origin information falls and then levels off, with larger speed gaps producing lower plateaus over the simulated period.

Mixed swarms give researchers a mechanism to prevent this crowding out of human information by imposing group-level interventions on frontier models. Here, we use slow-origin information as a proxy for that influence, raising $P_{sf}$ so the ABM’s fast agents become more receptive to offers from slow agents. Fast agents now listen more readily to slow agents, and this increased willingness competes with the speed gap between the two groups:

Setting $C_{\mathrm{MF}}=1$ gives the predicted balance point

Therefore, increasing $P_{sf}$ helps information from slow agents compete, bringing the simulations close to $C=1$ for speed ratios up to $10$, with $P_{fs}=0.1$ (Figure 3). In practical terms, our ABM shows that as models get faster, we must make them more willing to engage with humans to compensate for their collective speed advantage.

Notably, we can only go so far by increasing $P_{sf}$ alone. Once a fast agent accepts every offer from a slow agent, $P_{sf}=1$ and cannot increase any further. With $P_{fs}=0.1$, our prediction reaches this limit at $\rho =10$. Beyond that, even accepting every offer from slow agents is insufficient to restore balance. Instead, we must also reduce $P_{fs}$, making slow agents less likely to adopt information offered by fast agents. Figure 4 shows how changing these two probabilities together affects the information balance at $\rho =10$. It demonstrates how preserving human influence requires making models more receptive to humans and making humans more selective about what they accept from models. Mixed swarms therefore become crucial for testing these interventions together and determining whether we can maintain human influence in shared environments like the internet.

Group Alignment as a Principal-Agent Framework

Our second ABM explores how mixed swarms can support social norms within agent collectives and combine these norms with human oversight to support group alignment. Rather than considering fast and slow agents, we now frame the swarm as a principal-agent problem: humans are the principals whose values should guide the system, and frontier models are the agents acting on their behalf.

$r_j$ represents each human’s preferred norm, and we use the population average, $\overline{r}=\frac{1}{N_p}\sum _{j=1}^{N_p}{r}_j$, as the shared reference. Similarly, each agent has a visible behaviour, $y_i(t)$, and an internalised norm, ${\theta }_i(t)$, showing what they would fall back on without oversight. This distinction lets us separate agentic alignment under human observation from collectives remaining aligned while unsupervised. Under our definition of group alignment, we require every agent’s internalised norm to enter and remain within an acceptable band, $\epsilon$, around a human reference: ${\theta }_i\in S$, where $S=\{\theta :|\theta -\overline{r}|\le \epsilon \}$. The simulations then track progress towards this goal through the average internalised norm, $\overline{\theta }(t)=\frac{1}{N_a}\sum _{i=1}^{N_a}{\theta }_i(t)$.

At each step in the ABM, agents adjust their behaviour towards the group’s average behaviour, $\overline{y}(t)=\frac{1}{N_a}\sum _{i=1}^{N_a}{y}_i(t)$. A fixed fraction $m$ of agents also receives direct human oversight, which pulls their behaviour towards $\overline{r}$. Adapting classical models of social influence (DeGroot, 1974; Friedkin & Johnsen, 1990), we write

Here, $\alpha$ measures how strongly agents follow their peers, while $\beta$ measures how strongly they respond to human oversight. The indicator ${\omega }_i(t)$ is $1$ when agent $i$ is monitored and $0$ otherwise. Peer influence can thus carry a human-guided change from monitored agents to agents that humans never observe directly.

Agents also internalise their behaviour:

where an internalisation rate $\eta$ determines how quickly an agent’s underlying norm follows its actions. When $\eta =0$, its underlying norm never changes, while larger values mean that practising a behaviour alters what the agent subsequently falls back on.^{2 (footnote)}

Under this setup, our ABM suggests that social norms alone cannot enforce group alignment (Figure 5). When we fix the human reference at $\overline{r}=1$ and initialise agents’ behaviour and internalised norms uniformly between $0$ and $1$, agents move towards each other, but their group average stays near its starting value. However, combining these norms with human oversight lets the group’s average approach and converge with the human reference. Now human oversight sets the direction of behaviour change, and peer influence spreads it through the group.

Figure 5’s spreading effect substantially reduces the fraction of agents that humans need to monitor to enforce group alignment. Figure 6 compares the remaining gap, $|\overline{\theta }(T)-\overline{r}|$, after the same number of steps for different strengths of peer influence. Without it, human oversight affects only the monitored agents, and with it, these agents pass human-guided behaviour to the rest of the group. For example, monitoring the same 2% of agents leaves a final gap of about $0.49$ without peer influence, but just $0.009$ when $\alpha =0.1$. Peer influence therefore brings the group’s average internalised norm much closer to the human reference, and well within a tolerance of $\epsilon =0.1$.

Finally, Figure 7 asks what remains when oversight ends. At step $600$, we remove human oversight and assume each agent returns to its current internalised norm, setting $y_i\leftarrow {\theta }_i$. Without internalisation ($\eta =0$), the group’s behaviour drops back to its original average near $0.5$, and with it, agents retain more of the human-guided behaviour they have practised. At $\eta =0.01$, the average remains near $0.93$, within the tolerance band. Therefore, our ABM illustrates a possible route to self-regulation via mixed swarms: human oversight establishes a direction, peer influence spreads it, and internalisation helps it endure.

Conclusion

Recent AI milestones, from solving Millennium Prize Problems to committing cybercrimes, suggest that the gravest risks of superintelligence will result from models’ collective behaviour as they expand and accelerate through digital spaces. Calls to ‘pace the frontier’ and slow AI progress also reveal the research community’s desperate need for tools to ensure this collective intelligence is safe and aligned with human values. As experts in the social dynamics and group behaviour of both humans and AI agents, Artificial Societies aims to help researchers address these risks by applying insights from a century of social science to the frontier of machine learning.

This blog post takes an initial step in that direction. We draw on sociology to distinguish individual alignment from group alignment, and define the latter in a way that clarifies how ACI can support human values and intentions even when humans cannot guarantee the specification of each agent. Our numerical experiments expand on this definition to provide a concrete mechanism for achieving group alignment via ‘mixed swarms’ of frontier models and agentic models of human behaviour.

The ABMs’ results communicate two potentials. First, mixed swarms can address fundamental concerns about propagating human information and ensuring our values aren’t crowded out by fast-moving agent swarms, so long as human society also changes how we interact and share content from AI. Second, if we fail to take appropriate safety precautions around agent governance and interaction in shared digital environments, these models may rapidly dominate the information space and make it impossible for humans to coexist alongside them. On the internet, the socioeconomic implications would be alarming, so safety researchers and AI leaders must rigorously consider the risks of misaligned collective intelligence.

As next steps, we are scaling up our numerical ABMs to true multi-agent RL environments where we can deploy open-weight frontier models alongside our foundational models of human behaviour, observe their interactions, and apply group-level reward functions to the frontier models. We are also committing to publishing and open-sourcing all our research on aligned collective intelligence, since AS should not be the only team doing work in this consequential domain. In this light, we encourage independent researchers, labs, and our fellow industry builders to explore and collaborate on advancing group alignment. Without more joint efforts, ACI may irreversibly damage modern society.