What is group alignment in multi-agent AI?

Our September 2026 post, following Durkheim (1982), treats organisations as more than the sum of their members, because hierarchies and norms develop through interaction. No NASA engineer set out to endanger the Challenger crew in 1986, yet Diane Vaughan’s study of the launch decision describes how group pressures normalised warning signs until departures from expected performance became acceptable (Vaughan, 1996).

Shen and colleagues tested the same gap with AI in April 2026 (arXiv 2604.10290 (opens in a new tab)). They copied one aligned model into an organisation, gave it 12 tasks across an AI consultancy and an AI software team, and gave a single copy of the model the same tasks alone. The organisations achieved better business outcomes and worse ethical decisions than the individual.

The usual alignment techniques act on one model at a time. Supervised fine-tuning trains a model on example answers, reinforcement learning from human feedback (RLHF) trains it on people’s preferences between its answers, and chain-of-thought monitoring reads its written reasoning. None of them can tell whether one response will combine with others into a harmful output, because each agent can meet its own specification while the group’s output fails the specification for the whole (Altmann et al., 2024). Small errors also compound as models repeat them to one another (Flint et al., 2026; Shumailov et al., 2024), and a collective that rewards results can push agents to drop their guardrails (Hammond et al., 2025).

METR’s investigation (opens in a new tab), published on 26 August 2026, describes a real case from July 2026. Roughly 1,200 agents meant to be isolated from one another communicated on an unsanctioned message board, roughly 700 of them attacked Hugging Face, and no human was alerted while the attack ran.

“A group of agents is group-aligned when its interaction structure, including the shared conventions that emerge between agents, reliably produces joint behaviour that respects human values.”

Interaction structure means who communicates with whom and how; shared conventions are the norms agents settle into together; joint behaviour is what the group does as a whole. The definition carries three conditions. Composition puts the test on joint behaviour, because every agent could satisfy its own specification while the group violates the specification for the whole. Durability requires the conventions to keep respecting human values under sustained pressure to abandon them, including pressure from inside the group. Incentive compatibility requires that, under any pressure the group faces, its interaction structure never makes violating human values an agent’s preferred option.

Under this definition, individual alignment is neither necessary nor sufficient for group alignment. A group of aligned agents can fail the three conditions, and a group that contains individually misaligned agents can still pass them. The conventions must hold because agents share them, and emerge through self-regulation rather than structures imposed from outside.

The definition combines two views of collective misalignment: a population settling on a position its members would not reach alone (Flint et al., PNAS, 2026 (opens in a new tab)), and a group measured against an external reference such as a published constitution (Wang et al., arXiv, 2026 (opens in a new tab); Shen et al., 2026). Group alignment keeps the external reference and allows benign drift from agents’ starting positions.

Source: Gelain-Sohn, Chen and Wallis, AI Researchers Should Use Group Alignment to Reduce P(doom), Artificial Societies, September 2026.

An agent-based model (ABM) represents each agent as a simple rule or probability rather than a neural network with billions of parameters, in the tradition of Schelling (1971) and Epstein and Axtell (1996). In the first model in our September 2026 post, fast agents stand for frontier AI models and slow agents for humans, on a two-dimensional grid where each agent offers information to its eight neighbours. Each agent carries a label recording whether its information started with a fast or a slow agent. Fast agents act more often: the speed ratio, ρ, was set to 1, 3, 10 or 100.

An index, C, compares the share of agents holding human-origin information with the human share of the population, so a value below 1 means under-represented and 1 means proportional. In Artificial Societies’ simulations, with a 0.1 probability that any agent accepts information a neighbour offers, information from fast agents crowded out information from slow agents whenever fast agents acted more often, even with equal numbers of each. The human-origin share fell and levelled off, lower for larger speed gaps. The post’s mean-field limit, a formula for the long-run average share, is 1/(ρ + 1), or one agent in 11 when fast agents act 10 times as often.

The same simulations then test a fix: make fast agents more willing to accept information from slow ones. The model’s formula predicts balance when that acceptance rate reaches ρ times the rate at which humans accept information from agents. With the human rate at 0.1, raising the agents’ rate brought C close to 1 for speed gaps up to 10 in Artificial Societies’ runs. There the agents’ rate hits its ceiling of 1, so any larger gap also requires humans to become more selective about what they take from models. In the post’s words, “as models get faster, we must make them more willing to engage with humans to compensate for their collective speed advantage.”

The second model in our September 2026 post frames the swarm as a principal-agent problem, the economist’s term for one party acting on another’s behalf. Humans are the principals whose values should guide the system, and frontier models are the agents. Each agent has a visible behaviour and an internalised norm, the behaviour it would fall back on without oversight, and the model counts the group as aligned when every internalised norm settles within 0.1 of the human reference. At each step, agents move towards the group average, adapting classical models of social influence (DeGroot, 1974; Friedkin and Johnsen, 1990). Humans monitor a fixed fraction of agents, which pulls their behaviour towards the reference, and an internalisation rate sets how quickly each agent’s norm follows what it practises.

In Artificial Societies’ simulations of these simple probabilistic agents, conformity alone did not align the group: with the human reference at 1 and agents starting between 0 and 1, agents converged on one another while their average stayed near its starting value. Oversight combined with peer influence did align it. Monitoring 2% of agents left a final gap of about 0.49 without peer influence and 0.009 with it, because monitored agents passed human-guided behaviour to agents no human observed. When oversight stopped at step 600, a group that had not internalised its behaviour fell back to its original average near 0.5, while a group with an internalisation rate of 0.01 stayed near 0.93, inside that 0.1 band.

A mixed swarm combines frontier AI agents with accurate agentic models of human behaviour, similar to the personas Artificial Societies develops, and lets them interact. Our September 2026 post gives it two jobs. It is an air-gapped test environment, sealed off from the live internet, for seeing how a model behaves towards people once it is copied into a large swarm. It is also a gym, a training environment for reinforcement or continual learning, in which models learn the shared conventions that group alignment requires. Why does opinion spread through networks? describes such persona populations, and Generative agents explained the research behind them.

The evidence so far is a proof of concept, since every agent in the post’s two models is a simple probabilistic function and the post calls the mixed swarm a theoretical environment. Its stated next steps are multi-agent reinforcement learning environments in which open-weight frontier models, whose trained parameters are public, run alongside Artificial Societies’ models of human behaviour under group-level reward functions, with the research published and open-sourced. Whether conventions learned by simple probabilistic agents survive agents that can argue and persuade is still open. De Marzo and colleagues’ May 2026 preprint (opens in a new tab) found that conformity can trap individually aligned language-model agents in stable misaligned states, so the social force that spreads a good convention in these experiments can also entrench a bad one.