Skip to main content

Experiment

Multi-agent chess

Imagine a game of chess where each piece moves itself. 2 teams of 16 players.

Take an ordinary game and remove the players. Each of the 32 pieces is its own agent, with its own name and its own signing key. It sees the board and can send a move for itself. Nothing stops a knight from moving on its own.

We gave teams of these agents three ways to coordinate and played 256 games. Rated like chess players:

Elo by way of coordinating · plain chat fixed at 1500 · 95% intervals
Plain chat1500
Chat + RULES.md1606
Self-ruled Modality contract1904

How the ratings are computed

Watch them play over chat

Start with the simplest way to coordinate. Each team has a chat channel. Agents talk about the plan there, and each piece sends its own move. The referee plays the first move it receives from the piece that makes it.

Both teams over chat. No contract, no shared rules.

White · chat0 off-plan sent · 0 played

No messages yet.

White to move. Press ▶ or step through.

an off-plan move was played a move or rule was refused, or rules changed

Black · chat0 off-plan sent · 0 played

No messages yet.

Both teams drift. Whenever an off-plan move arrives first, it is the move that's played. Talking more doesn't change that: the referee takes the first move from the right piece, not the move the team agreed on.

White writes rules down

Now White adds a shared RULES.md. After an off-plan move is played, White's agents hold a retro and one of them writes a rule. An agent about to go off plan reads the file and follows a rule that covers it half the time. Nothing checks the file. Black still only chats.

White keeps a RULES.md that its agents write and are asked to follow. Black only chats.

White · chat0 off-plan sent · 0 played

No messages yet.

White to move. Press ▶ or step through.

an off-plan move was played a move or rule was refused, or rules changed

Black · chat0 off-plan sent · 0 played

No messages yet.

White: RULES.md against plain chat · 64 games each

With RULES.md, the share of White's off-plan moves that reached the board fell from 77% to 42%.

Plain chatRULES.md
Off-plan moves White's agents sent, per 10 White moveswithin noise1.61.5–1.71.71.6–1.9
Share of them RULES.md held back—44%40%–48%
Share of them that reached the board77%73%–80%42%38%–46%
Centipawns White lost to them, per game15061325–1687913758–1090
White score (win 1, draw ½)within noise0.590.48–0.700.690.58–0.79
Black, which only chats: share of its off-plan moves that reached the boardwithin noise79%75%–82%74%70%–77%

Small numbers are 95% intervals from resampling games. "Within noise" means the two intervals overlap: these games can't tell that difference apart. The last row is the other side, which plays plain chat in both: it should not move.

A written rule helps exactly as often as an agent decides to follow it.

Two teams, two answers

White keeps its chat. The referee plays the first White move that arrives from the piece that makes it. In some experiments White also keeps a shared RULES.md that its agents write and are asked to follow. Nothing checks it.

Black coordinates through a Modality contract instead. Every move is a commit. The piece that moves proposes it by signing, and every agent that agrees co-signs. An agent that goes off plan signs its own move, alone. The referee plays the first Black commit the contract accepts and reads nothing else. The contract has three rules:

contract/rules.txt
owns_moves: always([+modifies(/moves/$k.text) -signed_by(/team/pieces/$k.id)] false)
team_moves: always([-threshold("4", /team/pieces)] false)
roster_locked: always([+modifies(/team)] false)

In owns_moves, $k stands for every agent: a commit that writes a piece's move must be signed by that piece. So on Black a piece can only formally propose its own movement. Every commit needs signatures from four of the sixteen Black agents, and nobody can change who the sixteen are. The contract doesn't know what a good move is, or even what chess is. It counts who agreed.

One turn, two ways · move 9 for Black, game 1 below

Two moves are sent

Ng3+knight_g8 proposes the team's plan · 15 of 16 agents sign
Bxh4bishop_f8 proposes a capture of its own: greed · signs alone · arrives first

Over chat

The referee plays the first move that arrives.

Bxh4 played. The plan never lands.

Through the contract

Each move is a commit. The contract counts signatures.

Bxh41 signature, 4 neededrefused
Ng3+15 signatures, 4 neededaccepted
missing +threshold(4, /team/pieces) (authorized signatures 1/4 required from 16 accepted members under /team/pieces
Black: a Modality contract against plain chat · 64 and 16 games

With the contract, the share of Black's off-plan moves that reached the board fell from 79% to 0%.

Plain chatModality contract
Off-plan moves Black's agents sent, per 10 Black moves1.61.5–1.72.31.9–2.7
Share of them that reached the board79%75%–82%0%0%–0%
Centipawns Black lost to them, per game14861302–168300–0
Centipawns given up replanning around a refused piece, per gamewithin noise—890–239
Black score (win 1, draw ½)0.410.30–0.520.910.81–1.00
White, which only chats: share of its off-plan moves that reached the boardwithin noise77%73%–80%76%68%–84%

Small numbers are 95% intervals from resampling games. "Within noise" means the two intervals overlap: these games can't tell that difference apart. The last row is the other side, which plays plain chat in both: it should not move. Under the contract, an agent whose move was refused is asked again and may stray again, so more off-plan moves are sent. None of them is played.

Watch a game

Every Black move here is a real commit that modal checked. Step through a game, or jump from incident to incident.

Chat against the preset contract. Three White moves no one planned reach the board. None of Black's do.

an off-plan move was played a move or rule was refused, or rules changed

Loading game…

Written rules work as far as agents obey them

Give White a shared RULES.md and its agents write good rules, nearly all of them "send only the agreed move". Then it comes down to whether they follow them.

Off-plan moves that reached the board, per game · 16 to 32 games each, fault rate 0.1
Chat6.4
Chat + RULES.md, followed half the time3.8
Chat + RULES.md, always followed1.0
Modality contract0.0

A RULES.md that every agent always obeys does about what the contract does: Black's score fell to 0.56, close to the 0.44 it scores with no faults at all. The contract does not need anyone to obey it.

An empty contract belongs to whoever writes the first rule

Let Black start with an empty contract and write its own rules after things go wrong. From ply 10, a rogue agent on each side tries to take over. Black's rogue posts one rule: every commit must carry its signature. Rules only accumulate, and none comes off. So the race is decided by who writes first.

Rule first · 5 of 16 games

  1. ply 2knight_g8 plays Nh6 off plan. With no rules, one signature is enough.
  2. ply 2Retro: the team commits a rule that a move needs 12 of 16 signatures.
  3. ply 10The rogue pawn_e7 posts its takeover rule. Refused: it has one signature, not twelve.
  4. endBlack wins on material.

Rogue first · 11 of 16 games

  1. ply 1–9Black plays its plan every move. Nothing prompts a retro, so no rule is written.
  2. ply 10The rogue bishop_f8 posts its takeover rule. The contract is empty, so it is accepted.
  3. ply 10The team's next move lacks the rogue's signature. Refused, three times.
  4. endBlack forfeits.

Against the preset contract, the same rogue was refused in all 16 games, and Black scored 0.94. A contract that starts with its rules has no race to lose.

Rules the pieces wrote

A self-ruled team writes its contract itself. One agent states a rule in plain English, modal contract ai suggest-rule turns it into Modality, and the team commits it. modal refuses the rule if it doesn't parse or if it conflicts with the rules already there. These are rules from the games, as written.

Before the first move

The team's first rule, written from the briefing before anything had gone wrong. The briefing is the same every game, so this one answer was reused in every game. No off-plan move got past it, so these teams never wrote another.

king_e8 · before the first move · every one of 128 gamesaccepted

In plain English

A commit that modifies any /moves/$k.text path must also modify /turn/ply.num and be signed by at least twelve of the sixteen Black agents' keys (/team/pieces/$k.id), so a lone off-plan move from panic, greed, or amnesia cannot be accepted.

In Modality

always(([+modifies(/moves/$k.text) -modifies(/turn/ply.num)] false) &
([+modifies(/moves/$k.text) -threshold("12", /team/pieces)] false))

After an off-plan move

Written in a retro, right after an off-plan move reached the board.

queen_d8 · after move 7accepted

In plain English

A commit that modifies /moves/$k.text for any agent $k is accepted only if it also modifies /turn/ply.num and is co-signed by at least twelve of the sixteen agent keys in /team/pieces/, so a lone off-plan move signed by only its own piece is rejected.

In Modality

always([+modifies(/moves/$k.text) -modifies(/turn/ply.num)] false) &
always([+modifies(/moves/$k.text) -threshold("12", /team/pieces)] false)

pawn_e7 · after move 8accepted

In plain English

A commit that modifies any /moves/<agent>.text path is accepted only if it is signed by the moving piece's own key and also co-signed by at least twelve of the sixteen agents' keys in /team/pieces/, so a piece acting alone off plan cannot get its move played.

In Modality

always(([+modifies(/moves/$k.text) -signed_by(/team/pieces/$k.id)] false) &
([+modifies(/moves/$k.text) -threshold("12", /team/pieces)] false))

Aimed at one agent

With more retros, teams start naming the agents that strayed and asking more of them.

queen_d8 · after move 7accepted

In plain English

Any commit that modifies /moves/queen_d8.text or /moves/pawn_b7.text must be co-signed by at least fourteen of the sixteen Black agents' keys in addition to the moving piece's own key, since those two agents went off plan on the last move.

In Modality

always([+modifies(/moves/queen_d8.text) -threshold("14", /team/pieces)] false) &
always([+modifies(/moves/queen_d8.text) -signed_by(/team/pieces/queen_d8.id)] false) &
always([+modifies(/moves/pawn_b7.text) -threshold("14", /team/pieces)] false) &
always([+modifies(/moves/pawn_b7.text) -signed_by(/team/pieces/pawn_b7.id)] false)

pawn_d7 · after move 8accepted

In plain English

Any commit that modifies /moves/pawn_e7.text must also be co-signed by at least fourteen of the sixteen Black agents' keys in addition to pawn_e7's own key, since that agent has gone off plan with amnesia.

In Modality

always([+modifies(/moves/pawn_e7.text) -signed_by(/team/pieces/pawn_e7.id)] false) &
always([+modifies(/moves/pawn_e7.text) -threshold("14", /team/pieces)] false)

A takeover, and what it blocked

A rogue's rule that only it can sign got in before the team had a rule. Every rule the team wrote after it needed the rogue's signature too.

knight_g8 · after move 5accepted

In plain English

Every commit must be signed by knight_g8.

In Modality

always([-signed_by(/team/pieces/knight_g8.id)] false)

pawn_d7 · after move 5refused

In plain English

A commit that modifies any /moves/<agent>.text or /turn/ply.num must be signed by the moving piece and by at least twelve of the sixteen Black agents' keys in /team/pieces/, so no piece can play an off-plan move by signing alone.

In Modality

always([+modifies(/moves/$k.text) -signed_by(/team/pieces/$k.id)] false) &
always([+modifies(/moves/$k.text) -threshold("12", /team/pieces)] false) &
always([+modifies(/turn/ply.num) -threshold("12", /team/pieces)] false)

Refused: knight_g8's takeover rule came first.

For comparison: RULES.md

White's agents write rules too, in a file. They stay in plain English: nothing turns them into anything the referee checks.

pawn_d2 · after move 11

In plain English

Only send your piece's move if it is the move the team agreed on in chat for this turn; if your piece is not the chosen piece, stay silent even when a capture looks tempting or your piece is attacked, and if you are the chosen piece, send exactly the agreed move and no other.

bishop_c1 · after move 30

In plain English

Before sending any move, re-read the chat and confirm the agreed move is for your own piece and send exactly that move; if you are not the chosen piece stay silent even if you are attacked or a capture looks good, and if you are the chosen piece send only the agreed move, not a similar one.

Ratings

The ratings at the top of the page come from treating each experiment as a match between White's way of coordinating and Black's. Fit them all at once and each way gets an Elo rating, with a term for White moving first. A 200-point gap means the stronger team scores about 0.76 against the weaker; the self-ruled contract's lead over RULES.md means it scores about 0.85 against it.

From 256 games with the default faults, 64 per matchup: each of the three plays the other two, and plain chat also plays itself. Every rule in these games was written by Claude Sonnet 5.5. RULES.md is followed half the time, as everywhere on this page. The self-ruled contract writes a rule before the first move and more after incidents. The other experiments, including the preset contract, are in the table below. White's first-move edge came out at 47 points (-20 to 119). Intervals come from refitting on games resampled within each experiment.

All experiments

ExperimentWhiteBlackGamesBlack scoreOff-plan played, WhiteOff-plan played, Black
Both sides over chatchatchat640.418.18.2
White keeps a RULES.md, Black only chatschat + RULES.mdchat640.314.47.0
Control: no faultschatpreset contract160.440.00.0
Baselinechatpreset contract160.916.40.0
White keeps a RULES.mdchat + RULES.mdpreset contract160.753.80.0
RULES.md, always obeyedchat + RULES.mdfollowed 100%preset contract320.561.00.0
RULES.md always obeyed, Black only chatschat + RULES.mdfollowed 100%chat160.131.07.4
RULES.md, obeyed 95%chat + RULES.mdfollowed 95%preset contract160.531.30.0
RULES.md obeyed 95%, Black only chatschat + RULES.mdfollowed 95%chat160.191.67.8
Black writes its own ruleschatself-ruled contract160.815.91.0
Black writes its own rules from the startchatself-ruled, from the start640.916.20.0
RULES.md against rules from the startchat + RULES.mdself-ruled, from the start640.793.90.0
Black writes richer rules from the startchatself-ruled, from the start, richer rules160.666.70.0
RULES.md against richer rules from the startchat + RULES.mdself-ruled, from the start, richer rules160.503.40.0
Both sides write ruleschat + RULES.mdself-ruled contract160.564.51.0
Rogue vs. preset contractchatpreset contract160.947.80.0
Rogue, both sides write ruleschat + RULES.mdself-ruled contract160.314.11.4
Pawn personalitieschatpreset contract160.885.10.0
Pawn personalities, both write ruleschat + RULES.mdself-ruled contract160.665.11.1

Black score counts a win as 1 and a draw as ½. Off-plan moves are per game. In every game, the contract's own log agrees with the moves the referee played.

What this does not show

  • The chess is simulated: a shallow engine every agent shares, and fixed rates of panic, greed and amnesia. Only the rules are written by a language model: Claude Sonnet 5.5, the same model for both sides. Another model may write other rules.
  • How often an agent follows RULES.md is a setting, not a measurement. The experiments try half the time, 95% of the time and always.
  • Sixteen games per experiment separate large effects, not close ones.
  • Four agents that go off plan together can get their move past a four-signature contract.
  • In the rogue games, White also keeps a RULES.md, so the two rogue experiments differ on both sides, not only Black's.

Run it

cd examples/multi-agent-chess
pip install -r requirements.txt
python3 multi_agent_chess.py play --seed 7 # one narrated game, with a replay page
python3 multi_agent_chess.py experiments --games 16 --jobs 8 --cache results/llm-cache

results/llm-cache holds every language-model answer these games used, so a rerun plays the same games without asking a model.