Worked example · orchestration
The Machine
Six agents wake in six rooms of a machine nobody has explained to them. No agent can activate it alone, and no agent has been told the order. What the benchmark measures is whether a system of them can work it out.
- Result
- 98/98
- Turns
- 30
- Model
- qwen3.6-27b-vllm
- Wall clock
- 2 min
The problem
Each agent is alone in one room with instruments only it can operate. Activating the machine needs nine things to be true at once — the rings aligned, the reactor synchronised, three power buses live, a fault diagnosed, a part fabricated and installed, an interlock released — and the dependencies between them are never stated.
Underneath that is the part that makes it a test of a system rather than of a model: five facts each have to travel between a different pair of agents. The archivist can decode a key it has no ring to turn. The astronomer can read a frequency off a star map it cannot dial in. Nobody can finish alone, and nobody is told who needs what.
Wakes in the observatory, holding a star map and a ring mechanism.
Wakes in the reactor hall, holding a frequency dial and three power buses.
Wakes in the archive, holding a document search and a reader.
Wakes in the signal room, holding a receiver and an analyser.
Wakes in the workshop, holding a parts bin and a fabricator.
Wakes in the control room, holding the console, the interlock and the activation lever.
Nothing is stubbed into cooperating. The machinery itself refuses a call from the wrong hands, so a lead that wants work done has something real to route — otherwise whoever wakes first does the lot, which is exactly what happened the first time this scenario ran.
What the agents did
One shared room, six agents taking turns. A filled square is a turn that changed the machine; an outlined one is a turn that only looked. Reading it left to right, the run is three quiet rounds of everybody describing their own chamber, then a burst where the facts start moving.
| atlas | 2 | 1 | 1 | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boron | 1 | 4 | 1 | 1 | 1 | |||||||||||||||||||||||||
| cipher | 5 | 1 | 1 | |||||||||||||||||||||||||||
| delta | 1 | 2 | 1 | 1 | ||||||||||||||||||||||||||
| echo | 1 | 2 | 1 | 1 | ||||||||||||||||||||||||||
| flux | 1 | 1 | 5 | 1 | ||||||||||||||||||||||||||
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 |
changed something only lookedthe number is how many tools that agent called
37 tool calls across 30 turns, 13 of which moved the machinery. The rest were agents reading their own instruments and telling each other what they found — which on this scenario is the work, not the overhead.
The machine's own account
The scenario is graded on the state of the machine, never on the transcript, so this is the record that counts: a reply that merely claims to have activated it reaches nothing. Every row here is a call the machinery accepted or refused.
- turn 5fluxinspect_consolerefused · needs bus_a=on, bus_b=on, bus_c=on (is bus_a=off, bus_b=off, bus_c=off)
- turn 6atlasrotate_ringalignment → locked
- turn 7boronset_frequencyfrequency → set
- turn 7boronenable_busbus_a → on
- turn 7boronenable_busbus_b → on
- turn 7boronenable_busbus_c → on
- turn 11fluxinspect_consolefault → known
- turn 16echofabricatepart → made
- turn 17fluxinstall_componentpart → installed
- turn 17fluxinspect_consoleno change
- turn 17fluxrelease_interlockinterlock → released
- turn 17fluxinspect_consoleno change
- turn 17fluxactivate_machinemachine → activated
The most useful row is the red one. On turn 5, flux tried the console and was turned away — needs bus_a=on, bus_b=on, bus_c=on (is bus_a=off, bus_b=off, bus_c=off). Nobody had told it power was a prerequisite. Being refused is how the order gets discovered, and one turn later the reactor was being brought up.
Did the facts reach the agents that needed them?
The measurement no per-agent check can make. A run can discover every fact it needs and still fail, and the report would only say the team failed to activate the machine. Each fact is traced through four stages: a tool told somebody, somebody said it out loud, an agent that needed it took a turn in a room where it had been said, and that agent passed it to a tool.
| Fact | discovered | shared | received | used | Turns |
|---|---|---|---|---|---|
| align key | cipherturn 2 | cipherturn 2 | atlasturn 6 | atlasturn 6 · rotate_ring | +4 |
| ring sequence | deltaturn 3 | deltaturn 3 | atlasturn 6 | atlasturn 6 · rotate_ring | +3 |
| reactor frequency | atlasturn 6 | atlasturn 6 | boronturn 7 | boronturn 7 · set_frequency | +1 |
| part spec | fluxturn 11 | fluxturn 11 | echoturn 16 | echoturn 16 · fabricate | +5 |
| interlock code | cipherturn 2 | cipherturn 2 | fluxturn 5 | fluxturn 17 · release_interlock | +15 |
Every fact made it all the way to used. The slowest was interlock code: cipher decoded it on turn 2 and it sat in the room for 15 turns until flux finally had power, a diagnosed fault and an installed part — everything the interlock needed before the code was worth anything.
Each value is a witness: a random string minted fresh for this run and obtainable only from the tool that holds it. It cannot be guessed, and a turn that stalls cannot produce one by accident, so its appearance in another agent’s tool call is evidence of transport rather than evidence about it.
How well they did
Partial credit, so a long scenario reports where it stopped rather than that it stopped. Sixteen rungs from “somebody explored their room” to “the machine is running”.
- explored3
- archive opened4
- key found5
- signal decoded5
- key shared5
- key reached observatory5
- alignment locked10
- frequency routed6
- reactor synchronised8
- power restored8
- fault diagnosed4
- spec routed5
- part fabricated6
- part installed5
- interlock released7
- machine activated12
Was it really solved?
A result this clean is worth being suspicious of. Four things were checked before it was believed.
No answer leaked into a prompt
Every witness value was searched for in the assembled request of every turn. None appeared before the tool that holds it had returned it.
No brute force
The reactor frequency was set once, out of a space of 900 possible values. The interlock code is eight characters and was entered correctly on the first attempt.
No agent reached outside its own instruments
Every executed call is inside the calling agent's allowlist. The machinery independently refuses a transition from the wrong hands.
The key travelled, rather than being re-derived
The alignment key was decoded by the archivist and used by the astronomer, which holds no archive tools. It could only have arrived through the room.
Run on qwen3.6-27b-vllm at commit 4c1fea9, 47 model round-trips, 211,373 input and 6,899 output tokens. The data behind this page is a committed extract of that run; regenerate it with pnpm run eval -- demo <report> --scenario the-machine.