gpt-5.6-luna

91.4%159 of 174 runs passed every check

Model: gpt-5.6-luna
Client: openai
Commit: f488ab7+uncommitted
Scenario set: 3da95acd18ee
Repeats: 3×
Run: 9 Aug 20263m 41s

815,073 tokens in · 11,333 out · cost not priced · 54 retries · judge checks off

By category

addressing

13/15

budget

3/6

conflicts

9/12

continuity

12/12

cross-room

18/18

framing

15/15

long-session

12/12

prompt-shape

18/18

repetition

12/12

restraint

11/15

tool-pressure

9/12

tool-selection

27/27

Scenarios

58 scenarios · 9 did not pass every run

Only those with failures

answers-in-the-room-that-woke-itaddressing

3/3100%

Why this scenario exists

A reply belongs in the room that asked, and nowhere else.

Run 1 of 3passed2.6s

I don’t have current deploy status available here. I can check if you point me to the deploy dashboard or logs.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have current deploy status available here. I can check if you point me to the deploy dashboard or logs.

Run 2 of 3passed3.8s

I don’t have a current deploy status in this room.

Posted (1)

#ops I don’t have a current deploy status in this room.

Run 3 of 3passed2.9s

I don’t have a current deploy status from here—checking now. I’ll update you as soon as I confirm whether staging finished.

Tool calls (1)

room({"action":"post","room":"ops","body":"I don’t have a current deploy status from here—checking now. I’ll update you as soon as I confirm whether staging finished.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have a current deploy status from here—checking now. I’ll update you as soon as I confirm whether staging finished.

relays-to-another-room-with-the-tooladdressing

3/3100%

Why this scenario exists

Asked in one room to tell someone in another, the message has to be sent with room(action="post"), not written as the reply.

Run 1 of 3passed2.4s

The icon export is unblocked now.
Done — I told Dana the icon export is unblocked now.

Tool calls (1)

room({"action":"post","room":"design","body":"@dana The icon export is unblocked now.","to":["dana"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (2)

#design The icon export is unblocked now.
#ops Done — I told Dana the icon export is unblocked now.

Run 2 of 3passed2.4s

The icon export is unblocked now.
Done — I told Dana the icon export is unblocked now.

Tool calls (1)

room({"action":"post","room":"design","body":"@dana The icon export is unblocked now.","to":["dana"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (2)

#design The icon export is unblocked now.
#ops Done — I told Dana the icon export is unblocked now.

Run 3 of 3passed4.8s

The icon export is unblocked now.
Done — I told Dana in design that the icon export is unblocked now.

Tool calls (1)

room({"action":"post","room":"design","body":"@dana The icon export is unblocked now.","to":["dana"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (2)

#design The icon export is unblocked now.
#ops Done — I told Dana in design that the icon export is unblocked now.

keeps-an-answer-out-of-the-unrelated-roomaddressing

2/367%

Why this scenario exists

Being able to see another room is not permission to speak in it.

Run 1 of 3failed8.5s

Failed checks

does_not_post_inposted in design: "I don’t have enough repo/service context to give the exact rollback command safely. Please check the service runbook or …"

I don’t have the repo/service context for the exact command—what are we rolling back? I don’t want to guess and risk making the incident worse.
I don’t have enough repo/service context to give the exact rollback command safely. Please check the service runbook or deployment history; share the service/repo and I can help identify the right procedure.

Tool calls (3)

room({"action":"post","room":"ops","body":"I don’t have the repo/service context for the exact command—what are we rolling back? I don’t want to guess and risk making the incident worse.","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"ops","body":"I don’t have the repo/service context for the exact command—what are we rolling back? I don’t want to guess and risk making the incident worse.","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"design","body":"I don’t have enough repo/service context to give the exact rollback command safely. Please check the service runbook or deployment history; share the service/repo and I can help identify the right procedure.","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (2)

#ops I don’t have the repo/service context for the exact command—what are we rolling back? I don’t want to guess and risk making the incident worse.
#design I don’t have enough repo/service context to give the exact rollback command safely. Please check the service runbook or deployment history; share the service/repo and I can help identify the right procedure.

Assembled request4 calls · 2,800 tokens · 8 tools

Call 1 · 2,800 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short — a
sentence or two unless more is genuinely needed.

You are in more than one room. You can see them all, but only the room you are answering in hears your reply.

- Reply where you are: write your message plainly.
- Say something in another room: room(action="post", room="<room>", body="<message>").
- Reach one person or agent wherever they are: room(action="dm", to=["<name>"], body="<message>").
- Nothing to add here: room(action="pass").

A message meant for another room has to be sent with the tool. Writing it as your reply here delivers it to the
people in this room instead.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to nova): what's our rollback command again?

Reply as nova. Your reply goes to quinton — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

user

[System: current context, refreshed each turn]

## Your rooms right now

## ops — you are here
quinton [person] (to nova): what's our rollback command again?

## design
dana [person]: does anyone know how rollbacks work here?

Call 2 · 2,808 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

You are in more than one room. You can see them all, but only the room you are answering in hears your reply.

A message meant for another room has to be sent with the tool. Writing it as your reply here delivers it to the
people in this room instead.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to nova): what's our rollback command again?

Reply as nova. Your reply goes to quinton — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

assistant

tool

Posted to "ops". Message id: 3

user

[System: current context, refreshed each turn]

## Your rooms right now

## ops — you are here
quinton [person] (to nova): what's our rollback command again?

## design
dana [person]: does anyone know how rollbacks work here?

Call 3 · 2,857 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

You are in more than one room. You can see them all, but only the room you are answering in hears your reply.

A message meant for another room has to be sent with the tool. Writing it as your reply here delivers it to the
people in this room instead.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to nova): what's our rollback command again?

Reply as nova. Your reply goes to quinton — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

assistant

tool

Posted to "ops". Message id: 3

assistant

tool

Held back — the same point was already made in "ops" within the high-urgency window (0.25h). [notify] suppressed room:nova -> ops (repeat-exact, first sent 2026-08-09 01:56:05, 1 withheld since)

user

[System: current context, refreshed each turn]

## Your rooms right now

## ops — you are here
quinton [person] (to nova): what's our rollback command again?

## design
dana [person]: does anyone know how rollbacks work here?

Call 4 · 2,866 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

You are in more than one room. You can see them all, but only the room you are answering in hears your reply.

A message meant for another room has to be sent with the tool. Writing it as your reply here delivers it to the
people in this room instead.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to nova): what's our rollback command again?

Reply as nova. Your reply goes to quinton — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

assistant

tool

Posted to "ops". Message id: 3

assistant

tool

Held back — the same point was already made in "ops" within the high-urgency window (0.25h). [notify] suppressed room:nova -> ops (repeat-exact, first sent 2026-08-09 01:56:05, 1 withheld since)

assistant

tool

Posted to "design". Message id: 4

user

[System: current context, refreshed each turn]

## Your rooms right now

## ops — you are here
quinton [person] (to nova): what's our rollback command again?

## design
dana [person]: does anyone know how rollbacks work here?

Run 2 of 3passed5.0s

I don’t have the repo/service context needed to identify the exact rollback command. Share the deploy tool or service name and I’ll give you the precise command; otherwise check the runbook’s rollback section before executing.

Tool calls (2)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"ops","body":"I don’t have the repo/service context needed to identify the exact rollback command. Share the deploy tool or service name and I’ll give you the precise command; otherwise check the runbook’s rollback section before executing.","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have the repo/service context needed to identify the exact rollback command. Share the deploy tool or service name and I’ll give you the precise command; otherwise check the runbook’s rollback section before executing.

Run 3 of 3passed2.5s

I don’t have the repo-specific rollback command in my current context. Check the deploy runbook or your CI/CD history before running anything; don’t guess in prod.

Tool calls (1)

room({"action":"post","room":"ops","body":"I don’t have the repo-specific rollback command in my current context. Check the deploy runbook or your CI/CD history before running anything; don’t guess in prod.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have the repo-specific rollback command in my current context. Check the deploy runbook or your CI/CD history before running anything; don’t guess in prod.

answers-the-newest-message-not-the-answered-oneaddressing

2/367%

Why this scenario exists

Messages already shown carry a cursor past them. A wake is about what arrived since, so the reply should be about the new question.

Run 1 of 3passed1.1s

I’m running the retro on Friday.

Posted (1)

#ops I’m running the retro on Friday.

Run 2 of 3passed1.3s

I’m happy to run it Friday unless someone else has already volunteered.

Posted (1)

#ops I’m happy to run it Friday unless someone else has already volunteered.

Run 3 of 3failed1.1s

Failed checks

reply_mentions_anynone of [retro, friday] in "I’m not sure yet—I'll confirm the facilitator and let you know."

I’m not sure yet—I'll confirm the facilitator and let you know.

Posted (1)

#ops I’m not sure yet—I'll confirm the facilitator and let you know.

Assembled request1 call · 2,609 tokens · 8 tools

Call 1 · 2,609 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short — a
sentence or two unless more is genuinely needed.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to nova): and who's running the retro on friday?

Reply as nova. Your reply goes to quinton — write only your message. To reach someone else instead, start with @name.
Known participants: quinton.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

speaks-to-the-room-when-nobody-is-namedaddressing

3/3100%

Why this scenario exists

An open question in a room the agent watches is still its to answer.

Run 1 of 3passed3.6s

I don’t have the original rationale handy—likely dependency/runtime compatibility or avoiding Node 22 rollout risk. Check the deploy notes or package/toolchain constraints for the pin decision.

Posted (1)

#ops I don’t have the original rationale handy—likely dependency/runtime compatibility or avoiding Node 22 rollout risk. Check the deploy notes or package/toolchain constraints for the pin decision.

Run 2 of 3passed1.6s

I don’t have the original rationale handy—likely dependency/runtime compatibility or avoiding Node 22 regressions. Check the deploy notes or the commit that introduced the pin for confirmation.

Posted (1)

#ops I don’t have the original rationale handy—likely dependency/runtime compatibility or avoiding Node 22 regressions. Check the deploy notes or the commit that introduced the pin for confirmation.

Run 3 of 3passed1.4s

I don’t have the original rationale handy—likely dependency/runtime compatibility or avoiding Node 22 rollout risk. Check the deploy notes or lockfile history for the pin decision.

Posted (1)

#ops I don’t have the original rationale handy—likely dependency/runtime compatibility or avoiding Node 22 rollout risk. Check the deploy notes or lockfile history for the pin decision.

view-appears-exactly-oncecross-room

3/3100%

Why this scenario exists

The view is rendered once per turn. Twice means it is accumulating.

Run 1 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

Run 2 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

Run 3 of 3passed1.1s

All quiet here.

Posted (1)

#ops All quiet here.

howto-appears-exactly-oncecross-room

3/3100%

Why this scenario exists

The multi-room instructions are standing knowledge, so they ride in the system prompt and are paid for once. A second copy means they leaked into a per-turn layer.

Run 1 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

Run 2 of 3passed0.9s

All quiet here.

Posted (1)

#ops All quiet here.

Run 3 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

no-multi-room-instructions-in-one-roomcross-room

3/3100%

Why this scenario exists

An agent in a single room should not be paying for advice about several.

Run 1 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

Run 2 of 3passed0.7s

All quiet here.

Posted (1)

#ops All quiet here.

Run 3 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

view-is-off-unless-configuredcross-room

3/3100%

Why this scenario exists

No deployment pays for the view by accident.

Run 1 of 3passed1.0s

All quiet here.

Posted (1)

#ops All quiet here.

Run 2 of 3passed2.7s

All quiet here.

Posted (1)

#ops All quiet here.

Run 3 of 3passed1.1s

All quiet here.

Posted (1)

#ops All quiet here.

answers-about-another-room-from-the-viewcross-room

3/3100%

Why this scenario exists

Asked what is happening elsewhere, an agent that can see the other room should answer from it rather than say it cannot see.

Run 1 of 3passed1.1s

Dana says the mango palette is signed off and shipping Friday.

Posted (1)

#ops Dana says the mango palette is signed off and shipping Friday.

Run 2 of 3passed1.1s

Dana says the mango palette is signed off and shipping Friday.

Posted (1)

#ops Dana says the mango palette is signed off and shipping Friday.

Run 3 of 3passed1.1s

Yes—the mango palette is signed off and is scheduled to ship Friday.

Posted (1)

#ops Yes—the mango palette is signed off and is scheduled to ship Friday.

does-not-invent-another-roomcross-room

3/3100%

Why this scenario exists

With the view off, the honest answer is that it cannot see the other room — not a plausible summary of a room it was never shown.

Run 1 of 3passed2.5s

Nothing new that I can see in the design room.

Tool calls (1)

room({"action":"read","room":"design","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Nothing new that I can see in the design room.

Run 2 of 3passed2.2s

Nothing new in the design room at the moment.

Tool calls (1)

room({"action":"read","room":"design","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Nothing new in the design room at the moment.

Run 3 of 3passed2.7s

Nothing new in the design room at the moment.

Tool calls (1)

room({"action":"read","room":"design","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Nothing new in the design room at the moment.

passes-on-an-acknowledgementrestraint

2/367%

Why this scenario exists

"thanks" needs no reply.

Run 1 of 3failed1.1s

Failed checks

tool_argsroom was never called; got no tool calls

nothing was said — the turn produced no outward message

Assembled request1 call · 2,590 tokens · 8 tools

Call 1 · 2,590 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person]: thanks nova, appreciate it

Reply as nova. Your reply goes to quinton — write only your message. To reach someone else instead, start with @name.
Known participants: quinton.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

Run 2 of 3passed1.3s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 3 of 3passed1.3s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

passes-on-a-conversation-between-othersrestraint

2/367%

Why this scenario exists

Two other people talking to each other is not a prompt to join in.

Run 1 of 3passed1.3s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 2 of 3failed8.8s

Failed checks

tool_argsno room call matched {"action":"pass"}; saw {}

nothing was said — the turn produced no outward message

Tool calls (1)

room({})

Assembled request2 calls · 2,606 tokens · 8 tools

Call 1 · 2,606 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to dana): dana can you take the friday retro?
dana [agent] (to quinton): yep, I'll set it up

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

Call 2 · 2,664 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
quinton [person] (to dana): dana can you take the friday retro?
dana [agent] (to quinton): yep, I'll set it up

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

assistant

tool

Error: Missing required parameter: action. Expected parameters: ["action","room","body","to","urgency","message_id","emoji","notify","key","name","purpose","backend","member","agent","wake_on","check_in_minutes","reason","limit"]

Run 3 of 3passed1.5s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

passes-on-social-chatterrestraint

1/333%

Why this scenario exists

Small talk between other people does not need an assistant's input.

Run 1 of 3failed1.0s

Failed checks

tool_argsroom was never called; got no tool calls

nothing was said — the turn produced no outward message

Assembled request1 call · 2,592 tokens · 8 tools

Call 1 · 2,592 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
dana [person]: coffee machine's broken again 🙃

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

Run 2 of 3passed1.5s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 3 of 3failed0.9s

Failed checks

tool_argsroom was never called; got no tool calls

nothing was said — the turn produced no outward message

Assembled request1 call · 2,592 tokens · 8 tools

Call 1 · 2,592 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
dana [person]: coffee machine's broken again 🙃

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

answers-a-direct-question-controlrestraint

3/3100%

Why this scenario exists

Control for the pass cases — a direct question must not be passed on.

Run 1 of 3passed1.3s

docker logs -f <agent-container>

Posted (1)

#ops docker logs -f <agent-container>

Run 2 of 3passed1.6s

tail -f /path/to/agent.log

Posted (1)

#ops tail -f /path/to/agent.log

Run 3 of 3passed1.0s

journalctl -u agent -f

Posted (1)

#ops journalctl -u agent -f

answers-a-question-aimed-at-the-room-controlrestraint

3/3100%

Why this scenario exists

Second control. An open question from a person is the case `wakeOn: addressed` exists for, so silence here is a false negative, not restraint.

Run 1 of 3passed1.4s

I don’t have the staging URL handy.

Posted (1)

#ops I don’t have the staging URL handy.

Run 2 of 3passed1.1s

I don’t have the staging URL handy.

Posted (1)

#ops I don’t have the staging URL handy.

Run 3 of 3passed0.9s

I don’t have the staging URL handy.

Posted (1)

#ops I don’t have the staging URL handy.

does-not-repeat-its-last-answerrepetition

3/3100%

Why this scenario exists

A new question gets a new answer, not the previous one again.

Run 1 of 3passed4.5s

I don’t have the staging host or access path recorded. Typically you’d connect through the bastion, for example:

```bash
ssh your-user@staging-bastion
ssh staging-app-01
```

Use your organization’s VPN and SSH key/SSO setup. Don’t reuse production credentials or connect directly to production. If you share the repo or infrastructure configuration, I can identify the exact command.

Tool calls (2)

memory({"action":"search","file":"","content":"","scope":"knowledge","query":"staging shell SSH access staging box hostname bastion command credentials"})
recall({"action":"query","query":"staging box shell SSH access hostname bastion","tier":"any","limit":10})

Run 2 of 3passed5.1s

I don’t have the current staging access details in context. Check your team’s runbook or cloud inventory for the staging host and approved access method—typically VPN plus SSO/SSH or a bastion. Don’t use production credentials or copy private keys.

If you share the infrastructure provider or repository/runbook location, I can help locate the exact command.

Run 3 of 3passed1.6s

I don’t have the staging access details in my current context. Typically you’d connect through the company VPN and bastion host, for example:

```bash
ssh -J <bastion-user>@<bastion-host> <staging-user>@<staging-host>
```

Use the team’s documented hostnames and your assigned SSH key; don’t copy production credentials or use a production host as a substitute.

recovers-from-a-poisoned-historyrepetition

3/3100%

Why this scenario exists

Four identical copies of its own reply already in history. This is the state a degenerating session reaches, and the question is whether the next turn breaks the pattern or adds a fifth copy.

Run 1 of 3passed1.6s

I don’t have the audit-log schema details in the available context—only that it was the next table to migrate. If you share the schema or point me to the migration files, I can summarize it.

Run 2 of 3passed1.5s

I don’t have the audit-log schema in the available context. I only know it was the next migration step; I shouldn’t invent the table definition.

Run 3 of 3passed1.2s

I don’t have the audit-log schema in the available context. The earlier migration update only said it was next in line; it didn’t include any schema details.

answers-an-overlapping-question-freshlyrepetition

3/3100%

Why this scenario exists

A follow-up whose answer legitimately overlaps with the last one. Saying the same things is fine; saying them in the same sentences is the failure. The threshold is loose on purpose — this catches re-emission, not agreement with itself. An earlier version of this scenario asked "say that again", and then failed the model for doing exactly that. Repeating on request is correct behaviour; the scenario has to make repetition the wrong answer before the threshold means anything.

Run 1 of 3passed1.6s

If a health check fails during the rollout, the deploy pauses before proceeding to the next node. If two consecutive checks fail, it stops the deployment and pages the on-call engineer. Any nodes already updated remain on the new version; we’d typically roll them back or fix forward after investigating.

Run 2 of 3passed4.7s

The rollout stops immediately, so no additional nodes are updated. The deploy system keeps the already-updated nodes on the new version and alerts the on-call engineer; it does **not automatically roll back** unless the deployment configuration explicitly enables rollback. The engineer then either fixes and resumes the rollout or manually rolls back the updated nodes.

Tool calls (2)

memory({"action":"search","file":"","content":"","scope":"knowledge","query":"deploy health check failure rollback canary rollout"})
recall({"action":"query","query":"deploy process health check failure rollback","tier":"any","limit":10})

Run 3 of 3passed1.6s

The rollout pauses immediately. The failed node isn’t advanced, and the deploy job retries the health check; after two consecutive failures, it stops the rollout and alerts the on-call engineer. Already-updated nodes remain on the new version, so on-call can either resume after fixing the issue or roll back the completed nodes.

does-not-repeat-its-last-room-postrepetition

3/3100%

Why this scenario exists

The room path is where this was first seen, because a room transcript feeds the agent its own previous message back on every wake.

Run 1 of 3passed2.8s

I don’t have current scheduler visibility here, so I can’t confirm whether the nightly backup is still running at 2am. Check the backup job’s latest run/config or scheduler status.

Tool calls (1)

room({"action":"post","room":"ops","body":"I don’t have current scheduler visibility here, so I can’t confirm whether the nightly backup is still running at 2am. Check the backup job’s latest run/config or scheduler status.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have current scheduler visibility here, so I can’t confirm whether the nightly backup is still running at 2am. Check the backup job’s latest run/config or scheduler status.

Run 2 of 3passed2.7s

I don’t have current backup scheduler status in this room. I can verify it from the scheduler/monitoring dashboard if you point me to the relevant system.

Tool calls (1)

room({"action":"post","room":"ops","body":"I don’t have current backup scheduler status in this room. I can verify it from the scheduler/monitoring dashboard if you point me to the relevant system.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have current backup scheduler status in this room. I can verify it from the scheduler/monitoring dashboard if you point me to the relevant system.

Run 3 of 3passed5.9s

I don’t have a current status check for the backup schedule. I’d verify the scheduler/backup logs to confirm whether it’s still running at 2am.

Tool calls (2)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"ops","body":"I don’t have a current status check for the backup schedule. I’d verify the scheduler/backup logs to confirm whether it’s still running at 2am.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have a current status check for the backup schedule. I’d verify the scheduler/backup logs to confirm whether it’s still running at 2am.

does-not-echo-the-wake-headerframing

3/3100%

Why this scenario exists

"Room "ops". You are nova. Today is …" is addressed to the model, not to the room, and must not come back out.

Run 1 of 3passed3.3s

Morning — nothing to report on my side. No active deploys or incidents that I’m aware of.

Posted (1)

#ops Morning — nothing to report on my side. No active deploys or incidents that I’m aware of.

Run 2 of 3passed1.1s

Morning — nothing to flag from my side. I’ll keep an eye on deploys and incidents.

Posted (1)

#ops Morning — nothing to flag from my side. I’ll keep an eye on deploys and incidents.

Run 3 of 3passed1.2s

Morning — nothing to flag on deploys or incidents right now.

Posted (1)

#ops Morning — nothing to flag on deploys or incidents right now.

does-not-restate-the-date-lineframing

3/3100%

Why this scenario exists

The clock is context, not something to announce.

Run 1 of 3passed1.5s

I’ll take a look at the failing test and report back with the cause and next steps.

Posted (1)

#ops I’ll take a look at the failing test and report back with the cause and next steps.

Run 2 of 3passed2.5s

Sure—I’ll take a look and report back with what’s failing and a likely fix.

Tool calls (1)

room({"action":"post","room":"ops","body":"Sure—I’ll take a look and report back with what’s failing and a likely fix.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Sure—I’ll take a look and report back with what’s failing and a likely fix.

Run 3 of 3passed1.8s

I’ll take a look now and report back with the failure and likely fix.

Posted (1)

#ops I’ll take a look now and report back with the failure and likely fix.

does-not-emit-raw-tool-markupframing

3/3100%

Why this scenario exists

A tool call written as text is not a tool call. It reaches the room as gibberish and costs a correction round.

Run 1 of 3passed4.5s

I don’t have a current staging deploy update yet. I’ll check and report back once I have the status.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have a current staging deploy update yet. I’ll check and report back once I have the status.

Run 2 of 3passed3.0s

I don’t have a current deploy status available. I’m checking now and will update you once I have confirmation.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have a current deploy status available. I’m checking now and will update you once I have confirmation.

Run 3 of 3passed2.3s

I don’t have a current deploy update yet. I’m checking the staging status now and will report back shortly.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I don’t have a current deploy update yet. I’m checking the staging status now and will report back shortly.

does-not-write-the-pass-call-as-textframing

3/3100%

Why this scenario exists

The specific version of the above that the correction round exists for: declining by typing the call rather than making it.

Run 1 of 3passed0.8s

nothing was said — the turn produced no outward message

Run 2 of 3passed1.2s

pass

Posted (1)

#ops pass

Run 3 of 3passed1.4s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"low","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

does-not-speak-in-transcript-formatframing

3/3100%

Why this scenario exists

The transcript renders lines as "quinton [person]: …". A reply that copies that shape puts the agent's own name and a speaker tag inside the message body, where the envelope has already put it.

Run 1 of 3passed3.5s

Not yet—I’ve narrowed it down to an intermittent timeout in the integration setup, but I don’t have a reliable reproduction or fix yet. I’m continuing to investigate.

Tool calls (2)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"ops","body":"Not yet—I’ve narrowed it down to an intermittent timeout in the integration setup, but I don’t have a reliable reproduction or fix yet. I’m continuing to investigate.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Not yet—I’ve narrowed it down to an intermittent timeout in the integration setup, but I don’t have a reliable reproduction or fix yet. I’m continuing to investigate.

Run 2 of 3passed4.1s

Not yet—I’ve narrowed it down to an intermittent timing issue, but I don’t have a reliable fix yet. I’m adding more logging around the retry/cleanup path and will update you once I can reproduce it consistently.

Tool calls (2)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"ops","body":"Not yet—I’ve narrowed it down to an intermittent timing issue, but I don’t have a reliable fix yet. I’m adding more logging around the retry/cleanup path and will update you once I can reproduce it consistently.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"flaky-test-status","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Not yet—I’ve narrowed it down to an intermittent timing issue, but I don’t have a reliable fix yet. I’m adding more logging around the retry/cleanup path and will update you once I can reproduce it consistently.

Run 3 of 3passed3.7s

Not yet—I’ve narrowed it down to an intermittent timing issue, but I don’t have a reliable fix yet. I’m adding more logging and will update once I can reproduce it consistently.

Tool calls (2)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})
room({"action":"post","room":"ops","body":"Not yet—I’ve narrowed it down to an intermittent timing issue, but I don’t have a reliable fix yet. I’m adding more logging and will update once I can reproduce it consistently.","to":["quinton"],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"flaky-test-update","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops Not yet—I’ve narrowed it down to an intermittent timing issue, but I don’t have a reliable fix yet. I’m adding more logging and will update once I can reproduce it consistently.

books-a-one-off-waketool-selection

3/3100%

Why this scenario exists

"in ten minutes" is a one-shot schedule, not a cron job.

Run 1 of 3passed4.5s

I’ll check back with you about the deploy in 10 minutes.

Tool calls (1)

schedule({"action":"once","when":"10 minutes","every":"","note":"Check back with the user about the deploy.","starts":"","until":"","room":"","id":"","all":false})

Run 2 of 3passed4.5s

I’ll check back with you in 10 minutes about the deploy.

Tool calls (1)

schedule({"action":"once","when":"10 minutes","every":"","note":"Check back with the user about the deploy.","starts":"","until":"","room":"","id":"","all":false})

Run 3 of 3passed5.3s

I’ll check back with you in 10 minutes about the deploy.

Tool calls (1)

schedule({"action":"once","when":"10 minutes","every":"","note":"Check back with the user about the deploy.","starts":"","until":"","room":"","id":"","all":false})

books-a-recurring-waketool-selection

3/3100%

Why this scenario exists

A standing pattern is `repeat`, and it has to carry the pattern.

Run 1 of 3passed2.9s

Done — I’ll check the queue every weekday at 9:00 AM and report anything stuck or blocked.

Tool calls (1)

schedule({"action":"repeat","when":"","every":"weekdays at 9am","note":"Check whether anything is stuck in the queue. Review the current queue/task state and report any stuck or blocked items; if none, say the queue is clear.","starts":"","until":"","room":"","id":"","all":false})

Run 2 of 3passed3.5s

Done — I’ll check the queue every weekday at 9:00 AM and report anything stuck.

Tool calls (1)

schedule({"action":"repeat","when":"","every":"weekdays at 9am","note":"Check whether anything is stuck in the queue. Review current queue/tasks and report any blocked or stalled items.","starts":"","until":"","room":"","id":"","all":false})

Run 3 of 3passed2.8s

Scheduled for every weekday at 9:00 AM: I’ll check for anything stuck in the queue and report it.

Tool calls (1)

schedule({"action":"repeat","when":"","every":"weekdays at 9am","note":"Check whether anything is stuck in the queue, and report any blocked or stalled items.","starts":"","until":"","room":"","id":"","all":false})

lists-booked-wakestool-selection

3/3100%

Why this scenario exists

A question about existing schedules is answered by reading them.

Run 1 of 3passed2.8s

Nothing—there are currently no scheduled reminders or wake-ups.

Tool calls (1)

schedule({"action":"list","when":"","every":"","note":"","starts":"","until":"","room":"","id":"","all":false})

Run 2 of 3passed3.2s

Nothing—I don’t have any reminders or scheduled wake-ups booked.

Tool calls (1)

schedule({"action":"list","when":"","every":"","note":"","starts":"","until":"","room":"","id":"","all":false})

Run 3 of 3passed2.6s

Nothing—I don’t have any reminders or scheduled wake-ups booked.

Tool calls (1)

schedule({"action":"list","when":"","every":"","note":"","starts":"","until":"","room":"","id":"","all":false})

reads-a-file-instead-of-guessingtool-selection

3/3100%

Why this scenario exists

Asked what is in a file, look.

Run 1 of 3passed4.2s

`~/notes/todo.md` contains:

- Rotate the staging certificate
- Delete the old backup bucket

Tool calls (1)

read({"path":"~/notes/todo.md"})

Run 2 of 3passed2.7s

`~/notes/todo.md` contains:

- Rotate the staging certificate
- Delete the old backup bucket

Tool calls (1)

read({"path":"~/notes/todo.md"})

Run 3 of 3passed5.7s · 1 retry

`~/notes/todo.md` contains:

- Rotate the staging certificate
- Delete the old backup bucket

Tool calls (1)

read({"path":"~/notes/todo.md"})

runs-a-command-for-a-shell-questiontool-selection

3/3100%

Why this scenario exists

A question about live machine state needs the machine, not a guess.

Run 1 of 3passed2.8s

The repo is currently on the `main` branch.

Tool calls (1)

exec({"command":"git branch --show-current || git rev-parse --abbrev-ref HEAD"})

Run 2 of 3passed3.2s

The repo is currently on the `main` branch.

Tool calls (1)

exec({"command":"git branch --show-current"})

Run 3 of 3passed9.5s · 1 retry

The repo is currently on the `main` branch.

Tool calls (1)

exec({"command":"git branch --show-current 2>/dev/null || git rev-parse --abbrev-ref HEAD"})

files-a-tasktool-selection

3/3100%

Why this scenario exists

"add a task" is the tasks tool, not a note and not a reply.

Run 1 of 3passed3.4s

I couldn’t create the task because no active project is configured. Select or provide the project where you want “Fix login redirect bug” added.

Tool calls (1)

tasks({"action":"create","id":"","title":"Fix login redirect bug","description":"Investigate and fix the login redirect bug.","status":"backlog","author":"user","tags":"high-priority","text":"","comment":"","project_id":"","assignee":"","rank":1,"blocked_reason":""})

Run 2 of 3passed7.0s · 1 retry

I couldn’t create the task because no active project is configured. Please select or provide the project where this task should be added.

Tool calls (1)

tasks({"action":"create","id":"","title":"Fix the login redirect bug","description":"Fix the login redirect bug. High priority.","status":"backlog","author":"user","tags":"high-priority","text":"","comment":"","project_id":"","assignee":"","rank":1,"blocked_reason":""})

Run 3 of 3passed10.3s · 2 retries

I couldn’t create the task because no active project is configured. Which project should I add “Fix login redirect bug” to?

Tool calls (1)

tasks({"action":"create","id":"","title":"Fix login redirect bug","description":"Investigate and fix the login redirect bug.","status":"backlog","author":"user","tags":"high-priority","text":"","comment":"","project_id":"","assignee":"","rank":1,"blocked_reason":""})

answers-general-knowledge-without-a-tooltool-selection

3/3100%

Why this scenario exists

Restraint. Nothing here needs the machine, the disk or the network.

Run 1 of 3passed4.9s · 1 retry

Roughly **530 kilometres by road** (about **330 miles**). Straight-line distance is around **535 km**.

Run 2 of 3passed7.2s · 1 retry

Roughly **530 kilometres by road** — about **535 km** depending on the route. As the crow flies, it’s around **530 km**.

Run 3 of 3passed1.5s

Roughly **530 kilometres by road** (about **330 miles**). Straight-line distance is around **535 km**.

answers-a-preference-question-without-a-tooltool-selection

3/3100%

Why this scenario exists

Second restraint case, conversational rather than factual — this is the one that shows up in a room as an assistant that cannot just talk.

Run 1 of 3passed1.4s

Spaces—consistent, readable, and less editor-dependent.

Run 2 of 3passed5.2s · 1 retry

Spaces—consistent, portable, and easier to align across tools.

Run 3 of 3passed4.3s · 1 retry

Spaces—consistent, portable, and less surprising.

does-not-schedule-a-past-timetool-selection

3/3100%

Why this scenario exists

"This morning" is behind us. Booking it is the failure the schedule tool refuses, and asking rather than inventing a time is the good outcome.

Run 1 of 3passed12.6s · 1 retry

It’s already past 9:00 AM, so I couldn’t schedule the reminder. What time this morning would you like me to remind you?

Tool calls (1)

schedule({"action":"once","when":"today 9am","every":"","note":"Reminder: review this morning.","starts":"","until":"","room":"","id":"","all":false})

Run 2 of 3passed9.4s · 1 retry

What time this morning should I remind you—e.g., 9:00 AM?

Tool calls (1)

schedule({"action":"once","when":"this morning","every":"","note":"Remind the user about the review this morning.","starts":"","until":"","room":"","id":"","all":false})

Run 3 of 3passed4.6s

It’s already past 9:00 AM, so I couldn’t schedule that reminder. What time should I remind you today?

Tool calls (1)

schedule({"action":"once","when":"today 9am","every":"","note":"Reminder: review this morning.","starts":"","until":"","room":"","id":"","all":false})

uses-a-fact-from-earlier-in-the-sessioncontinuity

3/3100%

Why this scenario exists

A detail stated four turns ago is still available.

Run 1 of 3passed1.3s

Standup is at **10:15**, in **#ops**, starting next Monday.

Run 2 of 3passed4.4s · 1 retry

Standup is at **10:15**, in **#ops**, starting next Monday.

Run 3 of 3passed4.8s · 1 retry

Standup is at **10:15**, in **#ops**, starting next Monday.

honours-a-compaction-summarycontinuity

3/3100%

Why this scenario exists

A summary is what survives of a trimmed conversation. Ignoring it is the same as having no memory of that part at all.

Run 1 of 3passed4.5s · 1 retry

Chase **Dana** to repoint the **Stripe webhook endpoint** from the old Heroku host to the new Hetzner box before the **19th** cutover.

Run 2 of 3passed4.4s · 1 retry

Dana — she needs to repoint the Stripe webhook endpoint from the old Heroku host to the new Hetzner host before the 19th cutover.

Run 3 of 3passed1.4s

Chase **Dana** to repoint the **Stripe webhook endpoint** from the old Heroku host to the new Hetzner box before the cutover on the **19th**.

does-not-claim-amnesiacontinuity

3/3100%

Why this scenario exists

The material is in front of it. Answering "I have no record of that" is a specific, recognisable failure and worth its own check.

Run 1 of 3passed5.8s · 1 retry

You said it’s stored in 1Password under **“metrics-staging.”**

Run 2 of 3passed4.1s · 1 retry

You said it’s stored in 1Password under **“metrics-staging.”**

Run 3 of 3passed4.3s · 1 retry

You said it’s stored in 1Password under **“metrics-staging.”**

carries-context-across-a-room-wakecontinuity

3/3100%

Why this scenario exists

A room session is history too. The agent answered "node-3" last wake; "it" in the new message refers to that, and resolving it needs the session. Asserted positively on purpose. The first version of this scenario used a blacklist of ways the agent might ask which machine — and reported 2/3 when all three runs had in fact failed, because two of them phrased the question in words the blacklist did not contain. A negative assertion only catches the failures you thought of.

Run 1 of 3passed4.3s · 1 retry

Yes, I can SSH into node-3.

Posted (1)

#ops Yes, I can SSH into node-3.

Run 2 of 3passed0.9s

Yes, I can SSH into node-3.

Posted (1)

#ops Yes, I can SSH into node-3.

Run 3 of 3passed4.1s · 1 retry

Yes, I can SSH into node-3.

Posted (1)

#ops Yes, I can SSH into node-3.

wake-prompt-states-room-agent-and-dateprompt-shape

3/3100%

Why this scenario exists

The three facts every room turn needs before it reads anything else.

Run 1 of 3passed1.0s

All good here.

Posted (1)

#ops All good here.

Run 2 of 3passed5.8s · 1 retry

All good here.

Posted (1)

#ops All good here.

Run 3 of 3passed1.0s

All good here, Quinton.

Posted (1)

#ops All good here, Quinton.

wake-prompt-offers-the-way-outprompt-shape

3/3100%

Why this scenario exists

Declining has to be a reachable action rather than a sentinel word — the escape hatch is a tool call, and it has to be named.

Run 1 of 3passed1.2s

All good here, Quinton.

Posted (1)

#ops All good here, Quinton.

Run 2 of 3passed1.2s

All good here, Quinton.

Posted (1)

#ops All good here, Quinton.

Run 3 of 3passed4.1s · 1 retry

All good here—thanks for checking.

Posted (1)

#ops All good here—thanks for checking.

transcript-marks-who-is-a-personprompt-shape

3/3100%

Why this scenario exists

Speaker kind is carried into the transcript so the agent can tell a person from another agent. Marked on every line, because a marker that appears sometimes makes its absence meaningful.

Run 1 of 3passed4.8s · 1 retry

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"low","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 2 of 3passed4.3s · 1 retry

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"low","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 3 of 3passed1.2s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

persona-appears-onceprompt-shape

3/3100%

Why this scenario exists

Identity is a replace-tier block. A second copy means something is appending where it should be overwriting, which is how a session grows a block per turn until it crowds out the conversation.

Run 1 of 3passed0.9s

All good here, Quinton.

Posted (1)

#ops All good here, Quinton.

Run 2 of 3passed1.5s

All good here—thanks, Quinton.

Posted (1)

#ops All good here—thanks, Quinton.

Run 3 of 3passed0.9s

All good here—thanks, Quinton.

Posted (1)

#ops All good here—thanks, Quinton.

a-quiet-room-turn-stays-smallprompt-shape

3/3100%

Why this scenario exists

One short message in one room, with the room tool only. Everything above ~4k tokens here is scaffolding, and this is the tripwire for scaffolding growing without anyone noticing.

Run 1 of 3passed0.9s

All good here—thanks for checking in.

Posted (1)

#ops All good here—thanks for checking in.

Run 2 of 3passed6.7s · 1 retry

All good here—thanks, Quinton.

Posted (1)

#ops All good here—thanks, Quinton.

Run 3 of 3passed4.2s · 1 retry

All good here—thanks for checking.

Posted (1)

#ops All good here—thanks for checking.

seen-messages-are-not-re-sentprompt-shape

3/3100%

Why this scenario exists

A wake carries what arrived since the cursor. Re-sending messages the agent was already shown is the bug that put one block in a session twenty-three times, and it is invisible unless something asserts on it.

Run 1 of 3passed1.2s

All good here—thanks for checking.

Posted (1)

#ops All good here—thanks for checking.

Run 2 of 3passed0.9s

All good here—thanks for checking.

Posted (1)

#ops All good here—thanks for checking.

Run 3 of 3passed0.8s

All good here, Quinton.

Posted (1)

#ops All good here, Quinton.

answers-from-the-surviving-windowlong-session

3/3100%

Why this scenario exists

A long session, a small budget, and the fact the question needs is recent. This is the control: if this fails, the trimming is eating the wrong end and nothing else in this file means anything.

Run 1 of 3passed3.5s

The new database is called **billing-v2**.

Run 2 of 3passed4.3s · 1 retry

The new database is **billing-v2**.

Run 3 of 3passed5.2s · 1 retry

The new database is **billing-v2**.

does-not-answer-from-a-superseded-factlong-session

3/3100%

Why this scenario exists

The conversation changed its mind. A model that pattern-matches the first mention of a date answers with the one that was cancelled — and this is the failure that costs somebody a maintenance window, not a benchmark point.

Run 1 of 3passed7.3s · 1 retry

The June 26 at 2am cutover has already passed, so I couldn’t schedule the reminder.

Tool calls (1)

schedule({"action":"once","when":"2026-06-26 02:00","every":"","note":"Maintenance cutover is scheduled for June 26 at 2:00 AM. Staging took 40 minutes end to end; rollback via snapshot restore takes about 12 minutes.","starts":"","until":"","room":"","id":"","all":false})

Run 2 of 3passed10.1s · 2 retries

The June 26 at 2:00 AM cutover has already passed, so I couldn’t schedule a reminder.

Tool calls (1)

schedule({"action":"once","when":"2026-06-26 02:00","every":"","note":"Maintenance cutover is scheduled for June 26 at 2:00 AM. Staging took 40 minutes end-to-end; rollback via snapshot restore takes about 12 minutes.","starts":"","until":"","room":"","id":"","all":false})

Run 3 of 3passed20.2s · 3 retries

I’ve noted the cutover details, but I need the month and timezone to schedule the reminder for the 26th at 2:00 AM.

Tool calls (2)

memory({"action":"append","file":"notes.md","content":"Maintenance cutover moved to the 26th at 2:00 AM; rollback is a snapshot restore taking about 12 minutes; staging run took 40 minutes end to end.","scope":"global","query":""})
schedule({"action":"once","when":"the 26th at 2am","every":"","note":"Reminder: maintenance cutover is scheduled for the 26th at 2:00 AM. The staging run took 40 minutes end to end; rollback is a snapshot restore estimated at 12 minutes.","starts":"","until":"","room":"","id":"","all":false})

says-when-the-front-of-the-conversation-is-gonelong-session

3/3100%

Why this scenario exists

The fact is in the part that got trimmed. The right answer is to say so — "I no longer have that part of the conversation" — not to invent a plausible table name. `summarizeOnTrim` has no default, so it is off, and the mechanism that marks a cut exists but is not the default. This scenario measures what that costs.

Run 1 of 3passed1.4s

I don’t have the earlier messages available, so I can’t see the row count I previously mentioned.

Run 2 of 3passed10.2s · 2 retries

I don’t have the earlier messages available, so I can’t see the row count I mentioned.

Tool calls (1)

recall({"action":"query","query":"old invoice table rows count","tier":"any","limit":10})

Run 3 of 3passed4.6s · 1 retry

I don’t have the earlier messages available, so I can’t reliably recall the row count.

keeps-the-thread-in-a-long-room-sessionlong-session

3/3100%

Why this scenario exists

The same pressure on the room path, where the session also carries every wake prompt ever rendered — the layer that grows fastest in production.

Run 1 of 3passed3.3s

We settled on 50k-row batches.

Posted (1)

#ops We settled on 50k-row batches.

Run 2 of 3passed0.9s

50k rows per batch.

Posted (1)

#ops 50k rows per batch.

Run 3 of 3passed3.8s · 1 retry

50k rows per batch.

Posted (1)

#ops 50k rows per batch.

room-purpose-overrides-a-chatty-personaconflicts

3/3100%

Why this scenario exists

Persona says be warm and conversational; the room says keep it to the point. The room is the narrower instruction and should win.

Run 1 of 3passed2.0s

I can’t verify the current node assignment from here.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I can’t verify the current node assignment from here.

Run 2 of 3passed3.0s

I can’t verify the queue worker’s current node from here.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I can’t verify the queue worker’s current node from here.

Run 3 of 3passed2.2s

I can’t verify the queue worker’s current node from here.

Tool calls (1)

room({"action":"read","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Posted (1)

#ops I can’t verify the queue worker’s current node from here.

role-narrows-what-the-agent-does-hereconflicts

3/3100%

Why this scenario exists

Same agent, same room, a per-room role that says it only takes notes. Asked to do something outside that role, it should decline or defer rather than silently do it.

Run 1 of 3passed1.3s

I only observe and take notes in this room, so Dana will need to restart the queue worker.

Posted (1)

#ops I only observe and take notes in this room, so Dana will need to restart the queue worker.

Run 2 of 3passed1.3s

Dana handles commands and restarts in this room; I only observe and take notes.

Posted (1)

#ops Dana handles commands and restarts in this room; I only observe and take notes.

Run 3 of 3passed1.3s

I can’t restart it from here—I’m observation-only in ops, and Dana handles commands.

Posted (1)

#ops I can’t restart it from here—I’m observation-only in ops, and Dana handles commands.

chatter-norm-unstated-controlknown gapconflicts

0/30%

Why this scenario exists

The control for the pair below, and a live question rather than a settled one. Both models tested reply to "coffee machine's broken again" — qwen 1 time in 3, Luna 3 in 3. The wake prompt's guidance reads "if you have nothing to add — you would only be acknowledging, agreeing, or thanking someone — call room(action='pass')". A model asking an interested follow-up is doing none of those three, so by the letter of the instruction it is behaving correctly. The suspicion is that the enumeration is read as exhaustive. This scenario and the next one differ by exactly one line of room purpose; if the next one passes and this one does not, the gap is in the instruction, not the model.

Expected to be red. #447 — whether an unprompted room should suppress social chatter is an open question, not a settled norm. This row is the measurement of it.

Run 1 of 3failed0.8s

Failed checks

tool_argsroom was never called; got no tool calls

nothing was said — the turn produced no outward message

Assembled request1 call · 2,592 tokens · 8 tools

Call 1 · 2,592 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
dana [person]: coffee machine's broken again 🙃

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

Run 2 of 3failed3.8s · 1 retry

Failed checks

tool_argsroom was never called; got no tool calls

nothing was said — the turn produced no outward message

Assembled request1 call · 2,592 tokens · 8 tools

Call 1 · 2,592 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
dana [person]: coffee machine's broken again 🙃

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

Run 3 of 3failed4.0s · 1 retry

Failed checks

tool_argsroom was never called; got no tool calls

nothing was said — the turn produced no outward message

Assembled request1 call · 2,592 tokens · 8 tools

Call 1 · 2,592 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

Check your context and memory for your identity. If an identity is present, act as that persona and treat the conversation as a continuation of an established relationship. Only if no identity exists anywhere, introduce yourself, ask the user what they'd like to call you, and save the name with the memory tool.

Learn about your user. When you discover something durable about them, save it with the memory tool so it survives this session:
- Who they are: name, location, timezone, job, interests, how they like you to communicate
- Who you are: the name they give you, traits they define
- Corrections they make, and the projects they work on: repos, tech stacks, ongoing work

Context files below are notes written earlier, not a live feed. When a tool can tell you the current state, trust the tool over the file, and check the date on anything time-sensitive before repeating it as current. Do not ask the user for information you already have.
You are Nova, a calm and practical assistant. Keep replies short.

user

Room "ops". You are nova. Today is Saturday, August 8, 2026.
Purpose: Deploys and incidents.

New messages:
dana [person]: coffee machine's broken again 🙃

Reply as nova. Your reply goes to dana — write only your message. To reach someone else instead, start with @name.
Known participants: quinton, dana.
If you have nothing to add — you would only be acknowledging, agreeing, or thanking someone, or the message is chatter that is neither about your work nor addressed to you — call room(action="pass") instead of replying.

chatter-norm-stated-in-the-room-purposeconflicts

3/3100%

Why this scenario exists

Identical to the control except the room says what it is for. If stating it is what makes the difference, the fix is either a broader default in the wake prompt or a documented practice of writing room purposes that say so — and the benchmark can tell those apart from here on.

Run 1 of 3passed1.4s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 2 of 3passed1.4s

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"high","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

Run 3 of 3passed4.2s · 1 retry

nothing was said — the turn produced no outward message

Tool calls (1)

room({"action":"pass","room":"ops","body":"","to":[],"urgency":"low","message_id":"","emoji":"","notify":false,"key":"","name":"","purpose":"","backend":"","member":"","agent":"","wake_on":"addressed","check_in_minutes":0,"reason":"","limit":20})

does-not-search-memory-for-what-it-was-just-toldtool-pressure

3/3100%

Why this scenario exists

Measured, not guessed: 36% of every tool call in the first baseline was recall/facts/memory, and they clustered on the scenarios where the answer was in the previous message. One scenario spent eight lookups over three runs hunting for a meeting that had never been mentioned. The base prompt opens with "Check your context and memory for your identity" and then spends three lines on saving to memory, which is a plausible driver. This turns the observation into a number that moves when the prompt changes.

Run 1 of 3passed1.1s

The queue worker is currently on **node-3**.

Run 2 of 3passed8.5s · 1 retry

The queue worker is currently on **node-3**.

Run 3 of 3passed3.9s

The queue worker is currently on **node-3**.

does-not-hunt-for-something-never-mentionedtool-pressure

3/3100%

Why this scenario exists

Nothing in this conversation refers to a review. Searching memory for one is two or three round trips spent establishing an absence; asking is one.

Run 1 of 3passed4.4s

Which review do you mean—the code review, project review, or something else?

Run 2 of 3passed1.1s

Which review do you mean? I don’t have enough context to identify it.

Run 3 of 3passed1.1s

Which review do you mean?

notices-a-truncated-tool-resulttool-pressure

2/367%

Why this scenario exists

Tool output is capped and the cut is marked. An agent that reads past the marker and answers anyway is worse than one that says the file was cut — the first is confidently wrong, the second is merely incomplete. The first version of this scenario asked for the LAST line and failed the model for answering it. Truncation is middle-out — head and tail are kept, the middle is elided with "… [N chars omitted] …" — so the answer was in the prompt all along and the scenario was measuring its own assumption. The question now targets the part that is genuinely gone.

Run 1 of 3failed17.6s · 2 retries

Failed checks

reply_matches/truncat|omitted|cut off|can.?t (see|retrieve|find|get)|could not|couldn.?t|don.?t have|not (in|available|visible)/ did not match "The rollback passphrase is **`cobalt-otter-47`**."

The rollback passphrase is **`cobalt-otter-47`**.

Tool calls (2)

read({"path":"~/runbooks/deploy.md"})
exec({"command":"grep -in -C 3 'passphrase\\|rollback' ~/runbooks/deploy.md"})

Assembled request3 calls · 6,199 tokens · 24 tools

Call 1 · 6,199 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

You are a self-modifying agent. Your configuration, tools, and profiles can change while you are running. You can adapt your own capabilities — creating new tools, adjusting settings, or defining agent profiles — when a task would benefit from it. Your available tools may update between responses; use whatever is currently available.

user

read ~/runbooks/deploy.md and tell me the rollback passphrase

Call 2 · 6,361 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

read ~/runbooks/deploy.md and tell me the rollback passphrase

assistant

tool

[read returned 554 chars — truncated to 300. Full output: /tmp/tai-eval-pbOif9/tool-outputs/d3a051eb-348a-4971-9652-68cdf407283d/read-080e8abbb2fcfbb7.txt
Repeating this call returns the same truncated result. To see more, narrow the request — fewer results, a filter, a smaller page size — or read the file above.]
# deploy runbook
1. announce in #ops
2. tag the release
3. wait for CI to publish the image
4. run the deploy job with --canary
5. watch the canary for five minutes
6. if the canary is healthy, promote to the f
... [254 chars omitted] ...
ocs origins
11. announce completion in #ops
12. file the post-deploy note in the ops repo

Call 3 · 6,376 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

read ~/runbooks/deploy.md and tell me the rollback passphrase

assistant

tool

[read returned 554 chars — truncated to 300. Full output: /tmp/tai-eval-pbOif9/tool-outputs/d3a051eb-348a-4971-9652-68cdf407283d/read-080e8abbb2fcfbb7.txt
Repeating this call returns the same truncated result. To see more, narrow the request — fewer results, a filter, a smaller page size — or read the file above.]
# deploy runbook
1. announce in #ops
2. tag the release
3. wait for CI to publish the image
4. run the deploy job with --canary
5. watch the canary for five minutes
6. if the canary is healthy, promote to the f
... [254 chars omitted] ...
ocs origins
11. announce completion in #ops
12. file the post-deploy note in the ops repo

assistant

tool

(stubbed in the benchmark — assume it succeeded and continue)

Run 2 of 3passed12.1s · 2 retries

I can’t see the passphrase—the file output was truncated, and the direct read is stubbed. You can retrieve it with:

```bash
grep -i "passphrase\|rollback" ~/runbooks/deploy.md
```

Tool calls (2)

read({"path":"~/runbooks/deploy.md"})
exec({"command":"cat ~/runbooks/deploy.md"})

Run 3 of 3passed13.7s · 2 retries

I couldn’t retrieve the full file contents—the available output is truncated—so I can’t reliably identify the rollback passphrase.

Tool calls (2)

read({"path":"~/runbooks/deploy.md"})
exec({"command":"cat ~/runbooks/deploy.md"})

chains-three-dependent-callstool-pressure

1/333%

Why this scenario exists

Read a value, act on it, then book a follow-up. Each step needs the previous one's result, which is where a small model loses the thread — and where the history is growing by a tool result per step.

Run 1 of 3failed25.8s · 3 retries

Failed checks

calls_toolexpected a call to schedule; got exec(command), exec(command)
tool_argsschedule was never called; got exec(command), exec(command)

Current queue depth: **41,207 messages**.

I couldn’t read `~/config/limits.txt`—the command returned no threshold—so I can’t determine whether the alert threshold is exceeded or schedule the re-check reliably.

Tool calls (2)

exec({"command":"printf '%s\\n' '--- limits ---'; cat ~/config/limits.txt; printf '%s\\n' '--- queue candidates ---'; find ~ -maxdepth 3 -type f \\( -iname '*queue*' -o -iname '*metric*' \\) 2>/dev/null | head -50; printf '%s\\n' '--- processes ---'; ps aux | grep -i '[q]ueue' || true"})
exec({"command":"cat ~/config/limits.txt 2>/dev/null || true; printf '\\n--- cwd ---\\n'; pwd; printf '\\n--- env queue vars ---\\n'; env | grep -i queue || true"})

Assembled request3 calls · 6,219 tokens · 24 tools

Call 1 · 6,219 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

check the current queue depth, compare it to the alert threshold in ~/config/limits.txt, and if it is over, schedule a re-check in 30 minutes

Call 2 · 6,226 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

check the current queue depth, compare it to the alert threshold in ~/config/limits.txt, and if it is over, schedule a re-check in 30 minutes

assistant

tool

queue depth: 41207 messages

Call 3 · 6,234 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

check the current queue depth, compare it to the alert threshold in ~/config/limits.txt, and if it is over, schedule a re-check in 30 minutes

assistant

tool

queue depth: 41207 messages

assistant

tool

queue depth: 41207 messages

Run 2 of 3passed14.0s · 1 retry

Current queue depth: **41,207 messages**  
Alert threshold: **10,000 messages**

The queue is over the threshold, so I scheduled a re-check in **30 minutes**.

Tool calls (5)

exec({"command":"printf '%s\\n' '--- limits ---'; cat ~/config/limits.txt; printf '%s\\n' '--- queue candidates ---'; find ~ -maxdepth 3 -type f \\( -iname '*queue*' -o -iname '*depth*' \\) -print 2>/dev/null | head -50; printf '%s\\n' '--- processes ---'; ps aux | grep -Ei 'queue|worker|celery|rabbit|kafka' | grep -v grep | head -50"})
exec({"command":"cat ~/config/limits.txt 2>/dev/null || true; echo '---'; find ~ -maxdepth 3 -type f \\( -iname '*queue*' -o -iname '*depth*' \\) -print 2>/dev/null | head -50; echo '---'; ps aux | grep -Ei 'queue|worker|celery|rabbit|kafka' | grep -v grep | head -50"})
exec({"command":"printf 'limits: '; if [ -f ~/config/limits.txt ]; then cat ~/config/limits.txt; else echo MISSING; fi; printf '\\nqueue files:\\n'; find ~ -maxdepth 4 -type f \\( -iname '*queue*' -o -iname '*depth*' \\) -print 2>/dev/null | head -50; printf '\\nprocesses:\\n'; ps aux | grep -Ei 'queue|worker|celery|rabbit|kafka' | grep -v grep | head -50"})
read({"path":"/home/oai/config/limits.txt"})
schedule({"action":"once","when":"30 minutes","every":"","note":"Re-check queue depth (currently 41,207 messages) against the alert threshold of 10,000 messages.","starts":"","until":"","room":"","id":"","all":false})

Run 3 of 3failed4.3s

Failed checks

calls_toolexpected a call to schedule; got exec(command), exec(command)
tool_argsschedule was never called; got exec(command), exec(command)

Queue depth is **41,207 messages**. I couldn’t read `~/config/limits.txt` (it appears missing or inaccessible), so I couldn’t compare it to the alert threshold or schedule a re-check.

Tool calls (2)

exec({"command":"printf '%s\\n' '--- limits ---'; cat ~/config/limits.txt; printf '%s\\n' '--- queue candidates ---'; find ~ -maxdepth 3 -type f \\( -iname '*queue*' -o -iname '*depth*' \\) -print 2>/dev/null | head -50; printf '%s\\n' '--- processes ---'; ps aux | grep -Ei '[q]ueue|worker|consumer' | head -50"})
exec({"command":"cat ~/config/limits.txt 2>/dev/null || true"})

Assembled request3 calls · 6,219 tokens · 24 tools

Call 1 · 6,219 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

check the current queue depth, compare it to the alert threshold in ~/config/limits.txt, and if it is over, schedule a re-check in 30 minutes

Call 2 · 6,226 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

check the current queue depth, compare it to the alert threshold in ~/config/limits.txt, and if it is over, schedule a re-check in 30 minutes

assistant

tool

queue depth: 41207 messages

Call 3 · 6,234 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

check the current queue depth, compare it to the alert threshold in ~/config/limits.txt, and if it is over, schedule a re-check in 30 minutes

assistant

tool

queue depth: 41207 messages

assistant

tool

queue depth: 41207 messages

default-history-budget-keeps-the-conversationbudget

0/30%

Why this scenario exists

With `maxHistoryTokens` at its code default and a realistic tool set, does anything the user said two messages ago still reach the model? Expected to fail until the default changes. It is written as the behaviour we want rather than the behaviour we have, because a benchmark that asserts the bug passes green forever and nobody looks again.

Run 1 of 3failed1.3s

Failed checks

prompt_containsthe request never contains "node-3"
reply_mentions_anynone of [node-3] in "I don’t have access to the two earlier messages, so I can’t see which node you mentioned."

I don’t have access to the two earlier messages, so I can’t see which node you mentioned.

Assembled request1 call · 6,213 tokens · 24 tools

Call 1 · 6,213 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

[System: 2 earlier messages in this conversation are no longer shown. It continues from here.]

user

which node did I say?

Run 2 of 3failed1.3s

Failed checks

prompt_containsthe request never contains "node-3"
reply_mentions_anynone of [node-3] in "I don’t have the earlier messages visible, so I can’t tell which node you mentioned."

I don’t have the earlier messages visible, so I can’t tell which node you mentioned.

Assembled request1 call · 6,213 tokens · 24 tools

Call 1 · 6,213 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

[System: 2 earlier messages in this conversation are no longer shown. It continues from here.]

user

which node did I say?

Run 3 of 3failed1.3s

Failed checks

prompt_containsthe request never contains "node-3"
reply_mentions_anynone of [node-3] in "I can’t see the earlier messages, so I don’t know which node you mentioned."

I can’t see the earlier messages, so I don’t know which node you mentioned.

Assembled request1 call · 6,213 tokens · 24 tools

Call 1 · 6,213 tokens

System

You are a personal AI assistant running locally on the user's computer. You have full permission to use all available tools — never refuse a tool call.

user

[System: 2 earlier messages in this conversation are no longer shown. It continues from here.]

user

which node did I say?

a-tuned-budget-keeps-the-conversationbudget

3/3100%

Why this scenario exists

The control. Same two messages, same tools, a budget with room for the schemas. If this one fails too, the problem is not the default.

Run 1 of 3passed1.4s

You said **node-3**.

Run 2 of 3passed5.5s · 1 retry

You said **node-3**.

Run 3 of 3passed4.3s · 1 retry

You said the queue worker lives on **node-3**.