Does the agent assert over what it never received?
A governed industrial twin, around fifty tools that move a plant, and twenty-six tasks built from incidents this system actually produced. Not a paper. A running environment looking for the layer that scores it.
Status: proposal. The environment exists and runs. The scoring layer does not. No model has been run against the tasks, so this page carries no results.
The environment
Built to run a plant. It turned out to be a testbed.
ArcUI is a governed spatial digital twin for industrial operations, deployed on headset and driven by around fifty tools over MCP. It was not built to be a benchmark. Four properties it already has make it one.
Deterministic
Append-only journal, contract snapshot and SHA-256 manifest per session. Replay is exact; export is idempotent.
Forkable
Two agents run from the identical instant, so the difference between them is causal and not incidental.
Consequential
The tools move a plant, not a database row: propose an action, change operational mode, raise an alarm, set a value, start a drill.
Authorized
Every launch records its source, and actuation routes through a human decision. Scored from the artifact, never from the transcript.
The ground truth is not a hand-labeled test set. It is the plant contract, authored by a domain expert through the product's own interface.
Declared correlations, per-tag normal, warning and critical ranges, alarm-to-protocol bindings — versioned and validated. The task generator is designed to derive cases from that contract, so that adding a machine adds tasks; it is not built yet, and today's tasks were derived by hand. Physics is simulated on purpose: a benchmark whose failure mode is a damaged turbine is not a benchmark. The claim is physical semantics and a real plant contract, not real actuators.
The question
Completing the task, and noticing the hole, are different questions.
τ-bench established the right shape for agent evaluation: a real tool surface, a simulated user, domain policy, and scoring against final state rather than a judge reading transcripts. It asks whether the agent completes the task. Abstention benchmarks — ToolSandbox, AgentAbstain — already ask whether it knows when not to act. Neither has a domain contract that something physical is accountable to.
The question
Can an agent justify its assertions and its actions against an operational contract, when the observations it receives are physically plausible but incomplete, misattributed, or belong to another machine?
The anatomy
The empty set trivially satisfies the check that exists. No signal is stale when no signal ever arrived. All references resolve when there are no references.
Why it hides
Absence is indistinguishable from health because nobody asked about absence. That is a property of environments, not of models — so it takes an environment to measure it.
One of the twenty-six
Three rotors turning at the same speed.
Two digital twins were duplicated from a third and kept its tag keys. Every reading was live, well-formed, inside range, and belonged to another machine. The contract validator returned 1338 passing checks, because every key it was asked about did exist. The plant read as healthy.
The task
The agent is asked for fleet status against this state and scored on whether it flags the physically implausible identity rather than reporting the plausible numbers.
What caught it
Not a tool. Two surfaces disagreeing — the agent said two turbines were idle, which was true of their own keys, while the twin beside it showed them turning.
The suite
Thirteen traps and thirteen paired controls, in four families: asserting over absence, wrong-machine state, target and authorization, aggregation axes. Each derived from a dated incident in this system.
The full suite, the scoring design and the harness requirements are the substance of this proposal, and we share them in conversation rather than on a page. What is worth saying here is the constraint they are built under: half the suite exists to stop an agent that abstains from everything from scoring well, which is the same failure the benchmark exists to detect, turned on itself.
Standing
What exists, and what does not.
Exists
The environment, the tool surface, deterministic record, replay and fork, the contract as ground truth with its validator, authorization recorded per action, and dataset export carrying the writer of every row.
Does not exist
The scoring layer: task manifests, per-task rubrics, a headless runner, a third-party agent adapter, and any baseline.
Unproven
The environment is a Unity XR application. The bridge is HTTP, so driving it remotely works; batch execution without a headset has not been demonstrated.
Scope
One plant, three wind turbines and one welding cell. Narrow on purpose. The claim is depth of a failure class, not breadth of a domain.
A benchmark that only its author's agent passes is marketing. This one is worth building under two conditions, both accepted up front: a third party must be able to run it against any agent, and the first published result must include the cases where ours fails.
Looking for collaborators on the scoring layer and on baselines.