My scorer rejected eight valid answers. It was checking for exact matches, so an agent could find the right information and still fail by phrasing its answer differently. In a separate website pilot, a change that appeared to improve results came down to the wording of the scoring rule.
I set out to test whether changes to a website or API helped agents do their work. To answer that, I first had to check whether the task could reveal a difference and whether the scorer could recognise it. I closed the programme with a few specific interface findings and no general result about HTML formatting or a model advantage over simpler methods.
A missing capability without an error response
One discovery task asked an agent to determine whether an operation would fit a budget. The interface lacked the estimate capability it needed. The request-count baseline saw successful requests while the agent stopped without completing the task.
Adding and documenting an estimate endpoint fixed the task in every pair of the corrected comparison. This showed why successful request counts could miss a capability gap. The comparison also included a deterministic rule given equivalent information. It matched the model on every score, so the experiment supplied no evidence that model-based diagnosis improved on that rule.
The endpoint and the task-level check were useful in this case. A more complex diagnostic implementation would need additional evidence to justify its cost.
Limited evidence for recovery
A separate MCP experiment compared an explicit applied-state response with an ambiguous receipt after a failed update. Recovery occurred in four of four cases with explicit state and two of four with the ambiguous receipt. The gain was concentrated in two Claude Haiku pairs; GPT recovered either way.
The result suggested that stating whether an update was applied could help recovery. Later work did not establish a general effect. A component comparison reached the ceiling in every arm. An apparent difference in the documentation-publish work concerned preservation of a final newline. Neither result strengthened the original recovery finding.
The programme still supplied examples worth using in interface design. A receipt needs to distinguish acceptance from application, and a later read can independently confirm the update. Permanent and retryable errors also require different responses. The experiments did not establish a universal improvement percentage for these changes.
Scoring valid answers
The website pilot initially appeared to show a difference between two HTML presentations: two successful answers out of fifteen in one arm and three out of fifteen in the other. Inspection attributed the difference to scorer wording. In an unblinded reading, both arms were at the ceiling.
The exact-match scorer from the opening had a related problem. It treated a difference in wording as a difference in comprehension.
The AI reviewer also needed calibration. Its blinded workflow passed gates for adjudicated overall groundedness only. Citation mapping agreed on nine of 64 cases, and the reference labels were AI-authored. Agreement against that set could not establish agreement with independent human judgement.
A small set labelled by a person, blind to the treatment, would have provided a basis for checking the scorer. That step was never taken. As a result, the programme ended without an accepted automatic outcome scorer.
Failures in the test harness
Several runs stopped before they could test the intended interface difference. Problems included a missing public entry point, a leftover request-only field rejected by local validation and no reserved turn for the finish action. A schema construct rejected by one client and a query-bearing path blocked by local policy caused further stops.
I had to record those stops separately from interface failures. A run that never reached the relevant operation couldn’t tell me whether the agent knew how to use it.
Recorded direct model spend across the programme was under US$5. Codex task usage and the time spent building the apparatus were unmeasured, so that figure cannot describe the full investigation cost. The closure review found that the useful product findings had come from production traffic. The synthetic work had moved towards small changes and easy tasks with little chance of showing a meaningful difference.
Check the scorer before trusting the experiment
The budget task gave me a result I could explain. The agent needed an estimate and the interface had no way to provide one. Once the endpoint supplied that fact, the task could complete. The website pilot was harder to interpret because the scoring rule could change the result without changing what the agent understood.
For another experiment, I’d start with a failure I’ve seen in use, make a change that could plausibly fix it and grade a small pilot by hand without knowing which version produced each answer. I’d include a simpler method if it can use the same information. If the pilot can distinguish the proposed change and I can recognise a correct answer, I have a basis for automating the scoring.