Field Kit.

For a while this page displayed what each run cost, and those numbers were fabricated: a hardcoded constant plus your character count divided by four. Fixing it exposed a second problem. Prompt caching had never fired at all, because the cache marker sat on a prompt too short to qualify on either model.

Both are fixed, and everything below is what I rebuilt. The lesson is narrower than test your code: measure the thing you are claiming. I had written confidently about model routing on a page whose own instrumentation was guessing.

Experience Mapper answers from a written record of nine projects I have worked on. The first version scored that record against your question and passed the model only the best four entries. It kept dropping the one that actually fit. Nine entries is small enough that a vector database would be pure overhead, so now I send all of it and let the model choose, with a lexical ranking attached as a hint rather than a filter.

That also made caching possible. A cached prompt only works on an exact prefix match, so the corpus sits in front of the cache marker and your question sits behind it. My first design cached the retrieved subset, which changed with every question, so nothing ever hit. The trade-off: each request now carries about 8,000 tokens of corpus, and a one-hour cache write costs twice a normal request while a read costs a tenth. You break even on the third question.

Decision Simulator runs on Haiku 4.5, where the first words appear in about a second against two to five for Sonnet. The question is a straight either/or, so speed matters more than nuance. Experience Mapper and Compass Check run on Sonnet 5, where the answer needs judgment and a few more seconds is a fair price.

The catch showed up in caching. Haiku will not cache a prompt under 4,096 tokens, and Decision Simulator’s prompt is deliberately lean. Padding it past the threshold would cost the exact speed I picked Haiku for, so it runs uncached and the panel says so. I measured both thresholds against the live API rather than trusting the docs: 1,024 on Sonnet, 4,096 on Haiku. Below the line the marker is ignored silently.

Every tool returns JSON matching a schema I define, and the model is constrained to that shape rather than asked for it. The page validates the result again before rendering, so an answer it cannot draw becomes a handled error instead of a broken page.

Having a real contract made an old problem obvious. Decision Simulator used to rate each option out of five and compute a weighted total to two decimals. It cannot measure these options, so those numbers were invented. It now names which option wins each point and why, which was the only part carrying information.

A model will answer a question it has no business answering in the same tone it uses when it knows. Asking for hedging in the prompt works some of the time, which is not good enough. So confidence, what it is unsure about, and the alternatives it set aside are required fields. A response without them fails validation before it reaches the page.

Compass Check takes this furthest. The most useful thing it produces is not its answer, it is the list of what you have not told it. That now sits directly under the answer.

Token usage only exists once the model has finished, long after the response headers have gone out, so it cannot travel in a header. The server appends it as a framed trailer at the end of the same stream and the client strips it off before the JSON parser sees it. When a number is not known yet, the panel says so rather than estimating.

Answers render as they arrive, so the first words appear in a second or two instead of ten. That means reading half-built JSON — unfinished arrays, fields that do not exist yet — which every renderer treats as an expected state before validating the finished object in full.

Rate limits, timeouts, overload and malformed output each map to a specific message that says what happened and whether trying again will help. Requests are validated on the way in and responses on the way out.

You cannot eyeball a system that answers differently every time, so there is a suite that runs the real tools against real questions and grades what comes back. Every question is asked 3 times, because one good answer proves nothing. Last full run on Aug 24, 2026: 312 of 321 checks passed across 3 tools.

Most checks are plain code: exactly two options, no invented numbers, and every company Experience Mapper cites actually exists in my corpus. One re-derives the cached prefix for two different questions and fails if a single character differs, which is precisely the bug that sat here unnoticed.

A recent catch: I added my current role at Lyft, where the work is still in progress. The prompt asks each example to lead with a failed first attempt, so the model produced one — a failure at my current job that never happened. Entries can now be marked in progress, and a check fails the run if any past-tense outcome is attributed to them.