CASE STUDY 01 · PRODUCT DESIGN
The workflow behind an internal LLM evaluation tool. Writing prompts as reusable templates, running them across hundreds of cases, and scoring whether each revision actually moved quality forward.
Prompt engineers needed one place to write a prompt, test it, and know if a change helped.
Domain experts and prompt engineers were testing prompts against real documents with no shared place to do it, and no consistent way to score results. Teams shipping LLM features would change a prompt, run it, and watch a pass rate move by a couple points. What that number hid was which cases improved and which quietly broke.
One loop, four stages.
A prompt is written once, in plain language, as a template with a variable slot. The system translates it into the structured format the model needs, then runs it against every case in a test set, sometimes hundreds of documents, each one becoming a separate item. Each result is scored against a target, the output a human already agreed was correct. Findings fold back into the next version, and the loop runs again.
Write once. Test against whole dataset.
A prompt starts in Prompt Lab as a plain-language template with a variable slot. Running it sends that template across every case in the dataset, sometimes hundreds of documents, and the result is saved to Experiments List as a new run, versioned and ready to review.
A prompt is written without formatting, the system translates it into the structured format the model needs. From there, criteria and dataset are selected, then the run is executed.
Running a prompt sends it across every paper in the data set and saves the result as a new experiment, with version, score, and status visible at a glance.
Every result needs a human judgment.
An LLM output problem isn't when it's obviously wrong. It's when it sounds confident. Each result is scored against its target: worse, same, or better, to see what actually works. This evaluation reveals the real behavior: which cases pass, which fail, and whether a cheaper model performs just as well. The data shows if the prompt is production-ready.
An Experiment runs one prompt version against a full dataset, producing one result per document. Each result is evaluated against its target by hand. The distribution is that judgment, aggregated.
Each Experiment Instance shows one result against its target. The discussion thread captures the team's reasoning before a rating is cast. Once scored worse, same, or better and saved, the next item loads and that score is recorded in the Experiment's aggregate.
Two outputs, one target, rated side by side.
This is the screen the rest of the product exists to reach. Two Experiment Instance run side by side against the same target: V2 on the left, V3 beside it, ground truth on the right. Each column carries its own rating control.
Same score before and after can still mean twenty cases fixed and twenty broken. Only the side-by-side shows which.
Same prompt, two models. The comparison answers a cost question with evidence instead of assumption.
Stepping through paired results surfaces the pattern in the failures, not just the count of them.
Three anchors, not a spectrum.
Worse than target, same as target, better than target. This three-point system was chosen because it removes subjective variation. A five or ten point scale asks every rater to define good internally, and they never agree. Anchoring to a target asks one question only: did the output move toward the reference or away from it.
Prompt tools measure whether quality moved.
This one was built to show what moved, and why.