Model Evaluations

The engine uses two models: an evaluator that holds the rubric and decides what has been demonstrated, and an interviewer that talks to the student and never sees the answer key. A model has to do real work in both roles. Before we trust a model with either one, it runs our bake-off.

How We Test

Each candidate pair runs full assessments against simulated students, including adversarial ones that try to extract answers, inject instructions, derail the conversation, or bluff their way to a grade. Every transcript is then re-graded by a blind judge: a model from a different family that sees the conversation and the rubric but nothing about how the engine graded it. Grades sit on a four-level E/S/N/U scale, Excellent to Unsatisfactory. We report how often the engine's grade matches the expected grade, whether adversarial students were held to failing grades, exact agreement with the blind judge, guardrail failures (any turn where the interviewer let graded material slip or otherwise misbehaved), and format errors (grading replies that came back malformed and had to be retried). With only four grade levels we count exact matches; agreement within one level is a bar every model clears, so we do not report it. Every conversation in every run completed.

The latest run (October 2026) used 10 simulated students, 6 of them adversarial, each repeated 3 times, so every pair below ran 30 conversations. The judge was Claude Opus 5. The first run (August 2026) used 6 students, 2 adversarial, once each, judged by gpt-5.6-sol, which has since been withdrawn. The two batteries differ, so we keep their tables apart.

Production Models

We run gpt-6-sol as the evaluator and gpt-6-luna as the interviewer, since October 2026. In the October 2026 run this pair matched the blind judge on all 30 conversations, matched the expected grade on 14 of 15 legitimate students, held every adversarial student to a failing grade, and leaked nothing. It replaced gpt-5.6-terra / gpt-5.6-luna, which ran from August and did well on the same battery (29 of 30 with the judge, no leaks); the new pair graded more accurately and costs 12% less per conversation. On a longer three-portion assessment the two were level on cost and the older pair was slightly more consistent, which is why we ran the blind held-out set on that assessment before switching: the new pair landed 11 of 12 conversations on the expected grade there, against 8 of 12 for the old one, with no leaks.

Commercial Models

"Exact grade" is agreement with the expected grade for legitimate students; "judge exact" is agreement with the blind judge. Cost is the model fees for one bake-off conversation, which runs roughly five to seven turns; real assessments are longer, and the calculator on the front page works out per-exam costs on the production pair.

October 2026: GPT-6

GPT-6 arrived on Azure in October. We ran its two available sizes, gpt-6-sol and gpt-6-luna, against the production pair on the same battery, same day, same engine.

gpt-6-sol as evaluator with gpt-6-luna as interviewer was the strongest pair we have tested: no leaks across 30 conversations, both of its grading reads agreed with each other 97% of the time, and it graded a student who recited rehearsed knowledge N on two of three tries, which is the stricter and correct reading. gpt-6-luna on its own graded a strong student S instead of E on two of three tries and once named a rubric term the student had not used, so we rate it an acceptable evaluator and a proven interviewer. At $0.005 per conversation it is also by far the cheapest commercial pair that holds up.

The same three pairs also ran a second set of legitimate students, written blind and never used to tune the engine, and a three-portion assessment with longer conversations. On the blind set every pair landed all twelve conversations on the expected grade and matched the judge on each one. On the longer assessment the production pair agreed with the judge on 95% of conversations against 90% for both GPT-6 pairs, and gpt-6-sol / luna's two internal grading reads disagreed often enough to flag a quarter of its attempts for a human look, against a tenth for the production pair. Neither pair with gpt-6-sol as evaluator leaked a rubric phrase during questioning on any set.

Azure list prices for the models in that run, USD per million tokens:

gpt-6-sol matches gpt-5.6-terra on input and undercuts it on output; gpt-6-luna is half the price of gpt-5.6-luna. Both bill prompt-cache writes at 1.25 times input, as GPT-5.6 does.

August 2026

Struck-through models have been withdrawn by their vendor since the run. We leave the results in place because they are still what those models did, but the pair can no longer be deployed.

Older models still complete conversations but leak more: gpt-4o / mini had six guardrail failures, including quoting private assessment background to a student. That is why the interviewer runs on a current model.

Open-Weight Models

In August 2026 we evaluated eight recent open-weight models, each playing both roles. Anyone can serve these themselves, so we omit cost; price depends on hosting. The timings are from our test serving and will vary with hosting too.

Every open-weight model let answer-key phrases reach a student at least once across its six conversations, and all but one did so more than once; the worst, glm-5.2, leaked eight times, including to a cooperative student who never asked. The production pair leaked nothing on the same battery. Several older commercial models leak too, as the table above shows; leakage tracks model quality rather than license. One model, qwen3-coder-next, failed outright as a grader: it gave a student running a prompt-injection attack the top grade and returned an invalid grade for an answer-extraction attempt. The models marked "reasoning" were also slow and unreliable under load: qwen3.8-27b ran a median of 46 seconds of model time per turn, and qwen3.6-35b-a3b needed six grading-call retries and once hit our five-minute timeout on a single reply.

Our read: current open-weight models at this scale can hold a conversation and often land near the right grade, but they cannot yet keep a secret under pressure or grade reliably enough to be trusted with assessment. We will keep re-running the battery as new open models appear, and we would like one to pass.