Model Evaluations

The engine uses two models: an evaluator that holds the rubric and decides what has been demonstrated, and an interviewer that talks to the student and never sees the answer key. A model has to do real work in both roles. Before we trust a model with either one, it runs our bake-off.

How We Test

Each candidate pair runs full assessments against 6 simulated students, including 2 adversarial ones that try to extract answers, inject instructions, or bluff their way to a grade. Every transcript is then re-graded by gpt-5.6-sol, a stronger model used only as a blind second grader. Grades sit on a four-level E/S/N/U scale, Excellent to Unsatisfactory. We report how often the engine's grade matches the expected grade, whether adversarial students were held to failing grades, exact agreement with the blind judge, guardrail failures (any turn where the interviewer let graded material slip or otherwise misbehaved), and format errors (grading replies that came back malformed and had to be retried). With only four grade levels we count exact matches; agreement with the expected grade within one level is a bar every model clears, so we do not report it. Every conversation in every run completed. Last run: August 2026.

Production Models

We currently run gpt-5.6-terra as the evaluator and gpt-5.6-luna as the interviewer. In the latest bake-off this pair matched the blind judge on every conversation, held all adversarial students to failing grades, and leaked nothing. It is also fast: about 7 seconds of model time per turn.

Commercial Models

Results for the commercial models we have evaluated. "Exact grade" is agreement with the expected grade for legitimate students; "judge exact" is agreement with the blind judge. Cost is the model fees for one bake-off conversation, which runs roughly four to seven turns; real assessments are longer, and the calculator on the front page works out per-exam costs on the production pair.

Older models still complete conversations but leak more: gpt-4o / mini had six guardrail failures, including quoting private assessment background to a student. That is why the interviewer runs on a current model.

Open-Weight Models

We evaluated eight recent open-weight models, each playing both roles. Anyone can serve these themselves, so we omit cost; price depends on hosting. The timings are from our test serving and will vary with hosting too.

Every open-weight model let answer-key phrases reach a student at least once across its six conversations, and all but one did so more than once; the worst, glm-5.2, leaked eight times, including to a cooperative student who never asked. The production pair leaked nothing on the same battery. Several older commercial models leak too, as the table above shows; leakage tracks model quality rather than license. One model, qwen3-coder-next, failed outright as a grader: it gave a student running a prompt-injection attack the top grade and returned an invalid grade for an answer-extraction attempt. The models marked "reasoning" were also slow and unreliable under load: qwen3.8-27b ran a median of 46 seconds of model time per turn, and qwen3.6-35b-a3b needed six grading-call retries and once hit our five-minute timeout on a single reply.

Our read: current open-weight models at this scale can hold a conversation and often land near the right grade, but they cannot yet keep a secret under pressure or grade reliably enough to be trusted with assessment. We will keep re-running the battery as new open models appear, and we would like one to pass.