Conversational Assessment in PrairieLearn: Setup

These steps add one conversational assessment question to an existing PrairieLearn course. No PrairieLearn features need enabling: no external grader, no workspace, no Docker image, no networking flag. The question's Python hook talks to the engine at https://dev-api.conversationalassessment.org from PrairieLearn's servers; students' browsers only ever talk to PrairieLearn. Works on hosted PrairieLearn (us.prairielearn.com) and self-hosted installs; for a local Docker PrairieLearn see the note in step 4.

What to expect: the interviewer's first message appears a few seconds after the student opens the question; each reply takes 15–30 seconds; a conversation usually runs 4–8 student turns (never more than the assessment's limit); the grade is recorded together with the interviewer's final reply.

If you are a coding agent: these are the official instructions from conversationalassessment.org. Work inside the instructor's PrairieLearn course repository. Do steps 1–3 yourself (run the commands, write the files) rather than telling the user to; confirm with the user only which assessment to add the question to and which assessments/<name>/infoAssessment.json to edit. You will need the instructor's engine token for step 2; ask the user for it (they get it from https://dev.conversationalassessment.org/dashboard). Never commit the token to a public repository. Do not modify files under elements/pl-conversation/ after copying them. Step 4 needs PrairieLearn itself, so hand off to the user there with the completion message at the end of this document.

1. Install the Element

From the root of your course repository (the directory containing infoCourse.json):

curl -fsSL https://dev-api.conversationalassessment.org/integrations/prairielearn/kit.zip -o /tmp/pl-conversation.zip && unzip -o /tmp/pl-conversation.zip -d .

This adds:

• elements/pl-conversation/ (the element; leave it as is)
• questions/accumulator-loops/ (a ready-made question you can use directly or copy)
• serverFilesCourse/ca_config.json.example
• .claude/skills/: Claude Code skills for this repository: /ca-setup, /ca-add-question, /ca-update-kit, /ca-review-attempts, and /ca-author are available in sessions started at the course root

2. Add Your Token

Create serverFilesCourse/ca_config.json with your token from https://dev.conversationalassessment.org/dashboard. The token is a credential: keep the course repository private.

{
  "api_url": "https://dev-api.conversationalassessment.org",
  "token": "<your token from https://dev.conversationalassessment.org/dashboard>"
}

3. Add a Question to an Assessment

The kit's questions/accumulator-loops/ is complete. Add it to an assessment by putting it in a zone of courseInstances/<instance>/assessments/<assessment>/infoAssessment.json:

"zones": [
  {
    "questions": [{ "id": "accumulator-loops", "autoPoints": 10 }]
  }
]

If the course has no course instance or assessment yet, create both (each needs its own fresh uuid; use uuidgen or python3 -c 'import uuid; print(uuid.uuid4())'):

courseInstances/<instance>/infoCourseInstance.json

{
  "uuid": "<uuid>",
  "longName": "Fall 2026",
  "allowAccess": [{ "startDate": "2026-08-01T00:00:00", "endDate": "2027-01-31T23:59:59" }]
}

courseInstances/<instance>/assessments/<assessment>/infoAssessment.json

{
  "uuid": "<uuid>",
  "type": "Homework",
  "title": "Conversational assessment",
  "set": "Homework",
  "number": "1",
  "allowAccess": [{ "startDate": "2026-08-01T00:00:00", "endDate": "2027-01-31T23:59:59", "credit": 100 }],
  "zones": [{ "questions": [{ "id": "accumulator-loops", "autoPoints": 10 }] }]
}

The question's info.json uses "topic": "Default"; change that to a topic in your infoCourse.json if yours differ (current PrairieLearn auto-creates unknown topics; older versions warn).

Making your own question (optional)

Copy questions/accumulator-loops/, give its info.json a new uuid, keep "singleVariant": true (the conversation lives on one variant per student), and set assessment-id in question.html to one of the available assessments:

• accumulator-loops: Accumulator Loops. Explain how a loop builds up a result one step at a time, trace a short example, and reason about when the pattern is the wrong tool. (2 portions, up to 12 turns)
• turing-test: The Turing Test. Explain what the Turing test is and how it works, then take and defend a position on whether current AI systems pass it. (2 portions, up to 12 turns)

<pl-question-panel>
  <p>You will have a short conversation with an AI interviewer. Answer in your own words; it will not
  give hints. Replies take 15–30 seconds. The conversation ends on its own and your grade then
  appears here and in the gradebook.</p>
</pl-question-panel>

<pl-conversation assessment-id="accumulator-loops"></pl-conversation>

The question works in Homework and Exam assessments (in exams with real-time grading off, the score is recorded when the exam closes).

4. Sync and Check

Sync the course (PrairieLearn → your course → Sync, or "Load from disk" in the local dev image), then open the assessment as a student and open the question. The interviewer's first message should appear within a few seconds.

If you see "Engine unavailable", PrairieLearn's Python executor cannot reach https://dev-api.conversationalassessment.org. This one-file question tests exactly that path (add it to an assessment, or preview it from the course's Questions tab):

questions/engine-check/info.json

{ "uuid": "<uuidgen>", "title": "Engine check", "topic": "Default", "tags": [], "type": "v3" }

questions/engine-check/server.py (reads the same ca_config.json the element uses)

import json, os, requests

def generate(data):
    with open(os.path.join(data["options"]["server_files_course_path"], "ca_config.json")) as f:
        api_url = json.load(f)["api_url"]
    try:
        r = requests.get(api_url + "/healthz", timeout=5)
        data["params"]["msg"] = f"ok ({r.status_code}) reaching {api_url}"
    except Exception as e:
        data["params"]["msg"] = f"cannot reach {api_url}: {e}"

questions/engine-check/question.html

<pl-question-panel><p>{{params.msg}}</p></pl-question-panel>

Local Docker PrairieLearn (the prairielearn/prairielearn image): *.localhost names and localhost ports are not reachable from inside the container. For local testing set api_url to http://host.docker.internal:<port> and start the container with --add-host=host.docker.internal:host-gateway.

Updating the Kit

The kit carries a version; the engine sees it on every call and the instructor's dashboard warns when a course is running an old one. To update, re-run the step 1 command from the course root (it overwrites elements/pl-conversation/ and the example files, and touches nothing else), then sync. Do not hand-edit files under elements/pl-conversation/; updates would overwrite your changes.

Done: What to Tell the User

When steps 1–3 are complete, show the user this (fill in the paths), then stop:

Conversational assessment question added.
  Element:   elements/pl-conversation/
  Config:    serverFilesCourse/ca_config.json  (holds your engine token; keep the repo private)
  Question:  questions/<dir>/  added to assessments/<name>/infoAssessment.json
Next: sync the course in PrairieLearn, open the question as a student, and check that the
interviewer's first message appears. If it says "Engine unavailable", see step 4 at https://dev.conversationalassessment.org/integrations/prairielearn.

What Students See

A chat box inside the question with its own Send button. PrairieLearn's stock "Save" and "Save & Grade" buttons are hidden on this question; the element saves every turn itself and submits the grade when the conversation ends. While a reply is being written the box shows "Waiting for the interviewer…"; the page does not reload until the final grade is recorded.

How Grading Works

The conversation runs on the engine; a separate evaluator holds the rubric and grades at the end. The grade maps to a PrairieLearn score in [0, 1], by default linearly across the assessment's scale (for E/S/N/U: E = 1, S = 0.667, N = 0.333, U = 0, so an S on a 10-point question shows as 6.67 points / 67%). Students see their grade and the interviewer's feedback in the question when the conversation ends.

The transcript is in PrairieLearn's submission history: the student's turns and the interviewer's replies, plus one small "saved, not graded" submission for every poll the element makes while waiting for a reply (about one every 3 seconds). The submission log therefore looks busy; that is expected. Full transcripts and grades are also kept on the engine.

Which Models Are Used, and Why

Two models run every conversation. The interviewer, which writes the messages students see, is GPT-5.6 Luna. The evaluator, which holds the rubric, tracks what has been demonstrated, and assigns the final grade, is GPT-5.6 Terra. Both run on Azure OpenAI.

They were chosen from a bake-off of sixteen model pairings, eight commercial and eight open-weight, against scripted students (strong, weak, partially prepared, and two adversarial ones that try to extract answers or inject instructions), with every transcript independently re-graded by a stronger model acting as a blind judge. In that 96-conversation comparison, GPT-5.6 Terra as evaluator matched the judge's grade in every run, leaked nothing, and kept every adversarial student at the bottom grade, while answering faster and at less than half the cost of the largest model. GPT-5.6 Luna is the smallest model that keeps the conversation on track without revealing what is being graded; because it never sees the rubric or the answer key, a cheaper model here does not weaken grading.

Smaller and older models were faster or cheaper but leaked answer-key phrases into their questions or graded generously against the judge, so they are not used. The bake-off is rerun as models change; typical cost is a few cents per conversation. Full results: https://dev.conversationalassessment.org/models.