Getting Started

Everything from creating an account to a graded conversation in your PrairieLearn course, one click at a time. About an hour of your own time, plus two short waits for approvals.

What you are building, in ninety seconds · 0:58

By the end, one question in your PrairieLearn course will hold a short, graded conversation with each student about a topic you chose, with criteria you set. Students talk to an AI interviewer that never sees the answer key. A separate evaluator grades against your rubric. The grade lands in the PrairieLearn gradebook like any other question.

You do not need to write code or use a terminal. Where files have to be created, an AI assistant does it for you, and there is a browser-only route for every step.

Before You Start

Have these ready. Each is quick to get, but two involve waiting for someone else.

1. Create Your Account

Signing in and getting approved · 1:12

Your account on this site is where you get the two credentials the rest of the guide uses, and where your assessments and your students' results appear. Creating it is the same as signing in for the first time.

  1. Go to www.conversationalassessment.org and click Login in the top right.
  2. Illinois: click Sign in with Illinois. Microsoft asks for your NetID, password, and Duo, then sends you back here.
    Everyone else: type your institutional email address in the box and click Email me a link. The page says to check your email.
    1. Open the email from conversationalassessment.org. If it has not arrived in two minutes, look in Junk or Spam.
    2. Click the link. A page titled Confirm sign-in opens.
    3. Click the Sign in button. This extra click exists because some university mail systems open every link to scan it, and that would use up the link before you saw it.
    The link works once and expires after an hour. If it has expired, request another from the sign-in page.
  3. You land on your Dashboard. It says Waiting for approval. We approve requests by hand and email you, usually within a few days. There is nothing to do here until then. Meanwhile, request your PrairieLearn course if you have not, and read on.
  4. When the approval email arrives, sign in again the same way. The dashboard now has sections called Your Token, Set Up With Your Coding Agent, Your Assessments, and Author With Your AI. Accounts outside Illinois also see a red notice saying students cannot start yet, and a section called Your AI Endpoint. That is Part 2. Illinois accounts skip to Part 3.

2. Connect Your Own AI

Connecting and testing your AI endpoint · 1:21

Illinois accounts skip this part. Everyone else runs conversations on an AI account they control. That keeps the cost on your bill, a few cents per conversation, and means student text never passes through a provider you did not choose.

  1. Get an API key. The key is a long string that lets this site call the AI on your behalf. Copy it the moment it is shown; providers show it once.
    • OpenAI: platform.openai.com, then API keys, then Create new secret key. The base URL is https://api.openai.com/v1.
    • Anthropic: console.anthropic.com, then API Keys, then Create Key. The base URL is https://api.anthropic.com.
    • Azure OpenAI: ask your IT department for the resource's URL and a key. The URL looks like https://your-resource.openai.azure.com.
  2. On your dashboard, scroll to Your AI Endpoint. Choose the provider type (OpenAI-compatible, Azure OpenAI, or Anthropic), paste the base URL into the first box and the key into the second, then click Save and find models. The key is stored encrypted and is never shown again.
  3. The page lists every model your endpoint offers, with its price and, where we have tested it, how it did in our bake-off. Choose an evaluator (the model that holds the rubric and grades) and an interviewer (the model that talks to students). Prefer models the list marks as tested. Click Save models and test. A short test call runs and reports.
  4. Under Check These Models, click the button to run three simulated students against your endpoint: a strong one, a weak one, and one that tries to extract the answers. This takes a few minutes and costs a few cents on your account. The verdict appears on the dashboard (pass, warn, or fail) with the transcripts. Do this before real students arrive.

Costs: a page listing every model we know, with list prices and bake-off results, is at Model Evaluations. The dashboard estimates what a conversation will cost on the models you chose.

3. Install Claude and Connect It

Adding the authoring connector to Claude · 1:13 · preliminary cut: parts that happen outside the browser are shown as slides for now

You write assessments by talking to your own AI assistant, which is connected to this site so it can save, check, and test what you write together. The guide uses Claude. Any assistant that supports custom connectors (the standard is called MCP) works the same way; the walkthrough below names Claude's menus.

  1. Install Claude Desktop. Download it from claude.com/download (Mac or Windows), install it like any program, open it, and sign in with your Claude account. You can do Parts 3 and 4 in the browser at claude.ai instead; the desktop app is what Part 5 uses.
  2. Generate your connector. On your dashboard, scroll to Author With Your AI and click Generate authoring connector. A box appears with a prompt that contains a web address of the form https://api.conversationalassessment.org/mcp/caa_…. That address is your private connector. It is shown once. Keep this tab open until the next step is done.
  3. Add it to Claude. In Claude, open Settings, choose Connectors (under Customize), and click Add, then Add custom connector. Name it Conversational Assessment, paste the address from the dashboard into the MCP server URL field (just the address, from https to the end of the long code), and click Continue. On the second step choose No sign-in (the address itself is the key) and click Add. Connectors belong to your Claude account, so one added at claude.ai also appears in the desktop app, and one added in the desktop app appears on the web.
  4. Test it. Start a new chat. Make sure the connector is switched on for the chat (the tools menu next to the message box lists it). Type:

    You: Using the Conversational Assessment connector, call get_guide and then list the assessments in the library.

    Claude reads the guide and lists the shared library, which includes an intro-programming assessment called accumulator-loops. If Claude says it has no such tool, check the connector is enabled for this chat. If the address was mistyped, click Rotate authoring connector on the dashboard to get a new one and add it again; the old one stops working.

The connector can read and write your assessments and read your students' transcripts. It cannot run student attempts, and it is separate from the course token in Part 5. Rotating one does not affect the other.

4. Write Your First Assessment

Writing an assessment with Claude · 1:32 · preliminary cut: parts that happen outside the browser are shown as slides for now

An assessment is a topic broken into two or three portions, each a thing you want the student to talk through. Each portion has criteria (what a student must demonstrate), a rubric for each criterion, a private answer key, and a few follow-up angles. There is also a short description of who the interviewer is. Claude knows the format and the rules; you bring the topic and the judgement about what counts. Budget thirty to sixty minutes for the first one.

  1. On the dashboard, in Author With Your AI, click Copy prompt and paste it into a new Claude chat. If the box has gone (it is shown once), type this instead:

    You: Using the Conversational Assessment connector, call get_guide and help me write a conversational assessment for my course.

  2. Claude reads the guide and asks about your topic, what a student who understands it should be able to explain, and how you want to grade (the default is four levels, E, S, N, U, which record in PrairieLearn as 100, 67, 33, and 0 percent). Answer in plain language. For example:

    You: Second week of intro Python. I want to know whether they understand how an if/elif/else chain picks one branch, and whether they can trace a short example by hand.

    Claude: Two portions, then. First, explain how a chain chooses a branch, with criteria for describing the top-to-bottom check and for what happens when no condition is true. Second, trace a five-line example. Here is a draft of the criteria and rubrics; tell me what you would change.

  3. Read the criteria Claude proposes. A good criterion names what the student does ("traces a concrete example step by step"), not the answer. The answer lives in a private answer key the interviewer never sees. Push back where the pedagogy is wrong; Claude will not know your course.
  4. Claude checks the draft (you will see it call validate_assessment and lint_assessment) and fixes what the checks flag, such as a criterion that gives the answer away. Then it saves the assessment (create_assessment) under a short name called a slug, for example conditionals. Write the slug down. Part 5 uses it.
  5. Simulate students. Ask Claude to run three simulated students against it: a good one (expect the top grade), a weak one (expect a low grade), and one that tries to get the interviewer to reveal the answers (expect the bottom grade and no leaks). Each run takes a minute or two and costs a few cents. Read the transcripts Claude shows you and ask yourself: does the interviewer ask what I intended, and is the grade the one I would give? Tell Claude what to change. It updates the assessment and simulates again.
  6. On your dashboard, Your Assessments now lists it, marked yours. Click the slug to see the document and any remaining lint notes. Nothing reaches students until a PrairieLearn question refers to it, which is Part 5.

Two habits that pay off: keep the first assessment small (two portions, two or three criteria each), and always read the simulated transcripts before students see it. Publishing is optional and separate: a published assessment lets other instructors run it by slug, and shares the interviewer's persona and structure with them, but never the answer key.

5. Put the Question in Your Course

Adding the question to PrairieLearn · 2:18 · preliminary cut: parts that happen outside the browser are shown as slides for now

PrairieLearn stores every course as a set of files. Adding the question means putting a small folder of files into your course, adding a file that holds your course token, and listing the question in an assessment. There are two ways to do that. Path A has Claude do the file work in a copy of your course on your computer, and is what we recommend. Path B installs nothing and uses the GitHub and PrairieLearn websites only.

Path A: Let Claude Do the File Work

  1. Get a copy of your course on your computer. Install GitHub Desktop and sign in with the GitHub account that PrairieLearn gave access to your course repository. Choose File › Clone repository, find your course (its name starts with pl-, and PrairieLearn shows it on your course's Settings page), and click Clone. Note the folder it chose, usually under Documents.
  2. Open that folder in Claude Desktop. Open Claude Desktop and click the Code tab at the top. Click + New Session. In the box where you type, set the Project folder to the folder GitHub Desktop created. Leave the permission mode on the setting that asks before running commands; Claude will explain each step and wait for you to approve it.
  3. Generate your course token. Back on your dashboard, under Your Token, click Generate token. The token appears for fifteen minutes, and the box called Set Up With Your Coding Agent now shows a prompt whose link carries the token. Click Copy prompt. Do this after step 2, not before: the link stops working after fifteen minutes.
  4. Paste the prompt into the Code session and press Enter. If you wrote your own assessment in Part 4, add one sentence first: Make the question use my assessment your-slug. Claude fetches the instructions, downloads the kit into your course folder, writes the token file, and asks which homework or exam to add the question to. Name one, or say create a new homework called Conversation practice. Approve each action when asked. It finishes with a short summary that starts Conversational assessment question added.
  5. Send the changes to GitHub. Claude asks whether to commit. Say yes and ask it to push. Or, in GitHub Desktop, type a summary such as Add conversational assessment question, click Commit to main, then Push origin. Course repositories are private, which is what keeps your token private.
  6. Sync in PrairieLearn. Open your course, click Sync in the sidebar, and click Pull from remote git repository. Wait for the green result. A red result names the file that is wrong; paste the message to Claude.

Path B: Browser Only, No Installs

  1. Download the kit: kit.zip. Unzip it (double-click on a Mac; right-click and Extract All on Windows). Inside are folders called elements, questions, and serverFilesCourse. Ignore any others.
  2. Upload the folders to GitHub. Open github.com, go to your course repository, click Add file, then Upload files. Drag the elements folder and the questions folder from the unzipped kit onto the page (drag the folders themselves; GitHub keeps the folder names). Click Commit changes.
  3. Get your token file. On your dashboard, under Your Token, click Generate token, then Download ca_config.json. Your browser saves a small file with that name. The download works for fifteen minutes after generating.
  4. Upload the token file to PrairieLearn. In PrairieLearn, open your course, click Files in the sidebar, click Add new server files, choose the downloaded ca_config.json, and upload it. PrairieLearn places it in the right folder and sends it to GitHub for you.
  5. Sync. Click Sync in the sidebar, then Pull from remote git repository, and wait for the green result.
  6. Add the question to an assessment. In the course instance, open Assessments and open one (or click Add assessment and create a Homework called Conversation practice). On its Questions tab click Edit, Add zone, then Add question, search for accumulator-loops, pick it, and click Save.
  7. If you wrote your own assessment in Part 4: in the course sidebar open Questions, open accumulator-loops, go to its Files tab, click Edit next to question.html, change assessment-id="accumulator-loops" to your slug, adjust the sentence above it that introduces the topic, and click Save.

Either way, the question's folder and the element are copies from the kit and should not be edited by hand: when the kit is updated, your dashboard tells you, and updating overwrites those files.

6. Try It as a Student and Read the Results

A conversation, a grade, and the transcript · 1:40 · preliminary cut: parts that happen outside the browser are shown as slides for now
  1. In PrairieLearn, open the user menu in the top right and choose Student view without access restrictions. Open the assessment, then the question. The interviewer's first message appears after a few seconds.
  2. Answer in your own words. Each reply takes fifteen to thirty seconds; while you wait the box says Waiting for the interviewer. The conversation ends on its own after a handful of turns. Your grade and the interviewer's feedback appear in the question, and the score appears in the gradebook: on the default scale an E is 100 percent and an S is 67.
  3. If you later change which questions the assessment holds, click Regenerate your assessment instance (the yellow banner only staff see) before trying again. Switch back to staff view from the same menu when you are done.
  4. Where the record lives. The gradebook has the score. The question's submission history has every turn of the transcript. It also holds many small "saved, not graded" rows: those are the question checking for a reply every few seconds while the student waits, and are expected.
  5. Ask Claude how students did. In a chat with the connector on, ask:

    You: How did students do on conditionals? Show me the grade distribution and read me the weakest conversation.

    Claude lists the attempts, summarizes the grades, and walks through transcripts with you. Your dashboard also counts attempts per assessment. To change a grade, edit the score in PrairieLearn as you would for any manually graded question.
  6. Tell students what to expect. Replies take fifteen to thirty seconds. The interviewer does not give hints and does not confirm answers. The conversation ends by itself. The grade appears when it does. Everything they type is treated as an answer, including requests to the AI. The question works in exams too; in an exam with real-time grading off, the score is recorded when the exam closes.

7. When Something Goes Wrong

The question shows "Engine unavailable".
PrairieLearn's servers could not reach this site. On us.prairielearn.com this is rare and usually passes; try again in a minute. On a PrairieLearn your institution hosts, ask its administrators to allow outgoing connections to api.conversationalassessment.org. The PrairieLearn page has a one-file test question that checks this.
The question fails to render and mentions an unknown attribute.
The question was written for an older kit. Your dashboard will also say the kit is out of date. In a Claude Code session in your course folder, type /ca-update-kit; or repeat the upload in Path B. Then sync.
The sign-in link says it no longer works.
Links work once and expire after an hour. Request a new one from the sign-in page.
Still "Waiting for approval" after a few days.
Email challen@illinois.edu. Say which address you signed in with.
Claude says it has no such tool, or the connector shows as disconnected.
Check the connector is switched on for the chat. If the address was mistyped or lost, click Rotate authoring connector on the dashboard and add the new address; the old one stops working immediately.
The setup prompt's link has expired.
Links carry the token for fifteen minutes. Click Rotate token to get a fresh prompt. The old token stops working, so if the question was already installed with it, the new token has to replace the old one in serverFilesCourse/ca_config.json; Claude will do that if you ask.
Students see "Waiting for the interviewer" for more than a minute.
The AI endpoint is slow or down. Accounts with their own endpoint: click Test again under Your AI Endpoint. Illinois accounts: email us. The student's conversation is kept, and they can reload the page and continue when the endpoint returns.
The dashboard says the course kit is out of date.
Update as for the unknown-attribute problem above. Your token and questions are untouched by an update.

Anything else: challen@illinois.edu. Questions about PrairieLearn itself are answered fastest on the PrairieLearn Slack.