Skip to content

00 / Notes / 001 / 2026-11-02

How to usability test an AI feature

One session with an AI feature shows one draw from a distribution. What to hold fixed, how many runs to read, how to score, and what to report.

A usability test rests on one assumption: the product behaves the same way for every participant. It is what lets you blame the design when 4 of 6 people fail the same task.

An AI feature breaks it. Ask the same question a second time and you get a different answer. A single session shows one draw from a distribution, and a failure in that session has 3 possible causes: the design, the output on that run, or the participant. The test plan has to separate them.

What a fixed system gives you

On TAIS, the Army airspace system I redesigned as a General Dynamics employee, each of 5 field sessions used a 96-step task script scored with the System Usability Scale and NASA-TLX. The software gave every soldier the same response to the same step, so when the usability score rose from 68 to 83 across rounds, I could read the change as the effect of the design. A generative feature takes that away.

Hold the conditions fixed

Fix everything you control and record it before the first session:

  • The task wording and the success criteria.
  • The starting data: the same account, records, and history for every participant.
  • The model version, the system prompt, and any setting that shapes output. If a vendor can update the model under you, record the date and treat sessions on different dates as different conditions.
  • The build of the interface around the output.

Participants can type their own prompt or use one you supply. A supplied prompt isolates the output and the interface. A free prompt tests whether people can ask for what they need. Run them as separate tasks.

Choose live or replayed output

Output stays variable. Handle it 2 ways; most studies need both.

Live runs. Each participant gets a fresh answer. This shows whether the feature holds up in real use. It cannot show why one person failed.

Replayed runs. Capture answers ahead of time and serve the same one to every participant. Output becomes a constant, and you measure the interface around it: whether people can read the answer, check it, correct it, and recover when it is wrong.

Read the spread before anyone sits down

Before recruiting, run each task prompt repeatedly against the fixed configuration and read every answer. A workable starting point, not a research finding, is at least 10 runs per task, and more when those 10 disagree. The aim is to see the spread, not to compute a statistic.

Sort each answer into 1 of 5 bins:

  1. Correct and easy to check.
  2. Correct but hard to check.
  3. Partly wrong.
  4. Wrong and stated with confidence.
  5. Refused or empty.

The tally shows what participants are likely to meet: if 2 of 10 runs are wrong, a round of 5 to 7 people may see a wrong answer once or twice, or never. It also gives you real failures to replay.

One participant failing on one live answer is a single draw. Before calling it a design finding, look for it again on other runs, or from other participants given the same replayed answer.

Score what people did with the answer

Task success alone hides the case that matters most. Judge each task against the true answer, not against what the model said, and place it in 1 of 4 cells:

OutputParticipant ended up rightParticipant ended up wrong
CorrectThe feature workedDesign problem: right answer, lost in the interface
WrongRecovery: they caught itThe dangerous cell: they accepted it

Then record 2 more things.

Verification. Did the participant try to check the answer? Could they? How long did it take? Ask how sure they are before you reveal whether the answer was right. Confidence in a wrong answer is the number to watch.

When I was a designer at Amazon Web Services, Amazon Q refused every security question, because a wrong answer in security is dangerous. My design concept, shelved before the supporting API existed, answered from the customer’s own GuardDuty findings. Design the check in, then measure whether people use it.

Taking back control. Mark the moment a participant stops following the feature: editing the output, overriding it, or finishing by hand. Note the trigger. If the design has no point where a person can step in, that is a finding before the first session.

Keep the measures you already use, such as time on task and a standard questionnaire, but report them per cell. An average across right and wrong answers describes nothing.

Test failure on purpose

Waiting for a live failure wastes sessions. Take 2 or 3 real failures from your pre-session runs, including a wrong answer stated with confidence, and replay them to every participant.

  • Use failures the system produced. If you have to write one, say so in the report.
  • Put the first seeded failure after at least 1 clean task, and vary its position across participants. A wrong answer changes how people treat every answer after it.
  • Tell participants afterward which answers were wrong.

Watch whether they notice, how long it takes, what they do next, and whether they trust the next correct answer less.

What to report

  • The configuration tested: model version, prompt version, build, and date. The findings describe that configuration and expire when it changes.
  • The spread: runs per task and the tally by bin.
  • The 4-cell counts for each task. With 6 participants, write “4 of 6”, not 67%.
  • Verification attempts, successes, and time.
  • Each point where participants took back control, with its trigger.
  • Design findings and output findings in separate lists, because different people fix them.
  • What was not tested.

What is standard and what is mine

Fixed task scripts, preset success criteria, small rounds, and standard questionnaires are ordinary usability practice. The run counts, bins, cells, and seeded failures are my approach to what ordinary practice does not cover. Treat the numbers as starting points and raise them with the cost of a wrong answer.

About this note

Delivered by Brad Shroyer as an employee of the organization named. Principal experience, not Veteran Design contracts. AI tooling supports the work; I review and own the result.

Next step

If an AI feature is heading to your users without a test plan, see how I scope AI integration work.