How to use an AI question generator without losing quality

An AI question generator can turn a blank page into a draft question bank quickly. That is useful, but it is not the same as producing a valid assessment. A fluent question can still test the wrong outcome, contain an ambiguous answer or use distractors that give the answer away.

The useful question for an assessment team is therefore not “Can AI write exam questions?” It is “What process turns an AI draft into a question we are prepared to score?”

Recent research gives a measured answer. A 2026 field study involving nearly 1,700 students found that AI-generated questions produced through iterative critique and revision performed comparably to expert-created questions in that sample.

A separate systematic review in medical education found no overall difference in difficulty or discrimination, but also reported substantial variation between studies and limited evidence about equity. Its authors positioned AI as a supplement to expert development, with provenance, monitoring and human approval.

That distinction matters. The strongest workflow uses AI to accelerate drafting while keeping assessment decisions with people who understand the subject, the candidates and the consequence of the result.

1. Start with the decision the assessment must support

Do not begin with “Write 20 questions about workplace safety.” Begin with the learning outcome and the evidence a correct response should provide.

For each group of questions, define:

• the knowledge or skill being assessed;
• the level of reasoning required;
• the audience and expected prior knowledge;
• the permitted source material;
• the question type and approximate difficulty; and
• what a correct answer should demonstrate.

This makes the prompt a small assessment blueprint rather than a request for trivia. It also gives the reviewer something concrete to check. A polished question that does not support the intended decision is still the wrong question.

This is especially important now that many conventional assessments are effectively open book to AI-assisted candidates. UNESCO argues that assessment in the AI era should place more weight on critical thinking, synthesis and authentic demonstrations of understanding. AI-assisted authoring should help teams build those questions, not simply produce more recall items faster.

2. Generate from approved source material

A topic-only prompt asks a model to rely on broad background knowledge. That may be acceptable for brainstorming, but it is weak evidence for a scored assessment tied to a course, policy or certification standard.

A stronger workflow starts with the material candidates were expected to learn: a course manual, slide deck, policy, case study, diagram or selected passage. The generated question should remain traceable to the relevant part of that source.

Source grounding improves review in three ways:

1. the reviewer can verify the answer against the organisation’s material;
2. disagreements can be resolved against a known reference; and
3. outdated or unsupported content is easier to identify before publication.

It does not remove the need for subject expertise. A source can itself be incomplete, and a model can still interpret it badly. Grounding makes the draft inspectable; it does not make it automatically correct.

3. Use a question blueprint instead of accepting a pile

Twenty questions generated in one undifferentiated batch often repeat the same fact in different words. Before generation, specify the mix you actually need.

A practical blueprint can include rows for:

• question type;
• number of questions;
• target difficulty;
• topic or source section;
• whether a visual is required; and
• the intended evidence of learning.

Balance matters more than volume. A ten-question assessment with deliberate coverage is usually more useful than fifty near-duplicates. The blueprint also makes omissions visible: if every proposed item tests recognition, the team can add application or judgement questions before the paper reaches candidates.

4. Review the stem, answer, distractors and evidence separately

Do not review an AI-generated multiple-choice question as one block. Inspect four parts.

The stem: Is there one clear task? Remove unnecessary context, accidental clues and wording that tests reading complexity instead of the intended knowledge.

The correct answer: Is it fully supported by the approved source? Check units, dates, exceptions and jurisdiction-specific wording rather than relying on plausibility.

The distractors: Are the wrong options plausible to someone who has not mastered the material, while remaining demonstrably wrong? Avoid jokes, grammatical mismatches, overlapping options and one answer that is much longer or more precise than the others.

The evidence: Can the reviewer locate the source passage, figure or table that supports the answer and explanation? If not, return the candidate for revision or discard it.

This is where human judgement earns its place. AI is good at producing fluent options; fluency can hide a weak distinction between them.

5. Treat diagrams and images as evidence, not decoration

Some skills cannot be assessed well from text alone. A safety sign, chart, equipment diagram, radiograph or network topology may be the actual evidence a candidate must interpret.

When source documents contain useful visuals, keep the relationship between the image, the question and the answer explicit. Confirm that:

• the relevant figure was selected rather than a decorative image;
• labels and important details remain legible;
• the question does not reveal its answer through alt text or a caption;
• the explanation refers to what is genuinely visible; and
• the visual can be used under the organisation’s rights and privacy rules.

Generating a new visual should be a deliberate authoring choice, not an automatic embellishment. Reusing an approved source image is often better when exact technical detail matters.

6. Keep a human approval gate before publication

An AI draft should enter a review queue, not an active exam. The reviewer needs to be able to edit the stem, answers, explanation, difficulty and visual before accepting the question.

For consequential assessments, use at least two perspectives where practical: a subject-matter reviewer for correctness and an assessment reviewer for clarity, coverage and bias. Record who approved the final item and preserve the source reference used in that decision.

The goal is not to hide that AI assisted the drafting process. It is to keep accountability clear. The organisation publishing and scoring the question remains responsible for it.

7. Evaluate questions after candidates answer them

Review does not end when the exam opens. Candidate responses reveal problems that are difficult to see in advance. A consistent delivery, monitoring and reporting workflow gives the assessment team a place to examine those results after each sitting.

Monitor:

• the proportion of candidates answering correctly;
• whether stronger candidates are more likely to answer correctly;
• distractors that nobody selects;
• questions that produce unusual complaint or review rates;
• performance differences that may indicate unclear or culturally specific wording; and
• repeated edits or removals associated with a source, prompt or authoring pattern.

One study cannot guarantee the quality of the next generated question. The 2026 field evidence is encouraging precisely because it evaluated questions with real learners and an iterative refinement process. Your own response data should become the next review cycle.

A quality checklist for AI-generated exam questions

Before adding an AI-assisted question to a paper, confirm that:

• it maps to a defined learning outcome;
• it is grounded in approved source material;
• its type and difficulty fit the assessment blueprint;
• the stem asks one unambiguous question;
• the correct answer and explanation are supported by evidence;
• distractors are plausible, distinct and demonstrably wrong;
• any visual is relevant, legible and permitted for use;
• a qualified person has reviewed and approved it; and
• the team will evaluate item performance after delivery.

AI question authoring is most valuable when it shortens the distance from source material to a reviewable draft. It should not shorten the review itself.

examina.io’s AI-assisted Designer follows that controlled path: authors choose current text, saved resources or uploaded source files; define the required question mix; generate source-backed candidates with evidence references; and review each candidate before adding it to the paper. Documents and relevant visuals can be used together, while failed or unsupported candidates do not become exam content.

Read the AI question-authoring guide for the complete workflow, or create an account to open the Designer and build an assessment from your own approved material.

Sources

• Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study
• Can large language models generate exam questions comparable to humans?
• What’s worth measuring? The future of assessment in the AI age