Assessing with evidence
How can you verify that an assessment measures the target competencies and produces consistent decisions across assessors?
3500 € per session, excl. VAT
In short
Over two days, totalling 14 hours excluding breaks, you build an assessment prototype aligned with observable competencies. Generative AI helps prepare assessment tasks, criteria and feedback, under human oversight. You test scoring consistency, identify biases and formalise a pilot before any rollout.
Why this training?
A score can conceal a poorly aligned assessment task, an ambiguous marking scheme or requirements that vary between markers. These shortcomings undermine decisions about whether competencies have been achieved and make remedial measures difficult to justify. AI speeds up the drafting of scenarios and rubrics, but may introduce questionable answers or biases. The training structures their verification through observable evidence, human calibration and tests that challenge the proposed outputs. Running case study: The fictitious organisation Avelcor Assistance is preparing its advisers to handle complaints. Its theoretical questionnaire does not demonstrate their ability to resolve a case. You design a practical simulation, a rubric and a pilot using entirely fictitious responses. Transfer to the workplace: Subsequently, arrange double marking of a small, authorised sample, analyse disagreements and decide on revisions before extending the approach. Appoint someone responsible for validation and retain successive versions.
Objectives of this training
- Link each competency to evidence and an assessment task
- Build a marking scheme with observable levels
- Verify proposals produced using AI
- Measure discrepancies between two markers
- Formalise a pilot protocol and its decision rules
Who is it for and what are the prerequisites?
Roles and tasks concerned
- Educational leads and learning designers developing assessments
- Subject matter leads and quality managers responsible for validating acquired competencies
Prior skills
- Be able to structure and verify a prompt for generative AI
- Have previously designed a learning activity or marked an assessment
- Be able to calculate proportions in a spreadsheet
What you will work on
Frame the decision
Objective: Define the conclusions that the assessment can support. Diagnostic, formative, summative; consequences of an error.
Define the evidence
Objective: Make competencies verifiable. Alignment, coverage, behaviours and records.
Choose the assessment tasks
Objective: Avoid measuring memorisation alone. Authenticity, difficulty, assessment conditions.
Prototype with AI
Objective: Produce scenarios that can be checked. Provided context, constraints, verification of model answers.
Build the marking scheme
Objective: Make proficiency levels explicit. Descriptors, weightings, critical errors, justified thresholds.
Examine fairness
Objective: Identify barriers unrelated to the competencies. Linguistic biases, accessibility, permitted use of AI.
Calibrate markers
Objective: Compare interpretations of the marking scheme. Independent double marking and adjudication.
Measure robustness
Objective: Identify unstable criteria. Exact agreement, absolute discrepancies, limitations of a small sample.
Prepare feedback
Objective: Make the decision explainable. Evidence, remedial action, appeals and human validation.
Plan the pilot
Objective: Make expansion conditional on the test results. Sample, indicators, thresholds, retention of records.
What you take away
- Competency, evidence and assessment task matrix
- Assessment prototype with instructions, model answers and rubric
- Test report and revision log
- Pilot plan with responsibilities and indicators
Method and assessment
- Short instructional inputs followed by immediate practice using the case study that runs throughout.
- Individual work followed by work in pairs, with peer review of the outputs.
- Facilitated group discussion: what works, what needs correcting and what remains a human decision.
- Preparation: Bring an anonymised reference framework and assessment, or use the fictional dossier provided. Check which tools your organisation authorises and the rules for transmitting data.
Observable criteria: Each target competency corresponds to observable evidence. ; Each level of the marking scale has distinct descriptors. ; Discrepancies in double marking are calculated and explained. ; Generated content is checked against the reference framework. ; The pilot specifies thresholds, responsibilities and conditions for revision..
Frequently asked questions
Does AI award the final mark?
No: any suggestions are still checked; the decision rests with an authorised assessor.
Can we use actual submitted work?
Only if anonymised, in accordance with internal rules and using authorised tools.
Does the test prove definitive validity?
No: it identifies weaknesses; further testing is still necessary.
Is a specialist tool needed?
An authorised assistant, a spreadsheet and shared documents are sufficient for the prototype.
Last updated: