Measuring conversational quality

How do you determine, based on evidence, whether an assistant’s answers are acceptable for your profession?

1960 € per session, excl. VAT

In short

This 7-hour day, excluding breaks, enables you to build an evaluation protocol tailored to business needs: compile a corpus of questions, define a scoring rubric, calibrate evaluators and compare two versions of responses. You leave with a practical set of documents for deciding which corrections to prioritise and monitoring quality without confusing fluency with accuracy.

Why this training?

A convincing response may contain a fabricated reference, omit a technical restriction or recommend an action that conflicts with a manual. An overall assessment cannot pinpoint these flaws or determine their severity. A documented evaluation links each judgement to a business expectation and supporting evidence. It makes disagreements open to discussion, distinguishes disqualifying errors from writing flaws and prevents a favourable average from concealing a dangerous response. Running case study: The fictitious organisation « Briselis Capteurs » uses a product support assistant. Two versions of responses differ across twelve fictitious requests concerning compatibility, installation and returns. The team must decide which corrections are needed before release, using the manuals and service rules provided. Transfer to the workplace: Over the following two weeks, apply the rubric to an anonymised sample from your work, have critical cases reviewed by a subject matter expert and record discrepancies before any further comparison.

Objectives of this training

  • Compile a corpus covering common requests, ambiguities and critical cases
  • Define scorable criteria with examples and acceptance thresholds
  • Score responses by linking each discrepancy to a source
  • Measure disagreements between evaluators and adjudicate them
  • Compare two versions and prioritise corrections

Who is it for and what are the prerequisites?

Roles and tasks concerned

  • Product managers and quality managers for conversational assistants
  • Support subject matter experts and business analysts responsible for validating responses

Prior skills

  • Use a generative assistant regularly and be able to formulate a structured instruction.
  • Understand the documentation requirements and risks of the field represented.
  • Be able to filter a table and calculate a proportion.

What you will work on

  1. Define the expected quality

    Objective: Link acceptability to support risks. Accuracy, relevance, completeness, confidentiality and appropriate abstention; the difference between style and reliability.

  2. Build the reference corpus

    Objective: Cover situations that genuinely differentiate performance. Common cases, ambiguous requests, missing information; source-backed expectations rather than a single wording.

  3. Make scoring reproducible

    Objective: Turn criteria into observable rules. Scoring scale, anchor examples, weighting and rejection independent of the overall score.

  4. Calibrate evaluators

    Objective: Reduce conflicting interpretations. Proportion of agreement, factual justification, preference bias; limitations of AI-assisted judgement.

  5. Compare without concealing regressions

    Objective: Reach an evidence-based decision between two versions. Scores by criterion, acceptance rate, critical errors and limitations of a small sample.

  6. Correct, then re-evaluate

    Objective: Verify an improvement without adapting solely to known examples. Prioritisation of flaws, traceability of changes and checks on reserved cases.

What you take away

  • Reference corpus with expected outcomes and sources
  • Scoring rubric with anchor examples and rejection rules
  • Response comparison table and adjudication log
  • Prioritised list of corrections and re-evaluation protocol

Method and assessment

  • Short input sessions followed by immediate practice using the running case study.
  • Individual work followed by work in pairs, with peer review of the outputs.
  • Guided group discussion: what works, what needs correcting and what remains a human decision.
  • Preparation: Select a business procedure that may be shared and identify three response errors considered critical. Check access to the tools authorised by the organisation; work only with fictitious or anonymised cases.

Observable criteria: Each case includes a verifiable expectation and an identified source. ; Each scoring level has a concrete example. ; Every critical error triggers the defined rejection rule. ; Disagreements are counted and then adjudicated with justification. ; The comparison uses the same corpus and explicitly states its limitations..

Frequently asked questions

Are programming skills required?

No, a scoring table is sufficient for the exercises.

Can AI assign the scores?

AI can propose an analysis; critical decisions remain subject to human verification.

Is there a perfect reference response?

The expectations describe verifiable elements, not mandatory wording.

Is the score sufficient to authorise release?

No: disqualifying errors, corpus coverage and subject matter validation remain decisive.

Last updated: