Reliability after modification

How can you check that an AI assistant stays reliable after a change?

  • ChatGPT
  • Claude
  • Gemini

1960 € per session, excl. VAT

In short

AI champions and assistant owners learn to check an assistant after every change: build a reference set with edge cases, compare two versions on a fixed corpus, classify errors and decide whether to keep or roll back. Deliverables: a regression test set, a scoring grid and a comparison report with a documented version decision.

Why this training?

The problem
Modified AI assistants can degrade existing performance without quick detection by the team.
What it costs today
Unexpected errors appear, users lose trust, and decisions made based on AI outputs are fragile.
Why this skill
This skill is necessary to maintain the quality and relevance of AI assistants after each evolution.
What you gain
The participant and the company ensure that modifications to AI assistants do not compromise result quality or operational reliability.

Objectives of this training

  • Define what a reliable assistant means in its business context with clear criteria.
  • Build a reference set including edge cases to evaluate assistant performance.
  • Design and use a scoring grid to consistently evaluate AI responses.
  • Compare two versions of an AI assistant to identify regressions and improvements.
  • Make an informed decision to maintain or roll back a new AI assistant version.

Who is it for and what are the prerequisites?

Roles and tasks concerned

  • AI Referents
  • Internal Assistant Owners
  • Product Quality Teams

Prior skills

  • AI Fundamentals and AI Prompting completed, or equivalent experience (a learning prerequisite, not a registration condition).
  • Possess an existing assistant or prototype
  • Understand prompting basics
  • Know the assistant's business use cases
  • A paid subscription to an AI solution offering the features used, the organisation’s authorisation for third-party tools, and the actual accounts and access rights.

To prepare before the session

  • Have an existing AI assistant or prototype
  • Select the main tool (ChatGPT, Claude, or Gemini) during planning
  • Prepare the key business use cases of the assistant to be tested

What you will work on

  1. Define Reliability

    The participant accurately defines the reliability expectations for an AI assistant within their business context.

  2. Build a Reference Set

    The participant creates a set of representative test cases, including edge cases, to evaluate the assistant.

  3. Score with a Grid

    The participant applies a standardized and reproducible method to evaluate the quality of the assistant's responses.

  4. Compare Two Versions

    The participant can objectively assess performance differences between two versions of an AI assistant.

  5. Decide and Version

    The participant establishes a clear procedure for making decisions regarding AI assistant updates.

What you take away

  • Non-regression test set
  • Scoring grid
  • Comparison report and version decision

Method and assessment

  • Work on an existing assistant or supplied prototype
  • Fixed corpus replayed at every version
  • Calibrated human scoring, with any evaluator model only as a reviewed aid

The set-up is assessed on reference set coverage, clarity of expected results, reproducibility of the grid, rigour of the fixed-corpus comparison and justification of the version decision.

Frequently asked questions

What concrete result does this course deliver?

The team has a reusable regression process to decide every new version on evidence. Deliverables: regression test set, scoring grid, comparison report and version decision.

Who is this course for and what are the prerequisites?

It is designed for: internal ai champions, owners of assistants or internal gpts, quality and product teams. An existing assistant or prototype, basic prompting knowledge and familiarity with the assistant's business cases. One main tool (ChatGPT, Claude or Gemini) is chosen during scoping; mastering all three at once is not the aim.

What data and access need to be arranged?

Test cases are fictional or anonymised. The assistant used must be authorised in the organisation; access is checked before the session. Required tools and licences are checked before the session; no subscription is included in the price.

What practical exercise is completed?

Participants design a reference set with edge cases, expected results and criteria, compare two versions on a fixed corpus, classify errors, measure correction cost and stability, then decide to keep or roll back.

What are the limits and who validates the results?

A test set reduces regression risk without removing it. The course does not build the assistant and does not delegate all scoring to another LLM: the version decision stays with the assistant's owner.

Can the course be adapted to our line of work?

During scoping, the main tool and working assistant are chosen: document assistant, internal GPT, writing agent or support assistant.

How is the session organised and what does it cost?

Full day, private in-person session for up to 12 participants. The fee is €1,960 excluding VAT per session. Date and location are agreed during scoping; each session runs in a single language: French, Dutch or English.

Related courses

Last updated: