Measuring hybrid search relevance
How can you demonstrate that hybrid search improves relevance without masking its regressions?
3500 € per session, excl. VAT
In short
Over two days, comprising 14 hours excluding breaks, build an evaluation prototype comparing lexical, vector and hybrid search. You prepare a corpus and relevance judgements, measure results by query category, then formalise a reasoned decision and a pilot test plan, without aiming for production-scale deployment.
Why this training?
An average score can conceal a failure to retrieve an exact reference or a critical procedure. Comparing approaches using the same queries reveals the trade-offs between lexical matching and semantic proximity. Generative AI can suggest query variants and assist with error analysis. Its suggestions remain separate from human judgements: it is neither the ground truth nor the tool used for the final calculation of metrics. Running case study: Avelcor Notices, a fictitious organisation, searches for procedures in technical manuals. Exact references are sometimes missed, and domain-specific wording retrieves outdated versions. A fictitious, version-controlled corpus is used to compare three configurations on troubleshooting queries. Transfer to the workplace: Adapt the protocol to an authorised, anonymised corpus, have the judgements validated by documentation specialists and launch a limited pilot with stopping thresholds, a designated owner and a review schedule.
Objectives of this training
- Build a representative, documented evaluation set
- Compare lexical, vector and hybrid configurations
- Calculate relevance and latency metrics
- Diagnose regressions by query category
- Define the acceptance conditions for a pilot
Who is it for and what are the prerequisites?
Roles and tasks concerned
- Managers responsible for document retrieval and search products
- Data engineers and developers responsible for a search prototype
Prior skills
- Understand indexing, embeddings and result ranking.
- Be able to run a script and work with structured files.
- Be able to use a generative assistant while verifying its outputs.
What you will work on
Define the decision to be made
Objective: Define the scope of the comparison. Document retrieval versus answer generation; critical use cases and constraints.
Stabilise the corpus
Objective: Avoid biased comparisons. Duplicates, versions, segmentation, metadata and access rights.
Compile the queries
Objective: Cover domain-specific challenges. Exact references, synonyms, ambiguities and questions with no relevant document.
Establish relevance
Objective: Build a human reference standard. Graded relevance, blind annotation, coverage of the results pool.
Assemble the test bench
Objective: Run three comparable configurations. Lexical ranking, vector ranking, rank fusion; parameters and logging.
Calculate the indicators
Objective: Measure ranking and coverage. nDCG@k, MRR, recall@k; limitations arising from incomplete judgements.
Test the trade-offs
Objective: Tune without contaminating the test. Fusion, candidate depth, ablation and overfitting.
Examine failures
Objective: Explain regressions. Obsolete versions, unsuitable segmentation, filters and unauthorised results.
Validate the configuration
Objective: Evaluate on the held-out test set. p95 latency, repetitions, hardware conditions and uncertainty in the differences.
Prepare the pilot
Objective: Justify a decision of limited scope. Scope, human oversight, monitoring and rollback.
What you take away
- Reference set with annotation guidelines
- Executable prototype and version-controlled configurations
- Comparative report with error analysis
- Pilot plan with thresholds and a reasoned decision
Method and assessment
- Short instructional inputs followed by immediate practice on the running case study.
- Individual work followed by work in pairs, with peer review of the outputs.
- Guided group review: what works, what needs correcting and what remains a human decision.
- Preparation: Check access to an isolated environment and the tools authorised by the organisation. Prepare a fictitious or anonymised corpus, its versions and business queries without sensitive data.
Observable criteria: Test queries are separate from tuning queries. ; Annotation disagreements are adjudicated and logged. ; Metrics can be recalculated from the exported results. ; Critical regressions are explained by query family. ; The pilot specifies thresholds for acceptance, stopping and reverting to the previous configuration..
Frequently asked questions
Do you need a search engine in production?
No, an isolated environment with executable configurations is sufficient.
Can AI annotate on its own?
No: reference judgements are validated by humans.
Must hybrid search always win?
No. A lack of improvement or a regression is an actionable result.
Are the generated answers also evaluated?
No, the protocol covers the retrieved documents; generation requires a separate evaluation.
Last updated: