Describe the task behind the query.
A search term is only part of a person’s intention. “Expense policy” might mean finding the current policy, checking an allowance, or locating the submission instructions. A result can match the words and still fail to answer the actual question.
For each important query, write a short task statement. Describe what the person needs to learn or do, which source would help, and what would make the result misleading. This creates a more useful starting point than asking whether the results feel accurate overall.
Use realistic language. Visitors may use abbreviations, older names, or words that differ from the content team’s taxonomy. An evaluation collection should represent those differences rather than sanitize them away.
Choose a small, varied collection.
Begin with a manageable set of queries rather than every possible question. Include a known document title, a broad topic, an ambiguous phrase, a common synonym, and a query with no suitable answer. Add tasks that matter because a wrong answer would cause confusion or extra work.
For each query, record a small number of acceptable results and a reason for their relevance. Some questions legitimately have several useful answers. The evaluation should allow that variety instead of assuming one universal ranking.
Keep a note of what the system is actually allowed to search. A missing document cannot be found by changing the interface if the document never entered the collection.
Inspect the collection before the ranking.
When a query performs poorly, first check the underlying content. Is the title descriptive? Are there duplicate versions? Is the relevant paragraph in an image that has not been indexed? Does the correct record have the metadata used by the filter?
These checks separate content problems from retrieval problems. A different engine may improve some outcomes, but it will not automatically establish which copy is current or who can access it.
Permission boundaries also belong in evaluation. A result preview should not reveal restricted information to someone who cannot open the source. Check both the result listing and the destination behavior.
Review the result as an interface.
A search result needs to explain its own relevance. A title alone may be insufficient when several records share similar names. A meaningful excerpt, content category, review date, or source label can help a reader choose without opening every item.
Pay attention to empty results. The interface can preserve the query, suggest a broader term, explain active filters, or provide another navigation route. A blank screen wastes information about what the person was trying to find.
Evaluate the whole task: query entry, filter adjustment, result interpretation, and arrival at the right information. An improvement in one step can be cancelled by friction in another.
Keep the comparison repeatable.
Run the same collection when changing field weights, indexing rules, metadata, or the retrieval approach. Record which tasks improve and which become worse. A small repeatable comparison is more informative than a demonstration built around a few impressive examples.
The collection can evolve as new needs appear. Keep its purpose clear: it supports decisions about a particular information experience. It is not a claim that every possible search has been solved.