Many people focus purely on the speed of search, often neglecting the quality of the results produced by the system. In most cases, people test out some small set of queries, eyeball the top five or ten and then declare the system good enough. In other cases, they have a suite of test queries to run, but they are at a loss for how to fix any issues that arise.

To solve this relevance problems takes a systematic approach, a set of useful tools and a dose of patience. This article will outline several approaches and tools. The patience part will come from knowing the problem is being looked at in a pragmatic way that will lead to a solution instead of a dead end.

Table of Contents

Introduction

Determining Relevance Quality

Log Analysis

Empirical Testing

Debugging Relevance Issues

Examining Explanations

Finding Content Issues

Other Debugging Challenges

Tools for Debugging

TREC Tools

Analysis Tools

Using Luke for Debugging

Dealing with Precision/Recall Conflicts

Conclusions

Resources

I'm working on Lucene integration and thought this might be a good guide when improving the relevancy of the test results. ATM I'm still on the level of 'good enough' and came across this write while searching for Lucene scoring.