In short: The RAG pattern for LLMs can be evaluated using QA pairs. Creating a "golden" dataset is expensive, but an auto-generated "silver" dataset can help guide RAG's development and initial retrieval process. In this article, one approach is proposed.

Working on the RAG (Retrieval, Augmentation, Generation) pattern for LLMs allows for using the LLM's natural language capabilities as a reasoning-capable frontend for providing context outside of the training data. It is crucial, however, to be able to evaluate the RAG pattern using question-answer pairs such that the built solution matches both real-world usecases and the quality of generated answers can be verified. This is especially difficult given domain specific and highly-technical documentation and context. Often, this is tackled using a golden dataset which aims to closely match actual user scenarios as well as a broad range of possible question/answer pairs. A golden dataset might consist of a question, a context and an ideal answer. The golden dataset is quite costly to generate and maintain as subject matter experts (SMEs) need to be highly involved in creating the dataset. This means it is often practically infeasible to generate a wide-spanning golden dataset for large amount of documentation.

While a nontrivial challenge, it is still desireable to have a golden dataset (co-)created by SMEs so that evaluation metrics closely match real-world performance. In addition to the golden dataset, however, the use of an automatically generated dataset (a synthetic or silver dataset) can offer a variety of benefits and uses. In our ongoing engagement, it will help us steer the development of (various parts of) our RAG solution while the golden dataset is still being built. Arguably, we cannot prove the relationship between a silver dataset and real world (engineering) use-cases, but steering the RAG ship with a silver compass is better than sailing blind. We will be using a silver dataset not only for evaluating the end-end RAG solution (i.e., from user query -> answer), but also for the performance of just the retrieval (RAG). For context, we currently use Azure Cognitive Search with hybrid searching (that is, both vector search and regular text search).

Retrieval evaluation

Measuring performance of retrieval using the silver dataset (trivially) works as follows:

1. Generate a set of question-answer pairs. Take note of the location (file, page, or section) of the retrieved information.
2. For each question: 1. send the question (query); 2. check if your retrieval (search) finds the correct section in the reference document.
   1. This step is crucial as retrieval is the most fundamental part of RAG. Without a proper context, the LLM cannot ground its answer and chances of hallucinations increase.
3. The naive metric: count the number of correct answers. Other metrics exist, but we won't be discussing them in this post.

This process can be automated, and for instance even be added to pipelines to automatically (sanity) check retrieval performance.

QA generation

Some methods exist to automatically generate QA-pairs for (technical) documentation. They include a sequence-to-sequence qa generator on huggingface for which I am not impressed by the performance of the generated QA pairs, and ragas testset generation (link to relevant code). Interestingly, the Ragas approach seems to make use of a critque LLM which decides whether or not the generated QA-pairs are useful.

Personally, I have found the simple approach of a strong prompt in addition to an LLM call with the context to work best. Using GPT4-32k and the prompt which follows, I have been able to reliably generate QA-pairs which are relevant to the context. Most importantly, the instruction to only generate technical question in addition to prompting the LLM to respond with None in certain cases seems to work well in ensuring relevant questions are generated. The latter is important since not all contexts (i.e., text-windows from the origin document) might contain information which is relevant to generate a QA-pair for in the first place.

The prompt is:

You are an AI whose purpose it is to generate question and answer pairs.

It is crucial these question answer pairs are specfic to the context the USER will give you and are related to TECHNICAL content, such that these question answer pairs cannot be retrieved otherwise. DO NOT make up questions and answers that are not related to the context the USER will give you, this will be heavily penalized.

If no technical question can be formulated, it is acceptable to return none. You are expected to return the question pair in JSON like so:

{
    "question": "What is the operating pressure of TK-3413?",
    "answer": "The operating pressure is 1.5 bar."
}

Examples:
USER:
"TK-3413 is a pressure vessel that is used to store water. It is used in the production of the Ford F-150. The operating pressure is 1.5 bar."
AI:
{
     "question": "What is the operating pressure of TK-3413?",
     "answer": "The operating pressure is 1.5 bar."
}
USER:
"The captial of France Paris, in Paris lays the Eiffel Tower. The Eiffel Tower is 324 meters tall."
AI:
{
     "question": "NONE", # No technical question can be formulated, and any search engine can retrieve this information, so None must be returned.
     "answer": "NONE."
}

A simple implementation can be found on my monorepo which includes a crude implemention of chunking PDFs followed by the QA generation.

Example technical document QA generations

The table below gives a sample of QA-pairs generated from pages 10~30 in the Ford Shop Manual for Series 2N 8N 9N (Expired copyright).