Retrieval before the model writes

Retrieval-augmented generation, usually shortened to RAG, is a pipeline with two jobs. Something searches a collection. A language model then writes, with the search results pasted into the prompt. The model is not "connected to your documents" in any stronger sense than that paste.

If you remember one failure mode, remember this one. The search missed the page, so the answer missed the page, and the prose still sounded finished.

The two steps

The search step is ordinary retrieval. It may be keywords, embeddings, or both. It returns a handful of chunks. The prompt step pastes those chunks into the context window along with the question and whatever instruction you set, then asks the model to answer.

Nothing in that second step goes back to the archive unless you built a loop that searches again. A single call sees the chunks you attached and the weights it already has. A fact that lives only in a file you did not retrieve is absent.

What the model does with a good passage

When the chunk contains the answer, in words, the model often copies or paraphrases it. Ask it to quote the sentence it used. A quote you can find in the chunk is evidence the chunk was in the request and was used. A quote you cannot find is a reason to stop.

Short, specific chunks work better than a pasted chapter. The chapter may contain the sentence, and the sentence may sit in the middle of a long window where models are weaker. Put the chunk that answers the question where it will be read, and cut the rest.

What the model does with a bad passage

If the nearest chunk is the wrong year, the wrong product, or a "do not" that looks like a "do," the reply follows it. The tone stays confident because the model is fluent, not because the chunk was checked.

The model can also ignore a correct chunk and answer from training. That shows up as a smooth paragraph that never overlaps the source, or as a citation to a document that was not in the prompt. Training data is not your archive. A title that sounds internal can be a guess.

More chunks are not a cure. They fill the window, they cost tokens, and they add neighbors that contradict the one good passage. Five relevant chunks beat thirty near misses.

How to tell which step failed

Log the chunks that were attached, next to the reply. If the answer is not supported by those chunks, the generator left the source, or the source was the wrong one. If the right page is absent from the log, the failure is the index, the chunking, or the query. Changing the chat model's temperature will not insert a page the search never returned.

A person who can open the file is still the check for anything you would sign, file, or send to a client. RAG is a way to put the right page on the screen. It is not a reviewer.

A log that shows which step failed

Keep, for one request, the question, the chunk ids, the first line of each chunk, and the reply. You do not need a platform for this. A text file is enough while you are learning where the pipeline breaks.

Question: "What is the notice period in the 2024 contractor terms?" Chunks returned: a 2019 PDF whose first line is "Notice: 14 days," and a benefits page that uses the word "notice" for a holiday. Reply: a tidy paragraph that says 14 days and cites "the contractor terms." The generator did what the chunks said. The failure is the index, which never offered the 2024 file. Changing temperature will not insert that file.

A different log: the 2024 chunk is present and its first line states 30 days, and the reply says 14 days anyway, with no quote. The generator left the passage. The fix is the prompt and the check, not another embedding model. Ask for the sentence it used, and reject the reply when that sentence is not in the chunk.

A third log: the right file is in the folder and in neither chunk. It was never embedded, or the job failed halfway. The folder listing and the index listing have to be compared. Retrieval cannot find a document the index does not contain.

Those three logs are sketches of the split, not measurements from a production system. Use them as the shape of the note you write when a real reply is wrong. The note should say which of the three you actually saw. A note that only says "the model was wrong" sends the next edit to the wrong step.