Naive RAG end to end, built once by hand without a framework

Read these first

• RAG architecture

What you will know

• Build working RAG end to end without a framework

  Keep parsing, chunking, embedding, indexing, retrieval, and prompt rendering as inspectable boundaries; opaque convenience calls defeat the exercise.

• Instrument each stage with inputs, outputs, and timings

  Record chunk counts, embedding dimensions, retrieved scores, prompt size, and per-stage latency so failures are observable rather than inferred from the final answer.

• Get a wrong answer and locate which stage caused it

  Capture retrieved chunks, scores, the rendered prompt, and the model output so the first divergence is attributable rather than guessed.

• Compare the hand-built pipeline with a framework implementation on the same corpus and queries

  Hold the model, data, and retrieval settings constant, then record changed defaults, outputs, latency, and dependency behavior.

• Patch one measured failure and compare the result against a baseline

  A change such as altering chunk boundaries or prompt assembly should improve a defined evaluation case without quietly degrading unrelated queries.

Chapters

1. 1

   Six inspectable boundaries, not one convenience call

   Parse, chunk, embed, index, retrieve, render: writing each stage yourself means every intermediate is a value you can print, and the extracted text, the chunk list, the vector array and the prompt string all exist as artifacts you can open. This chapter builds that path over a small corpus and marks the boundaries a single library call would swallow. The exercise is void the moment one call spans three stages.

   Read chapter
2. 2

   A document with zero chunks is missing, not merely unretrieved

   The 2024 refund policy sits in the corpus directory and the assistant keeps answering from the 2021 one. Per-document counters settle it in seconds: characters extracted, chunks produced, vectors written, because a scanned page set extracts almost nothing and yields zero of both while the directory still looks complete. You add counters, embedding dimensions, retrieved scores, prompt size and per-stage timings so absence becomes visible instead of inferred.

   Read chapter
3. 3

   The rendered prompt is the only proof of what the model saw

   Per query, joined into one record: the retrieved chunk ids and scores, the exact string sent to the model, its token count, and the raw output. That string is the sole artifact showing what the model actually received, and the count exposes a chunk the context budget silently dropped. Scattered debug lines cannot be joined, get switched off for cost, and 1% sampling guarantees the complained-about query is the one you did not keep.

   Read chapter
4. 4

   Retrieved at rank 1 and answered wrong is not a retrieval bug

   Two wrong answers with two different owners: in one the correct chunk never entered the top 5, in the other it sat at rank 1 and appears verbatim in the rendered prompt. Only the first can move with chunk size or top-k. The second lives in prompt assembly and generation, and raising top-k makes it strictly worse by adding distractors, conflicting document versions, and distance from the positions the model weighs hardest.

   Read chapter
5. 5

   An instruction to abstain fixes what top-k makes worse

   The chunk sat at rank 1 and the answer was still wrong, so the remaining move lives in the block you render: the order the chunks appear in, the delimiters that stop one document bleeding into the next, an instruction that confines the answer to the supplied context and demands an explicit abstention when the answer is not in it, a citation requirement that makes each claim checkable against the chunk it came from, and a decision about which chunk the context budget drops when they overflow rather than letting truncation decide for you. Then you close the loop a patch is supposed to close: change one of these, rerun the failing case and the passing ones against the same index, and keep the change only if the first moved and the rest held.

   Read chapter
6. 6

   A cosine floor and an 18-of-20 can both be measuring nothing

   Scores of 0.71 that answered correctly and 0.83 that answered from junk are enough to kill any fixed rejection threshold: cosine has meaning only relative to one model on one corpus, so a floor has to be calibrated on pairs you labelled yourself. Turn the same suspicion on your eval by rerunning it with the context block empty. If fifteen of twenty still pass, the score was mostly measuring the model's weights.

   Read chapter
7. 7

   Re-chunking moves every boundary in the corpus at once

   The split pricing table passes once you go from 400 tokens with no overlap to 1000 with 200, and the eval set slides from 16/20 to 14/20. Every boundary moved and every vector changed, so queries that used to land on a tight on-topic chunk now retrieve a diluted one, and at temperature 0 against a fixed index those regressions are deterministic rather than noise to ship past. Being told to ship anyway for the largest customer still leaves choices: widen the window only for the document class that holds tables, keep 400/0 and stop the table being split at parse time, or ship the two regressions named and owned. If the whole table sits in one chunk and still does not retrieve, or the table was destroyed before chunking ever ran, chunk size was never the knob.

   Read chapter
8. 8

   The framework's three-point win is usually a changed default

   Same corpus, same embedding model, same LLM at temperature 0, same top-k, and the reimplementation wins three eval queries while losing one. Hold the inputs fixed and diff the intermediates stage by stage: chunk count and boundaries, vector dimension and normalisation, distance metric, rendered prompt. A gap you cannot name is an improvement you cannot keep, since the next minor version restores the old default and the three queries regress with nobody able to say why.

   Read chapter