Building a local RAG, from PDF to answer

The three previous lessons covered the parts: what RAG is, how documents get split, where the vectors live. Here we line them up on one machine, without sending anything outside the house.

The minimal pipeline has five steps, and none of them needs a graphics card:

PDFs/notes → [extraction] → [chunking] → [embedding] → [store]
                                                          │
                     answer ← [model] ← [retrieval] ←─────┘

The language model only shows up at the last step. All the work that decides quality happens before it, which is why a mediocre RAG setup does not improve by swapping models.

Step 1: getting the text out (the most underrated part)

A PDF can hold text in two completely different ways, and the difference decides how much work is ahead of you.

Digital PDF                         Scanned PDF
the text is inside the file         the page is a photograph
as characters                       made of pixels
→ just read it                      → you need OCR to recognise it

Modern books, academic papers and handbooks produced by a word processor are digital: a basic library extracts the text in a second. Photocopies, digitised old books and scanned documents are images: without optical recognition you get an empty page. Then there are mixed cases, the nastiest ones: native body text with tables and formulas as pictures. There, extraction succeeds and silently loses exactly the parts you needed.

It helps to think in levels, cheapest first:

Level	What it uses	When it is enough
1	Native text extractors	Digital PDFs: most of them
2	OCR with layout analysis, on CPU	Scans, mixed documents, tables
3	Specialised tools, optional graphics card	Formulas, complex layouts
4	Vision-language models	Documents where everything else fails

The practical rule is to start at level 1 and look at what came out: open the extracted text of two or three documents and read it. Thirty seconds there save an afternoon of questions to a store full of nothing.

Two details that ruin otherwise successful extractions: two-column pages, which many extractors read straight across, interleaving the halves line by line, and repeated headers and footers, which end up in every chunk and become identical noise across the whole store.

Steps 2 and 3: splitting and embedding

We covered chunking: recursive, around a thousand tokens, with some overlap, and metadata attached to every chunk. One thing worth repeating here: you split documents, not questions — the question is embedded whole, with the same model.

Local embeddings need two decisions:

Which model. A multilingual one if your documents are not in English: several popular models are trained mostly on English and do noticeably worse elsewhere. Vector size matters less than it seems: bigger does not mean more precise, it means more storage and slower searches.

In batches, not one at a time. Computing embeddings chunk by chunk, one call each, is the mistake that makes indexing feel slow. Batches of a few dozen run far faster on the same machine.

Steps 4 and 5: storing and querying

The local store is the one that installs and just works: it keeps vector, text and metadata together and persists to disk by itself. At the size of a personal library — a few thousand chunks — the exact index is more than enough.

Querying is where two parameters that matter more than the model get decided.

How many chunks to retrieve. Three to five. The temptation to pass fifteen “to be safe” makes the answer worse: the model gets distracted and the context window fills up with mediocre material.

What to tell the model. The system prompt has to be explicit about two things: answer only from the provided context, and if the answer is not there, say so. Without the second sentence the model fills the gap from its general knowledge, and you have no way of noticing: the answer sounds exactly like a correct one.

SYSTEM:   Answer using only the context below.
          If the answer is not in the context, say you don't know.
          Cite the source in brackets.

CONTEXT:  [chunk 1 — handbook.pdf, p. 145]
          [chunk 2 — handbook.pdf, p. 146]

QUESTION: ...

A low temperature helps: creativity is not wanted here, sticking to what it reads is. Same reasoning as the lesson on generation parameters.

How to tell whether it works

A RAG setup is not judged by feel, it is judged in parts. And there are two parts, because there are two ways to fail.

Retrieval. Take ten questions you know the answers to and note, for each, which document and page holds it. Then look at the retrieved chunks, not the final answer: is the right chunk in the top three? If it is not, the problem is upstream and no model change fixes it.

Generation. If the right chunk was there but the answer is wrong or partial, the problem is the prompt, the temperature or the model.

That separation is the whole method: without it, people end up swapping language models to fix a chunking problem.

Typical failures, and what to touch

Symptom	First thing to check
Retrieves irrelevant chunks	Embedding model poorly suited to the language, or chunks too large
Fragmentary answers, no context	Chunks too small: raise size and overlap
Right chunk present but drowned in noise	Chunks too large, or too many passed to the model
Answers from itself, ignoring the documents	Weak system prompt, high temperature, small model
Painfully slow indexing	Embeddings one at a time instead of in batches
Invents things while citing sources	Missing the instruction to say “I don’t know”

That last row is the sneakiest: a system that cites its source looks trustworthy. The citation says where the retrieved chunk came from, not that the generated sentence is in it.

When it is not worth it

Building all of this pays off when documents are many, keep changing, or must not leave the house. For twenty pages you want to understand, pasting them into the prompt is still faster and more accurate. And if the goal is “search my own documents” without writing code, a local interface with RAG built in — Open WebUI on top of Ollama — gets there in half an hour. The value of building it by hand is different: seeing the parameters and being able to move them, which is also the only way to understand why an answer came out crooked.

In short

Concept	In one line
The pipeline	Extract, split, embed, store, retrieve, answer
The underrated step	Extraction: a scanned PDF without OCR yields empty pages
Check to run first	Read the extracted text before indexing
Embeddings	In batches, with a model suited to the documents’ language
How many chunks	Three to five, not fifteen
Prompt	“Only from the context” and “say if it is not there”: both lines
How to evaluate	Retrieval first, generation second: different failures
When to skip it	Few stable documents: pasting them is better