PDF Question-Answering Agent
Ask questions of a folder of PDFs and get answers with the page they came from.
The problem
Every organisation has documents nobody reads: policies, manuals, contracts, reports. The information is there and the cost of finding it is high enough that people guess instead. This is the smallest useful thing that fixes that, and it is the project most people should build first because it exercises the whole retrieval pipeline in miniature.
Who uses it: An operations team answering "what does our policy say about X" without opening a 90-page PDF.
Before you start
- Python basics
- An API key for any hosted LLM
Stack
Architecture
- 01Ingest: read each PDF, keep the page number with every extracted block of text
- 02Chunk on structure — headings and paragraphs — rather than a fixed token count
- 03Embed each chunk with its document title and page prepended, so it is findable by questions that never use its exact words
- 04Store the vector, the text and the source metadata together
- 05Query: embed the question, retrieve the top k, pass them to the model with an instruction to answer only from the context
- 06Return the answer plus the page citations it used
Build it, in order
01Extract text with page numbers preserved
The page number is the whole point — an answer without a citation cannot be checked. Store it alongside every chunk from the beginning rather than trying to reconstruct it later.
02Chunk on document structure
Split on headings and paragraph boundaries. Fixed-size splitting cuts tables in half and separates a clause from the definition it depends on, which is the single most common reason a demo works and a real corpus does not.
03Embed and store
Prepend the document title and section heading to each chunk before embedding. A bare paragraph is only retrievable by its own words; a chunk carrying its context is retrievable by the question a person actually asks.
04Retrieve and answer
Retrieve 5 to 10 chunks, pass them as clearly delimited context, and instruct the model to answer only from them and to say when they are insufficient. Return the citations with the answer.
05Build the smallest possible interface
One input, one answer, the citations underneath. Resist adding chat history until the single-question case is reliable.
How to know it works
- Write 20 questions with the page you know the answer is on
- Measure retrieval separately: did the correct page appear in the retrieved set at all
- Measure generation separately: given the correct page, was the answer right
- Splitting those two numbers tells you which half to fix; one combined score tells you nothing
What breaks on real input
- Scanned PDFs with no text layer return nothing — detect and report this rather than answering from an empty context
- Tables lose meaning when flattened; keep the section heading with them
- Near-duplicate documents (a policy and its previous version) retrieve interchangeably — include the date in the metadata and in the answer
- The model answers confidently from an irrelevant chunk unless explicitly told what to do when the context does not cover the question
Deployment
- FastAPI service with two endpoints: ingest and ask
- Run ingestion as a job, not in the request path — a 90-page PDF is not a 200ms operation
- Cache embeddings by document hash so re-ingesting an unchanged file costs nothing
Repository structure
ingest/ — extraction, chunking, embedding store/ — vector store interface, one implementation api/ — FastAPI app eval/ — the 20 questions and the scoring script README.md — how to run it against your own folder of PDFs
A layout to create. There is no repository to clone — building it is the point.
Explaining PDF Question-Answering Agent in an interview
Say the retrieval decisions out loud, because that is what is being tested. Why you chunked on structure rather than a token count, why the page number travels with the chunk, and how you know retrieval works separately from generation. If asked what you would change, the honest answer is usually re-ranking: vector search is approximate, and past a few thousand chunks retrieving twenty candidates and re-ranking beats retrieving five and hoping.
Resume lines
- Built a document question-answering service with page-level citations over a corpus of PDFs
- Separated retrieval and generation evaluation, isolating which stage caused each failure
Take it further
- Add a re-ranking step and measure the change against the same 20 questions
- Support asking across a filtered subset — one department, one date range
- Return "not covered in these documents" as a first-class answer rather than a fallback