Type-Aware Hybrid RAG for Factoid QA

Built a RAG system for short-answer factoid QA over UC Berkeley EECS pages. The pipeline crawls EECS subdomains into a JSONL corpus, generates candidate QA pairs with LLM assistance, and indexes chunks with enriched retrieval text (page title, URL host/path tokens, and content) to improve BM25 lexical matching.

At query time the system builds weighted query variants per question type (person, email, location, date/year, etc.), fuses BM25 scores, applies domain-specific reranking, and answers with an instruction-tuned LLM under strict short-answer formatting. A deterministic extractive fallback (regex + relation patterns + overlap scoring) takes over when the LLM is disabled.

Achieved token-level F1 of 76% on the validation set and 92% on the holdout-mini set with the LLM enabled. Inter-annotator agreement was 86.7% Exact Match and 92.4% token-level F1. Ablations isolate the LLM as the largest single contributor to end-to-end accuracy.

Affiliation

UC Berkeley

Partners

Report

Manuscript

Keywords

NLP
RAG
LLMs
BM25
BFS Crawl
Query Expansion
Slot Filling
Python
Slurm

Answering questions about Berkeley from Berkeley’s own website

The last project in UC Berkeley’s CS288 (Natural Language Processing) was to answer short factoid questions — who is the Director of External Relations, what is the graduate admissions email — using nothing but a crawl of the EECS website. My two teammates and I built the whole pipeline: a crawler that turns the site into a corpus, a type-aware BM25 retriever, an instruction-tuned LLM reader, and a deterministic extractor that takes over when the LLM is gone.

Built for CS288 at Berkeley, with compute from Google Cloud and VESSL AI.

Scoring well on the questions you can see is not the hard part; you tune against them until the number goes up. The work is surviving the questions you can’t see: a hidden grader, running a different LLM, over pages your crawl never emphasised.

The trap in a single F1 number

The system is scored on exact match and token F1 against short gold answers. But a high F1 can come from three different places. Retrieval put the right page in front of the reader and the reader read it. Or the deterministic extractor pulled the answer straight out of the text without the LLM. Or, the dangerous one, the LLM already knew the answer from training and would have said it whatever the retriever returned. The first two generalise. The third collapses the moment the grader swaps in a different model.

Diagram titled 'three sources of correctness': one question flows into a correct answer along three paths. Retrieval-grounded (robust), extractive fallback (robust), and LLM prior knowledge (brittle, in red). A note reads 'end-to-end F1 cannot tell these apart.' — Three routes to a correct answer. Two survive a distribution shift; one is borrowed confidence from the LLM’s memory, and end-to-end F1 cannot tell them apart.

So the pipeline is built to make those routes measurable. Beside the headline F1 it tracks URL recall@k (did the gold page reach the reader), answer-in-context (is the gold answer verbatim in the retrieved text), and fallback-only F1 (how far the system falls with the LLM switched off).

How it works

Type-aware hybrid RAG architecture: an offline crawler over eecs.berkeley.edu builds a corpus of 8,417 documents, chunked and enriched with title and URL tokens before BM25 indexing; at query time the question is typed, expanded into weighted variants whose BM25 scores are fused, reranked with type-aware bonuses, and read by an instruction-tuned LLM with postprocessing; a deterministic fallback handles the no-LLM path, and a soft question-prior module supplies URL bias without acting as a hard filter. — The architecture. Teal is the retrieve-then-generate path; dashed orange is the offline index and the deterministic fallback; dashed purple is the question prior, always a soft bias.

Turning a website into a corpus. A breadth-first crawler starts at the EECS homepage and walks the link graph, staying strictly inside the domain, normalising URLs so no page is crawled twice, and dropping everything that is not HTML. The result is 8,417 pages, the entire knowledge base the system is allowed to know anything from.

A breadth-first crawl graph rooted at eecs.berkeley.edu. Concentric BFS waves expand along link edges; green nodes are visited pages that enter the corpus, grey nodes are queued, and red cut edges show what is dropped: external domains, PDFs and images, cgi-bin endpoints, mailto links. The visited pages funnel into 8,417 JSONL records. — The crawl as a breadth-first walk. Green nodes enter the corpus; red edges are everything cut at the domain boundary.

Making the URL part of the text. The highest-leverage retrieval choice is what a chunk is represented as. Every chunk gets its page title, URL host, and URL path tokens prepended before BM25 sees it. BM25 rewards exact token overlap, and a question about “the Director of External Relations” overlaps a path like /people/staff/external-relations-staff/ far more cleanly than it overlaps the prose. Counterintuitively, the less the page text is cleaned the better: aggressive cleaning strips the exact strings BM25 relies on.

Retrieving by answer type. Before retrieving, the system classifies what kind of answer the question wants — person, email, location, date, program, yes/no — and expands it into weighted query variants tuned to that type. Email questions over-weight @ and contact-page n-grams; person questions over-weight role and title tokens. Scores are fused, and a type-aware reranker rewards chunks whose URL path matches the cue. The default top-3 is deliberately small: enough to clear the recall bar, small enough to keep distractors out of the reader’s context.

A retrieval pipeline for one question, 'What is the graduate admissions email?': typed as 'email', expanded into three weighted query variants, each scored by BM25 and fused, then reranked with a bonus for chunks whose URL path matches the email cue, producing a final top-3 for the reader. — Retrieval conditioned on answer type. An email question is pulled toward contact pages before the reader ever sees a chunk.

Reading, and a fallback for when there’s no reader. The LLM sees the question plus the top chunks and returns only the answer, which is postprocessed for the strict grader: course codes canonicalised, degrees and dates normalised, “unknown” standardised. When the LLM is disabled, a deterministic extractor swaps in: regex and entity detectors for emails, phones, dates, and names over the same chunks. Weaker on the hard cases, but it keeps the system standing when the reader is taken away.

A soft prior, never a hard filter. After the first version underperformed on the hidden dev set, a question-prior module was added: find the most similar past question of the same type and use the URL it resolved to as a bias on reranking. Because it risks overfitting to the curated file, every ablation is reported with and without it, and the headline number is the prior-disabled one.

Results

Validation token-F1 is 0.758 with the prior disabled, against an inter-annotator ceiling of 0.924 measured on a blind subset.

The diagnostics localise the gap. URL recall@4 is 73 % and answer-in-context is 81 %, so for nearly a fifth of questions the right evidence never reaches the reader: retrieval, not reading, is the bottleneck. And the ablation that mattered for the hidden grader:

Validation F1 drops 0.474 when the LLM is switched off (0.758 → 0.284).
Holdout-mini F1 drops only 0.274 (0.924 → 0.650) under the same switch.

The validation set is full of disambiguation cases, several emails on one contact page, that only the LLM resolves, so the fallback collapses there. On the cleaner holdout the extractor gets close to parity, which is the robustness you want when a different LLM is grading. The uncomfortable 0.474 is the honest measure of how much the system still leans on its reader.

What I took away

Scoring well on the slice you can see is easy. Knowing why it scores well, and whether that reason still holds when the distribution moves, is where the engineering was.