Building a RAG system you can trust

Hybrid retrieval, API-level citations, a fail-safe hand-off and an answer key: the engineering decisions behind a working RAG prototype over realistic business documents.

Most RAG demos stop at “it answered the question.” That’s really where the interesting work starts. Did it pull the right passage, or did it get lucky? Can someone check where every number came from? What happens when the answer just isn’t in the documents? And when something changes, how do you know it didn’t make things worse?

This project was created to work through those questions on something realistic: a prototype in The Lab at Red Tree Labs, built around the operations desk of a freight brokerage. To be clear, this isn’t a client system. The brokerage, Crosstie Freight, is fictional, and so are its customers, carriers, loads and the fourteen documents behind it, all written for this project. What’s real is the problem. That desk answers the same emails all day long. Why was I charged for waiting time? Why was my invoice short-paid? What do you need for a claim, and by when? The answers are spread across customer contracts, the carrier agreement, load files, carrier invoices and procedures. If someone answers from memory, that’s how a customer gets quoted the wrong rate.

Building it as a prototype also meant the stack could be chosen fresh, with costs kept close to zero. Here’s how it was built, and why each choice was made.

The pipeline

1. Index. Fourteen documents are split into 56 sections. Each section goes into Qdrant with two vectors: a dense embedding for meaning, and a sparse BM25 vector for exact words.
2. Retrieve. An incoming email (subject plus body) runs as one hybrid query with a metadata filter, and the top six sections are kept.
3. Generate. Claude drafts the reply from those six sections only. The API attaches the citations, and the draft is labeled either answered or needs a person.
4. Evaluate. Twenty-two test emails with an answer key score the search, the facts, the citations and the hand-offs, across four retrieval modes.

The stack is Next.js 16 and TypeScript on Vercel, Qdrant Cloud for vectors, Voyage AI (voyage-3.5) for embeddings, Claude for the writing, and PostgreSQL through Prisma to store each evaluation run.

Why Qdrant

Qdrant can run keyword and vector search together in a single query, and it filters by customer during the search rather than after. Its free tier also suited a prototype. pgvector was the obvious alternative, since the app already runs on Postgres, but Postgres doesn’t rank keywords the way BM25 does, so merging the two searches would have fallen to application code.

Chunking on structure, not size

Most RAG frameworks start you off splitting documents into equal-sized pieces, say every few hundred words. For contracts, that’s the wrong call, because the split ignores where one rule ends and the next begins. Take the rule for charging when a truck is kept waiting: $65 an hour after two free hours, capped at $520 per stop. A size-based split can cut that rule in two. The search finds the piece with the hourly rate but not the one with the cap, and the reply quotes a charge the contract doesn’t allow.

So each document is split on its own section headings instead. Every chunk is a complete clause or topic, and the document title goes at the top of every chunk. Without it, three customers’ “Cargo liability and claims” sections look almost identical to the search. With it, “Peachtree Packaging contract: Cargo liability and claims” stands out to both keyword and vector search. It’s about the cheapest way to give a chunk context, and it matters most when documents repeat each other.

Hybrid retrieval in a single Qdrant query

Keyword search and vector search fail in opposite ways, and freight email needs both.

Vector search is good with paraphrasing. “The shipper cancelled after our truck was already rolling” doesn’t share a single keyword with the clause called “Truck ordered not used,” but the meaning is close, so the vectors are close. Documents and questions are embedded with different input_type values, which is what Voyage AI, the embedding service, recommends when short questions are searched against long passages.

Keyword search is good with identifiers: load CT-88231, invoice INV-7719, $211.25. Embeddings tend to struggle with those. The BM25 sparse vectors are computed in TypeScript instead of by another service. The tokenizer keeps hyphens and dots inside words, so CT-88231 stays one token, and it indexes “$65” as both “$65” and “65.” Each term is hashed to a 32-bit index with FNV-1a, and the values are BM25 term-frequency weights (k1 = 1.2, b = 0.75). The collection declares the sparse vector with Qdrant’s modifier: "idf", so Qdrant handles inverse document frequency itself and keeps it current as the data changes. Put together, that’s BM25 without a separate search engine.

Both vectors live on the same record, so hybrid search is one request: two searches, merged with reciprocal rank fusion inside Qdrant.

{
  "prefetch": [
    { "query": { "indices": [...], "values": [...] }, "using": "bm25", "limit": 20, "filter": {...} },
    { "query": [0.013, -0.072, ...], "using": "dense", "limit": 20, "filter": {...} }
  ],
  "query": { "fusion": "rrf" },
  "limit": 6
}

Reciprocal rank fusion (RRF) merges the two lists by rank position, not by score. That’s useful because a vector similarity score and a BM25 score aren’t on the same scale, and it means there are no weights to tune. The trade-off is that it doesn’t know how strong each match was.

The bug that became a feature: metadata filtering

Before the live services were connected, Qdrant’s BM25 scoring was rebuilt offline, including its IDF formula, and keyword search was run against the test set. In that development check, keyword search found every required document in the top six for 15 of the 20 test emails that have required documents.

One miss really stood out. “What are our payment terms?”, sent by a steel company, doesn’t name anybody. Three contracts answer that question three different ways, and no ranking method can know which customer is asking. Better embeddings won’t fix that. It’s an architecture problem. The fix maps the sender’s email domain to a customer and applies a Qdrant filter (customer IN [that customer, "all"], on an indexed field). Now the search only sees that customer’s contract plus the company-wide documents. The same offline check went to 16 of 20, and the system can’t answer one customer’s question out of another customer’s contract anymore.

Two caveats here. Those are development numbers from a keyword-only copy of the scoring, not measured accuracy of the live system. And mapping by email domain is a demo shortcut, not real security, because a From address is easy to fake. In production, the customer would come from a logged-in identity. The part that carries over is where the filter runs: inside the query, before anything is ranked.

Most of the other misses were vocabulary gaps, like “hazmat” in the question and “hazardous materials” in the document. That’s exactly the case vector and hybrid search are there for.

Generation: citations the model can’t fake

The retrieved sections go to Claude as search_result content blocks with citations turned on. When Claude uses a passage, the API sends back a structured citation pointing at the exact search result it used. The model isn’t just typing “[1],” so a citation can only point at a passage that was actually retrieved. On the page, each citation is a link that opens the source text.

That tells you where an answer came from. It doesn’t tell you the answer is right. A model can cite the right section and still misread a number in it. That’s why the evaluation checks the facts against the answer key on its own, separate from the citations.

The system prompt tells the model not to fill gaps with general industry knowledge, and to show its math. For example, 3 hours 15 minutes of waiting time at $65 an hour is $211.25. It runs on Claude Opus 5 at low effort, since the hard part here is the search, not the writing. If a request gets declined, the API falls back to Opus 4.8 on its own.

Designing for “I don’t know”

In business RAG, the most dangerous failure isn’t a missing answer. It’s a confident wrong one. So every draft has to end with a status line, ANSWERED or NEEDS_HUMAN, which is stripped off before anyone sees the reply.

If that line is missing, the draft is treated as needing a person. Anything the documents can’t support, like a new price quote, a load that isn’t on file, or anything that’s “quoted case by case,” should get a polite holding reply and go to a person. This is a guardrail, not a guarantee. The model can still mark a wrong answer as ANSWERED, and that’s exactly what the evaluation counts.

Evaluation: an answer key, not vibes

The test set is 22 incoming emails, and each one has:

• Required documents: the search passes only if all of them show up in the top six.
• Required facts: things the reply has to say, with accepted alternate wordings. Matching ignores case and thousands separators, so “$1,000,000” and “$1000000” count as the same.
• Expected status: three of the emails should be handed off, not answered.

Every email runs four ways, on the same model: no retrieval, keyword only, vector only and hybrid. That comparison is really the heart of it. It shows what retrieval adds over the model on its own, and what each search method brings. And because the search is scored separately from the answer, when a reply is wrong, the scores show whether the search missed or the writing did. The summary tracks answer accuracy, correct hand-offs, citations to the right documents, and “confidently wrong,” meaning replies marked ANSWERED that fail the key. That last one is the number a business should care about most.

The “no retrieval” mode is the control. It’s the same task with no documents, told to answer from general knowledge, which is how an assistant without access to your files behaves. The four-way results will be published from a measured run, not from estimates.

Engineering within real constraints

Serverless time limits. Vercel functions here have a 60-second limit, so a full evaluation can’t run in one request. The browser sends each mode four emails at a time, and the server grades those four at once, so each request only takes about as long as its slowest model call. That lowers the chance of a timeout, but it doesn’t remove it. If a request does fail, that mode stops, shows the error, and gets left out of the comparison instead of showing a partial score.

Embedding rate limits. The demo runs on Voyage’s free tier, which allows a few requests and roughly 10,000 tokens a minute. The whole document set is about 5,000 tokens, so indexing is one request. The test emails are embedded once, when the index is built, and stored in a second Qdrant collection next to the text they came from. If a test email gets edited, that’s caught and it gets re-embedded instead of searched with an old vector. If Voyage reports the limit was hit, the call waits once, retries, and then fails with a clear message.

Keeping the index current. A rebuild deletes and recreates the collection instead of updating it in place. That way a removed section can’t hang around and keep getting retrieved.

What production would add

Everything above is built and running in the demo. This is what it doesn’t cover yet:

• A reranker on the top 20 merged results, kept only if the answer key showed it was worth the extra time.
• Document intake. Real contracts show up as PDFs, so that means text extraction, finding the structure, and re-indexing only what changed.
• Evaluation in CI, running on every prompt or search change and failing the build if things get worse.
• Real access control, based on a logged-in identity and enforced as Qdrant filters.
• Monitoring of what was retrieved, what was cited and how often things get handed off, so drift shows up before a customer notices.

Common questions

How do you combine the keyword and vector results?

Both searches run, and the two ranked lists get blended, so a passage near the top of both comes out ahead. The blend looks only at rank position, not at raw scores, which keeps it simple and hard to break. If one side ever needed more weight, the right move is to test that against the answer key rather than guess.

Would you add a reranker?

For production, probably. A reranker is a second, more careful model that re-orders the top results before the AI sees them. It adds a little time and cost to every question, so it’s only worth keeping if the answer key shows it actually improves the answers.

How do you keep the index current when documents change?

In the demo, a rebuild recreates the whole index, which takes seconds at 56 sections. At real volume, only the document that changed would be re-processed, with its old entries replaced, triggered whenever a file is uploaded or edited, instead of rebuilding everything on a schedule.

Why not fine-tune a model on the documents instead?

Fine-tuning is good at teaching a model a style or a behavior. It’s bad at facts that change, like rates and contract terms, and it can’t show where an answer came from. RAG looks the facts up fresh on every question and cites its source, and updating it means updating the index, not retraining a model.

Would this scale to millions of documents?

The vector database can handle that size. The harder work is everything around it: ingesting documents reliably, keeping embedding and model costs in check, and keeping answers accurate as the collection grows. Filtering by customer or document type before searching helps, and the answer key has to grow along with the data.

The pattern behind all five answers is the same: try the change, measure it against the answer key, and keep it only if the numbers improve.

The takeaway

Honestly, the model is the easy part. What makes a RAG system something you can trust is the engineering around it. Pick a search approach that fits how the documents are actually written. Give people citations they can check. Hand off to a person when the documents don’t support an answer. And keep an answer key that tells you whether a change actually helped. All of that is cheap to build in at the start and expensive to bolt on later. Measure before you claim anything, and be upfront about what a demo proves and what production still needs.

See the RAG work in The Lab →

Built with Next.js, TypeScript, Qdrant, Voyage AI, Claude (Anthropic) and PostgreSQL.