Production RAG Architecture: Hybrid Search, Reranking, Evals

Hamza AnisAuthor

September 26, 2026·13 min read

A production RAG architecture that holds up is a pipeline you can measure, not a prompt you keep tweaking. Build a retrieval evaluation set first. Then retrieve with hybrid search (vector plus keyword, fused with reciprocal rank fusion), rerank the top candidates, filter by permissions inside the query, and make the model answer only from cited context. Add agentic, multi-step retrieval only for the queries that need it. Most teams can run all of this on PostgreSQL they already operate.

We build RAG and AI automation systems that answer from a company's own documents. The pattern we see again and again: when answers are wrong, the team edits the prompt, when the real problem is that the right chunk was never retrieved. So this guide starts with retrieval.

When RAG fails, check retrieval first

A RAG answer can be wrong in two places: the retriever didn't return the chunk with the answer, or the model had the chunk and still answered badly. In our experience the first is far more common, and no prompt can fix it. You can't generate an answer from context you didn't retrieve.

That leads to the most important rule in this post: measure retrieval separately from generation.

Step 1: Build the evaluation set before the pipeline

Collect 100–300 real questions from the people who'll use the system: support tickets, Slack questions, search logs. For each, record which document chunks contain the answer. It's tedious, and it's the highest-leverage day of the whole project.

Then track, for every change to chunking, embeddings, or search:

• recall@k

  : the share of questions where an answering chunk appears in the top

  k

  results,

• MRR

  (mean reciprocal rank): how high the first answering chunk ranks.

// recallAtK returns the share of questions whose top-k results contain
// at least one chunk labelled as answering them.
func recallAtK(results map[string][]int64, relevant map[string]map[int64]bool, k int) float64 {
	if len(results) == 0 {
		return 0
	}
	hits := 0
	for q, ids := range results {
		if len(ids) > k {
			ids = ids[:k]
		}
		for _, id := range ids {
			if relevant[q][id] {
				hits++
				break
			}
		}
	}
	return float64(hits) / float64(len(results))
}

Run it in CI against a snapshot of the index. A change that drops recall doesn't ship.

Step 2: Chunking and metadata

• Chunk by structure, not by character count.

  Split on headings, sections, list items, and table rows so each chunk stands alone. Prepend the document title and section heading to each chunk's text before embedding.

• Keep chunks small enough to be specific

  and large enough to answer on their own. Test two or three sizes against your eval set rather than trusting a default.

• Store metadata on every chunk:

  document ID, source, tenant, access groups, language, and

  updated_at

  . You'll filter and cite with it.

Step 3: Hybrid search in PostgreSQL

Pure vector search is good at meaning but surprisingly bad at exact terms: SKUs, error codes, people's names, acronyms. Full-text search is the reverse. Run both and merge the rankings with Reciprocal Rank Fusion (RRF, Cormack et al., 2009), which needs no score normalization. Each result scores 1 / (60 + rank) in each list, and the scores are summed.

With pgvector and PostgreSQL's built-in full-text search, it's one schema and one query:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          bigserial PRIMARY KEY,
  doc_id      bigint      NOT NULL,
  tenant_id   bigint      NOT NULL,
  content     text        NOT NULL,
  embedding   vector(1536) NOT NULL,          -- match your embedding model's dimensions
  tsv         tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
  updated_at  timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX chunks_embedding_idx ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX chunks_tsv_idx       ON chunks USING gin (tsv);
CREATE INDEX chunks_tenant_idx    ON chunks (tenant_id);

-- $1 = query embedding, $2 = query text, $3 = tenant id
WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY dist) AS rank
  FROM (
    SELECT id, embedding <=> $1 AS dist
    FROM chunks
    WHERE tenant_id = $3
    ORDER BY embedding <=> $1
    LIMIT 40
  ) s
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY score DESC) AS rank
  FROM (
    SELECT id, ts_rank_cd(tsv, query) AS score
    FROM chunks, websearch_to_tsquery('english', $2) AS query
    WHERE tenant_id = $3 AND tsv @@ query
    ORDER BY score DESC
    LIMIT 40
  ) k
)
SELECT c.id, c.doc_id, c.content,
       COALESCE(1.0 / (60 + semantic.rank), 0)
     + COALESCE(1.0 / (60 + keyword.rank), 0) AS rrf_score
FROM semantic
FULL OUTER JOIN keyword ON keyword.id = semantic.id
JOIN chunks c ON c.id = COALESCE(semantic.id, keyword.id)
ORDER BY rrf_score DESC
LIMIT 20;

Notes from running this in production:

• The inner

  ORDER BY … LIMIT

  subqueries let Postgres use the HNSW and GIN indexes; the window functions only rank the 40 candidates.

• Filtering an approximate vector index (by tenant here) can return fewer than

  LIMIT

  rows. pgvector 0.8+ supports iterative index scans (

  SET hnsw.iterative_scan = relaxed_order;

  ). For large tenants, consider partitioning by tenant instead.

• Use

  websearch_to_tsquery

  so user input with quotes and minus signs behaves sensibly instead of throwing syntax errors. Note that it ANDs terms: a long natural-language question can match nothing, and a word like "timeout" won't match "timed out". Test keyword recall on your eval set, and consider OR-ing the important terms for long queries.

Planning a RAG or AI search rollout? Grab our free Backend Scaling Checklist. The same database, caching, and load-testing checks decide whether your retrieval layer holds up under real traffic.

Choosing and changing embedding models

• Pick on your eval set, not on a leaderboard.

  Public benchmarks cover general text; your corpus of tickets, contracts, or code behaves differently. Try two or three models and keep the one with the best recall@k at an acceptable cost and latency.

• Mind the dimensions.

  Larger vectors cost more storage and index memory. Several current models support shortened embeddings, which are worth testing if recall holds.

• Store the model name and version with every row.

  Vectors from different models aren't comparable, so a model change means re-embedding the whole corpus. Build the new embeddings into a separate column or table, run the eval set against it, and switch over atomically.

• Embed queries and documents the way the model expects.

  Some models use different prefixes or modes for queries versus passages; getting this wrong silently lowers recall.

Step 4: Rerank the candidates

Hybrid search gives you a good top 20; a cross-encoder reranker turns it into a great top 5. Rerankers score each (question, chunk) pair jointly, which is slower but much more precise than embedding similarity. Only rerank a small candidate set, and measure the gain on your eval set. If recall@5 doesn't move, drop the reranker and save the latency.

Step 5: Route by query difficulty

Not every question needs the full pipeline. Put a cheap classifier in front:

• No retrieval:

  greetings, questions about the conversation itself.

• Single-shot hybrid retrieval:

  most factual questions.

• Multi-step (agentic) retrieval:

  comparisons, multi-hop questions, "summarize everything about X." The model plans sub-queries, retrieves for each, and checks whether it has enough before answering.

Routing keeps the expensive path for the queries that need it, which matters for both latency and cost. We cover the cost side in detail in how to cut LLM API costs in production.

Step 6: Grounded generation with citations

• Pass chunks with IDs, and require the answer to cite them.

• Instruct the model to say it doesn't know when the context doesn't contain the answer, and test that it does, with eval questions that have no answer in the corpus.

• Validate citations after generation: every cited ID must be one you actually passed in.

• Put the stable instructions first and the retrieved chunks after them, so provider prompt caching still works on the prefix.

Step 7: Freshness and permissions

• Permissions belong in the retrieval query

  , never in a post-filter or the prompt. Every chunk carries tenant and access metadata, and the WHERE clause applies the current user's access. The model can't leak what it never received.

• Re-index on change

  , via change-data-capture or webhooks from the source systems, and delete chunks when documents are deleted. Stale chunks produce confident, wrong answers.

• Version your index.

  Rebuild into a new table or index, run the eval set against it, then switch atomically.

If the RAG system feeds automated actions rather than just answers, wrap it in the validation and human-review patterns from AI agents in production.

FAQ

What is the best RAG architecture for production?

A modular pipeline you can measure: a labelled retrieval evaluation set, hybrid retrieval (vector plus keyword search merged with reciprocal rank fusion), a reranker on the top candidates, permission filtering, and generation that must cite its sources. Add agentic or multi-step retrieval only for the queries that need it.

Why use hybrid search instead of pure vector search?

Vector search is good at meaning but weak on exact terms such as product codes, error messages, names, and acronyms. Keyword search is the opposite. Combining both and fusing the rankings retrieves the right chunk far more often than either alone.

Do we need a dedicated vector database?

Not to start. PostgreSQL with pgvector handles vector and full-text search in one place, with transactions, permissions, and backups you already run. Move to a dedicated engine when scale or latency measurements say so.

How do you evaluate a RAG system?

Separately for retrieval and generation. For retrieval, build a set of real questions labelled with the chunks that answer them and track recall@k and MRR. For generation, check whether answers are supported by the retrieved context and cite it, using human review plus automated grading.

How do you keep RAG answers from leaking data across users?

Filter by tenant and permissions inside the retrieval query itself, not after generation. Store access metadata on every chunk and apply it in the WHERE clause, so the model never sees content the user isn't allowed to read.

Ship RAG that answers correctly

We build retrieval systems grounded in your own data, with evaluation, permissions, and cost controls from day one, as part of our AI automation work. Book a technical discovery call and we'll look at where your current answers go wrong, and whether retrieval is the reason.

Written by Hamza Anis

CEO & Senior Backend Engineer at CapregSoft. Building high-performance backends and AI automation systems for funded startups and mid-market teams.