A very confident wrong part.

A production RAG system should separate candidate generation from final ranking. The first stage searches a large corpus cheaply and optimizes recall: put every potentially useful passage somewhere in a candidate set. The second stage spends more computation on that much smaller set and optimizes precision: put the evidence that actually answers this query at the top.

Dense retrieval maps the query and documents into independently computed vectors and compares them by cosine similarity, dot product, or distance. It is fast because document vectors are precomputed, but it compresses meaning into one vector and can miss exact identifiers, negation, rare terms, dates, or relationships not represented well by the embedding. Lexical retrieval such as BM25 preserves exact term evidence but misses semantic paraphrases. Hybrid retrieval unions or fuses both candidate sets; neither method is universally superior.

A cross-encoder reranker evaluates the query and each candidate together, allowing token-level interaction. It is usually more discriminating than comparing two independent embeddings, but its work grows with the candidate count. The causal chain is therefore candidate recall -> reranker quality -> context precision -> answer quality. If the needed document never enters the candidate set, the reranker and generator cannot recover it without another search.

Measure retrieval independently from generation. Recall@k asks whether any relevant evidence appears in the first k candidates. Mean reciprocal rank rewards placing the first relevant result early. Context precision asks how much of the material handed to the model is relevant. End-to-end answer accuracy alone collapses retrieval, ranking, prompting, and generation into one opaque number and gives you a miserable incident signal.

`candidate_k` controls how many first-stage results are admitted to reranking. Raising it can improve recall because the relevant passage has more chances to survive approximate search, but reranking cost, query latency, and downstream context pressure rise with it. Lowering it reduces work and can silently make some query classes unanswerable.

For hybrid retrieval, use explicit per-lane budgets rather than one magical integer: for example, 40 dense candidates plus 40 lexical candidates, deduplicated and reranked to 8 passages. Monitor candidate union size, overlap between lanes, recall@k by query class, reranker latency, and final context tokens. If lexical and dense results overlap heavily for a class, you may reclaim work; if they diverge, the second lane is buying coverage.

Tune from labeled queries, not vibes. Sweep candidate_k across 10, 20, 40, 80, and 160; plot recall against p95 latency and cost. Choose the smallest value that meets the retrieval SLO with margin, then keep a larger bounded retry profile for queries where the answerability classifier reports insufficient evidence. A global increase because one product area has poor recall merely invoices every healthy query for that area's corpus problem.

For N vectors of dimension d stored as 32-bit floats, raw vector storage is approximately 4Nd bytes before index overhead. One billion 1,024-dimensional vectors would require about 4.1 TB just for coordinates. Exact scanning is dominated by moving those bytes and computing distances, so large systems prune candidate partitions, use graph traversal, or compress vectors before rescoring a shortlist.

Quantization trades representation fidelity for smaller working sets and lower bandwidth. The relevant SLO is not compression ratio; it is recall at a fixed latency and cost on your corpus. A codec that shrinks the index fourfold but drops rare incident IDs, multilingual terms, or recent documents is a very efficient way to retrieve the wrong shit.

Keep the stages observable: partition probes, approximate candidates examined, bytes read, cache-hit ratio, candidate recall, rescore count, and p95 latency. Metadata filters should prune safely before expensive search when they are selective, but measure filter cardinality and empty-result rates. Precomputing popular query embeddings and caching stable retrieval results can help, provided cache keys include tenant, ACL revision, corpus generation, embedding model, retrieval policy, and freshness boundary.

The ingestion plane should parse documents, preserve source identity and version, create bounded chunks, attach tenant and ACL metadata, embed with a pinned model revision, and write to a new immutable index generation. Validate document counts, tombstones, embedding dimensions, sampled retrieval queries, and source links before atomically promoting that generation. Keep the previous generation available for rollback.

The query plane should resolve the caller's identity and authorization scope first, normalize the query, run lexical and dense retrieval within the permitted corpus, fuse and deduplicate candidates, rerank them, and apply an answerability threshold. Every returned passage carries source ID, document revision, chunk offsets, ingestion time, trust class, and retrieval scores. The generator receives evidence plus provenance, not anonymous text blobs.

Trace one retrieval decision as a release artifact: query hash, tenant, ACL revision, corpus generation, embedding revision, lexical and vector index revisions, candidate IDs and scores, reranker revision, selected passages, and final citations. That trace lets you distinguish model hallucination from stale ingestion, ACL filtering, approximate-search miss, reranker regression, or a deleted source that remained in the index. Without lineage, every RAG incident becomes a séance.

An attacker who can influence indexed content does not need a classic prompt injection to damage RAG. They can copy trusted terminology, repeat likely queries, forge metadata, hide contradictory instructions in high-ranking passages, or flood the corpus with near-duplicates. The result may be ranking capture: malicious material occupies the candidate budget and displaces legitimate evidence.

Treat ingestion as a supply-chain boundary. Authenticate source connectors, preserve immutable origin metadata, separate trust classes, scan and quarantine unexpected content, cap contributions by source, and detect near-duplicate bursts. Do not let document-provided titles, authors, or trust labels overwrite connector-derived metadata. A model-generated summary is derived data, not authoritative provenance.

Authorization belongs inside retrieval, not after it. Search only the corpus the caller may access, or use an index design that enforces equivalent pre-filtering. Fetching globally and deleting unauthorized hits afterward can leak scores or timing and can starve the permitted candidate set. Before any privileged action based on retrieved text, re-resolve the target from an authoritative API and enforce policy at the tool boundary; a convincing chunk is still not an IAM principal.