The problem
Large organizational collections contain duplicated, stale and differently permissioned material. When retrieval fails, generation cannot repair the missing evidence—and a fluent answer can make the failure harder to notice.
Design choices that matter
- Ingestion: Preserve document identity, ownership, dates and access metadata.
- Chunking: Split around meaning and document structure, not one universal token count.
- Candidate retrieval: Combine lexical and semantic methods where each adds value.
- Re-ranking: Spend extra computation only where it changes the decision.
- Evaluation: Measure whether the needed evidence was retrieved before grading the generated answer.
Key takeaway
Treat retrieval as a measurable information system. Separate retrieval quality from answer quality, enforce permissions before generation, and design caching around freshness as well as speed.
What to implement first
Build a small question set with known relevant sources, including difficult negatives and permission-sensitive cases. Compare simple lexical, vector and hybrid baselines before adding complexity. Track recall, ranking quality, latency and source freshness together.