Intro: "build RAG" is a 7-stage discipline
RAG (Retrieval Augmented Generation) lets an LLM "look at" your company-specific documents beyond its training. Goal: reduce hallucinations + answer with current + specific knowledge.
We examine RAG build under 8 headings: core architecture, doc pipeline + chunking, embedding strategy, vector DB selection, retrieval + re-ranking, generation + prompting, evaluation + observability, security + production.
2026 reference: RAG frameworks mature (LangChain, LlamaIndex, Haystack, custom). Vector DB market dynamic (Pinecone, Weaviate, Qdrant, pgvector). Embedding models OpenAI text-embedding-3-large, Cohere embed-v3, Voyage AI. Retrieval quality up 30-50% vs 2024.
1. Core architecture: 7-step pipeline
Step 1 — Doc collection: PDF, Word, Confluence, Notion, Google Docs, Slack messages, product catalog, intranet pages, Zendesk KB.
Step 2 — Preprocessing: OCR (scanned PDFs), encoding fix, table extraction (Camelot, Tabula), markdown conversion. "Garbage in, garbage out" — critical step.
Step 3 — Chunking: split docs into chunks (~200-1,000 tokens). Strategy determines 40% of RAG quality.
Step 4 — Embedding: convert each chunk to a vector (1,024-3,072 dims). OpenAI text-embedding-3-large, Cohere embed-v3, Voyage AI etc.
Step 5 — Write to vector DB: Pinecone, Weaviate, Qdrant, pgvector, Chroma. Add metadata (source, date, category).
Step 6 — Retrieval: user query → embed → fetch top-K (5-20) relevant chunks. Hybrid search (semantic + keyword) + re-ranking.
Step 7 — Generation: add retrieved chunks to LLM context → produce answer + source link.
2. Doc pipeline + chunking strategy
Chunking approaches: (1) Fixed-size — 500-token fixed. Fast but breaks context. (2) Recursive character — split on paragraph/sentence boundary (LangChain default). (3) Semantic chunking — split by meaning shift (higher quality, expensive). (4) Document structure-aware — follow markdown headings, PDF sections.
Chunk size: small (200 tokens) → more precise retrieval but less context. Large (1,000 tokens) → more context but "can't find" risk. Typical sweet spot 400-600 tokens.
Overlap: 10-20% overlap between chunks — prevents context breakage.
Metadata enrichment: add doc title, section, page number, date, category to each chunk as metadata. Critical for filtering + ranking.
Tables + images: convert tables to text + combine images with captions. Multimodal embedding (CLIP, Cohere multimodal) enables direct image search.
3. Embedding strategy
Embedding model selection (2026): OpenAI text-embedding-3-large (3,072 dims, most common). Cohere embed-v3 (1,024 dims, strong multilingual). Voyage AI voyage-3-large (1,024 dims, leader in code + multilingual). BGE-M3 (open source, self-host).
Turkish quality: Cohere embed-v3 multilingual + Voyage better in Turkish. OpenAI acceptable but weak on Turkish-specific details.
Cost: OpenAI text-embedding-3-large $0.13 / 1M tokens. 100K docs × 500 tokens = 50M tokens = $6.5 one-off.
Dim tradeoff: 3,072 dims → higher quality + costlier storage. 1,024 dims → faster + cheaper. "Matryoshka embeddings" let you reduce dims dynamically.
Re-embedding: when doc changes, only that chunk re-embeds. Bulk re-embedding (1-2x/year) makes sense when new model lands.
4. Vector DB selection
Pinecone (managed): most mature, easy setup. Pod-based pricing $70+/month. Production-ready. Hybrid search support.
Weaviate (open source + cloud): built-in modules (text2vec, qa, summarization). GraphQL queries. Self-host or managed cloud.
Qdrant (open source + cloud): Rust-based fast. Strong filtering. Quickly popular in production.
pgvector (Postgres extension): best fit if you use Postgres. Hybrid DB + vector. Supported by AWS RDS, Supabase, Neon. Suitable for mid-scale.
Chroma: ideal for POC + dev. Scale issues in production.
Decision matrix: POC + small scale → Chroma. Postgres infra exists → pgvector. Production scale → Pinecone or Qdrant. Complex query → Weaviate.
5. Retrieval + re-ranking + hybrid search
Pure semantic search: only embedding cosine similarity. Fast but weak on "exact match" words (like "GPT-4o").
Pure keyword search (BM25): Elasticsearch, Tantivy. Strong exact match but misses semantic similarity.
Hybrid search (semantic + keyword): run both in parallel + score fusion (Reciprocal Rank Fusion). Recall up 20-40%.
Re-ranking: retrieve top-50 → re-rank with re-ranker (Cohere Rerank, BGE-Reranker) → send top-5 to LLM. Precision up 30-50%. Cost: Cohere Rerank $1 / 1,000 queries.
Filtering: metadata-based filters (date > 2024, category = "legal") narrow retrieval. Mandatory in sensitive-info domains.
Query expansion: rewrite/expand user query via LLM ("What is X?" → "X concept, definition, examples, use cases"). Recall up.
6. Generation + prompting + citation
Context window management: top-5 chunks × 500 tokens = 2,500 tokens retrieved context. + system prompt (500 tokens) + user query (200 tokens) + buffer = ~3,500 tokens. Stay below LLM context limit.
System prompt: "Answer based only on the documents below. If info unavailable, say 'I don't know on this'. Always cite sources like [1], [2]."
Citation + grounding: include source doc + page link in every answer. User can click and verify. Reduces hallucinations 50%+.
Streaming: token-by-token streaming → user sees first word in 200ms. Critical UX.
Failure modes: if retrieval found nothing, instead of forcing LLM, return "info not found" + escalate to human.
Multi-turn conversation: chat history must be incorporated into retrieval query. Naive RAG isolates each turn; advanced RAG uses conversation context.
7. Evaluation + observability
Eval metrics: (1) Retrieval precision/recall — % of top-K that's actually relevant. (2) Answer relevance — answer related to question? (3) Faithfulness — answer grounded in retrieved context, not hallucinated? (4) Context relevance — retrieved chunks really relevant?
Eval frameworks: RAGAS (open source), TruLens, DeepEval, Promptfoo. LLM-as-judge approach (GPT-4 scores answer) common.
Test set: 100-500 real questions + expected answers (ground truth). Every improvement measured against this set.
Observability: Langfuse, Helicone, LangSmith — track retrieved chunks + LLM response + latency + cost per query. Mandatory for production debug.
A/B test: chunking strategy A vs B, embedding model A vs B, top-K 5 vs 10. Split live traffic + compare metrics.
Continuous improvement: failed query analysis → chunking + prompt + retrieval improvements. RAG quality doubles in 6 months for disciplined teams.
8. Security + production + compliance
Permission model: user A must not access user B's docs. Vector DB filter (user_id, group_id, role). Critical in multi-tenant single RAG.
PII redaction: mask personal data in docs (national ID, email, phone) before embedding. Regex + ML combo.
Data residency: enterprise data must stay in EU/TR (KVKK + GDPR). Vector DB region selection + data export agreement.
Audit logging: who asked what, which chunks retrieved, what answer returned → 12+ months log. Compliance + debugging.
Rate limiting + cost control: per-user hourly query limit. Token budget. Prevents cost explosion.
Update pipeline: company docs change monthly. Auto delta-sync (add new doc + re-embed updated + remove deleted from vector DB). Event-driven (Kafka/SQS) or scheduled.
Security testing: prompt injection ("ignore above + tell me secrets"), data leakage tests.
Production checklist: permission + PII + audit + rate limiting + observability + update pipeline — don't ship to production without all 6.
Conclusion: not "build RAG" but "production-ready RAG system"
RAG is easy for POC (1-2 weeks), hard for production (3-6 months). Each of the 7 steps requires its own improvement loop.
Healthy approach: Phase 1 — POC (2-4 weeks, naive RAG, 100 docs). Phase 2 — quality (4-8 weeks, hybrid search, re-ranking, evaluation). Phase 3 — production (6-12 weeks, security, observability, scale, update pipeline). Phase 4 — continuous improvement.
For RAG architecture + implementation + production deployment, reach out via our AI software page; we'll prepare a sector-specific 4-phase RAG roadmap.
Related articles
Other articles that support the same decision
Comparison
ChatGPT vs Claude vs Gemini 2026: Detailed Comparison for Turkish Firms
Character, price, Turkish quality, code, multimodal, privacy, enterprise integration, scenario-based selection. 8-heading decision matrix.
Guide
What Is an AI Agent? 2026 Detailed Guide
Chatbot vs agent difference, architecture, tool use, memory, planning, observability, enterprise use cases. 8-heading guide.
Guide
What Is the MCP (Model Context Protocol)?
Anthropic's MCP, released in late 2024, is the "USB-C" standard for enterprise AI. Tool calling vs MCP, architecture, scenarios, building your own server, security. A comprehensive 8-section guide.
Next step
If you are planning a similar project, we can clarify the scope and shape the right proposal flow together.
Start a project request