There is a question every enterprise team building with Claude eventually asks: how do I let Claude answer questions from my internal documents without sending my entire knowledge base to the cloud?

The answer is Retrieval-Augmented Generation — RAG. But not the naive kind where you dump documents into a hosted vector store and hope for the best. The kind where your documents never leave your infrastructure, your retrieval logic is fully under your control, and Claude Enterprise is used strictly as a generation engine that sees only the specific chunks it needs.

The architecture converges on a single pattern with striking consistency. And Anthropic’s own cookbook provides benchmarks that show exactly how much each optimization layer buys you.

The two paths

There are two fundamentally different approaches, and the choice between them is the first decision you need to make.

Path 1: Claude Projects (the zero-setup path)

Claude Enterprise has a built-in RAG mechanism. When you create a Project and upload documents, Claude processes them in-context. When your knowledge base grows large enough to approach the context window limit, Claude automatically switches to an internal RAG mode — a knowledge search tool that retrieves only the most relevant snippets. Capacity expands by up to 10x.

This requires zero engineering. You drag and drop files, and Claude handles the rest. Data is encrypted at rest and in transit, never used to train models, and isolated within your organization’s tenant. Google Drive integration is available for Enterprise tier.

For 80% of use cases, this is the right answer. No infrastructure to maintain, no chunking strategy to tune, no vector database to operate.

But you give up control. You cannot choose the embedding model, tune the retrieval strategy, implement hybrid search, add metadata filtering, or build evaluation metrics. If you need any of those — and most production teams eventually do — you need Path 2.

Path 2: Custom local pipeline (the control path)

Every component stays on your infrastructure. Documents are parsed, chunked, and embedded locally. Vectors are stored in a local database. When a user asks a question, the system retrieves the most relevant chunks from the local store, packages them with the user’s query, and sends only that small payload to Claude’s API for generation.

Your raw documents never leave your environment. Claude sees only the retrieved context — typically 5-10 text chunks totaling a few thousand tokens.

This is the architecture the rest of this article builds out.

The architecture

┌──────────────────────── YOUR INFRASTRUCTURE ────────────────────────┐
│                                                                      │
│  [Documents] → [Chunking] → [Local Embedding] → [Local Vector DB]   │
│                                                          ↑          │
│  [User Query] → [Query Embedding] → [Similarity Search] ─┘          │
│                                             ↓                       │
│                                      [Top-K Chunks]                 │
└──────────────────────────────────┬──────────────────────────────────┘
                                   │
                                   │ Encrypted API call
                                   │ (chunks + query only)
                                   ▼
                    ┌───── Claude Enterprise ─────┐
                    │   Generation only           │
                    │   No document storage       │
                    │   No training on your data  │
                    └─────────────────────────────┘

Layer 1: Document ingestion and chunking

The first decision is how to split your documents into chunks. This matters more than most people realize. A bad chunking strategy will make even the best embedding model retrieve garbage.

Do not use fixed-size splitting. Cutting every 500 characters will routinely break sentences in half, separate headings from their content, and split tables across chunks. It is the fastest way to destroy retrieval quality.

Use structure-aware chunking. Split by heading hierarchy, natural paragraphs, and semantic boundaries. Keep lists, code blocks, and tables intact. For FAQ documents, split by individual Q&A pairs.

Recommended parameters:

  • Chunk size: 512-1024 tokens
  • Overlap: 10-20% of chunk size (prevents context loss at boundaries)
  • Preserve metadata: source filename, page number, section heading, timestamps

Tools: LlamaIndex or LangChain for standard pipelines. MCP Local RAG (npx mcp-local-rag) for automatic semantic chunking that splits by meaning rather than character count. Runs entirely locally after initial download.

Layer 2: Embedding models

You need to convert each chunk into a vector representation. The critical decision: local or API.

For full data sovereignty (recommended): Run embedding models locally. Your raw document text never leaves your network.

Model Quality Speed Notes
BAAI/bge-large-en-v1.5 High Medium Recommended by 4 of 5 LLMs. Best quality for English.
BAAI/bge-m3 High Medium Strong for multilingual content.
all-MiniLM-L6-v2 Medium Fast Good for prototyping. Lower quality.
Nomic-embed-text Good Fast Good balance.

If API-based is acceptable: Voyage AI is Anthropic’s recommended embedding partner. Their voyage-2 and voyage-3 models are used in Anthropic’s own RAG cookbook and consistently rank at the top of embedding benchmarks. But this requires sending text to Voyage’s API — fine for non-sensitive data, not suitable for air-gapped deployments.

Layer 3: Vector database

Store your embeddings locally. The choice depends on your scale and requirements.

Database Best For Monthly Cost (1M vectors) Key Strength
ChromaDB Prototyping, small teams <$30 (single VPS) Embedded mode like SQLite. Zero config. Python-native.
Qdrant Production, complex filtering $30-50 Filters BEFORE vector search (faster, more accurate). Rust-based.
pgvector Already on PostgreSQL $0 incremental No new infrastructure. SQL filtering.
LanceDB Large self-hosted corpora <$30 Columnar format. Handles larger-than-memory datasets.
Weaviate Built-in hybrid search $50-100 Vector + BM25 in a single query. Highest resource needs.
Pinecone No infrastructure team $70-300+ Fully managed. Premium pricing at scale.

The honest take from production practitioners: “The vector database choice accounts for maybe 5-10% of your RAG system’s quality. Chunking strategy, embedding model, retrieval pipeline, and prompt engineering matter far more. Pick any reasonable option and spend your optimization energy on what actually moves the quality needle.”

For most teams starting out: ChromaDB. It carries you from prototype through production for millions of embeddings on a single modest VPS. If you need complex metadata filtering (legal, financial, regulated content), move to Qdrant.

Layer 4: Retrieval strategy

This is where the biggest quality gains live.

Do not rely on vector search alone. Semantic search is great for understanding meaning but terrible for exact keywords. If someone searches for “Project X-99 compliance audit,” vector search might return results about general compliance but miss the exact project code.

Use hybrid search. Combine vector similarity search with BM25 keyword matching. Run both retrievers in parallel and merge results using reciprocal rank fusion.

Retrieval Method Strength Weakness
Vector (semantic) Understands context, handles paraphrasing Misses exact keywords
BM25 (keyword) Exact term matching, fast, no embeddings needed Misses synonyms and concepts
Hybrid (combined) Best of both More infrastructure

Recommended weighting: 70% semantic, 30% BM25. Adjust based on your content — technical documentation with lots of specific terms benefits from more BM25 weight.

Overhead for adding BM25: ~20-30% storage increase, 1-2 weeks development time, ~30% increased infrastructure costs. Worth it for enterprise and technical content.

Layer 5: Reranking

After retrieving the top 10-20 chunks, apply a reranker to push the most relevant ones to the top. This is a second-pass relevance filter that uses a more sophisticated model than the initial retrieval.

Performance impact (from Anthropic’s benchmarks):

Metric Without Reranking With Reranking
MRR (Mean Reciprocal Rank) 0.74 0.87
End-to-end Accuracy 71% 81%

Tools:

  • Cohere Reranker: Best quality. API-based. ~$0.002 per query. Adds 100-200ms latency.
  • BAAI/bge-reranker-v2-m3: Local. No API calls. Slightly lower quality but zero per-query cost.

When to add: After your baseline pipeline works. Reranking is an optimization, not a foundation. Get retrieval working first, then layer this on.

Layer 6: Prompt construction and Claude API

Take the top-K retrieved chunks, format them into a structured prompt, and send to Claude’s API.

System prompt template:

You are an enterprise knowledge assistant. Answer the user's question 
using ONLY the provided context. If the context does not contain the 
answer, say "I don't have that information in the knowledge base."

For each piece of information, cite the source using [Source: filename, page X].

Context:
{retrieved_chunks}

Question: {user_query}

Settings:

  • Temperature: 0.0 to 0.3 (low for factual accuracy)
  • Use Claude’s XML tags (<context>, <question>) for structured prompting
  • Model: Claude Sonnet 4 for balanced performance/cost; Claude Opus 4 for complex reasoning

The game-changer: Contextual Retrieval

Anthropic published a technique that every team building RAG on Claude should know about. It is called Contextual Retrieval, and it addresses the fundamental problem with chunking: when you split a document, each chunk loses its surrounding context.

A chunk that says “The revenue was $4.2M” does not tell you which quarter, which product line, or which year.

The fix: Before embedding each chunk, use Claude to generate a one-sentence context summary describing where the chunk sits within the parent document. Prepend this context to the chunk before embedding it.

Before: “The revenue was $4.2M representing a 15% YoY increase.”

After: “This chunk is from Acme Corp’s Q3 2024 earnings report, Revenue section. The revenue was $4.2M representing a 15% YoY increase.”

The results from Anthropic’s benchmarks:

Approach Pass@5 Pass@10 Pass@20
Baseline RAG 80.92% 87.15% 90.06%
+ Contextual Embeddings 88.12% 92.34% 94.29%
+ Hybrid Search (BM25) 86.43% 93.21% 94.99%
+ Reranking 92.15% 95.26% 97.45%

Contextual embeddings alone provide the largest single improvement: +5 percentage points. Combined with hybrid search and reranking, retrieval failures drop by 47% — from 12.85% to 4.74%.

The cost problem is solved by Prompt Caching. Generating context for every chunk sounds expensive. But Claude’s Prompt Caching lets you cache the full document in context, then process subsequent chunks at a 90% token discount. One-time ingestion cost: approximately $3 for a typical dataset.

MCP: The emerging integration pattern

An increasingly popular pattern is wrapping your local RAG retriever as an MCP (Model Context Protocol) server. This lets Claude Enterprise and Claude Code query your local knowledge base as a native tool — no separate API layer needed.

Claude discovers the RAG tool automatically via MCP protocol. When a user asks a question that requires internal knowledge, Claude autonomously calls the RAG tool, retrieves relevant chunks, and generates an answer. The user never leaves their Claude interface.

claude mcp add local-rag --scope user \
  --env BASE_DIR=/path/to/your/documents \
  -- npx -y mcp-local-rag

This is ideal for teams that already use Claude daily for coding, document drafting, or analysis and want seamless access to internal knowledge bases without switching contexts.

Enterprise production considerations

Permission filtering

Filter the vector database by user permission tags at the retrieval stage — before results are returned. An HR user should never see engineering documents, even in retrieved context. Filtering after generation is a data leak.

PII redaction

Run retrieved chunks through a local PII scrubber (Microsoft Presidio runs locally) before sending to Claude’s API. Even though Claude Enterprise does not train on your data, minimizing sensitive information sent over the wire is a zero-trust best practice.

Evaluation

Build evaluation before optimizing. Use RAGAS or DeepEval to measure:

  • Faithfulness: Is Claude staying within the retrieved context?
  • Context relevance: Are the right chunks being retrieved?
  • Pass@k: Is the correct chunk in the top-k results?
  • MRR: How high is the first relevant result ranked?

Caching

Implement a semantic cache (GPTCache). If a user asks a question that is semantically identical to one asked yesterday, return the cached answer without calling Claude. This saves API costs and reduces latency.

Data policy

Claude Enterprise does not train on your data. Verify this in your enterprise agreement. API calls are stateless — Claude does not retain your context between requests unless you explicitly maintain conversation history.

The improvement ladder

If you are building this from scratch, here is the order to add each optimization. Each layer yields measurable improvement. Stop when the marginal gain does not justify the added complexity.

  1. Basic pipeline (local embeddings + ChromaDB + Claude API): 87% Pass@10
  2. + Contextual embeddings (prepend context summary to each chunk): 92% — best bang for buck
  3. + Hybrid search (add BM25 alongside vector search): 93%
  4. + Reranking (cross-encoder reorders top results): 95%

Each step is additive. You do not need to rebuild the pipeline — you layer optimizations on top.

For most enterprise teams in 2026:

Layer Choice
Orchestration LlamaIndex or LangChain
Embeddings BAAI/bge-large-en-v1.5 (local)
Vector DB ChromaDB (start) → Qdrant (scale)
Keyword Search Elasticsearch for BM25
Reranking BGE Reranker (local) or Cohere (API)
LLM Claude Enterprise API
Evaluation RAGAS
UI Streamlit (internal) or Chainlit

What Claude Enterprise actually provides

To be explicit about what you are and are not getting:

Claude Enterprise provides: API access to Claude models with higher rate limits, 500K-1M token context windows, SOC 2 / ISO compliance, data not used for training, enterprise support and SLAs, Prompt Caching with workspace-level isolation, Projects with built-in RAG.

Claude Enterprise does not provide: Document hosting, vector store management, retrieval logic, or chunking. That is entirely your infrastructure.

This is by design. Your data stays yours. Claude is the generation engine, not the knowledge store.

The bottom line

Building local RAG on Claude Enterprise is not a single product decision — it is a series of architectural choices, each with clear tradeoffs. The consensus across five independent analyses and Anthropic’s own documentation is remarkably clear:

Start with Claude Projects if you need quick wins with zero engineering. Build a custom pipeline when you need control over retrieval quality, data sovereignty, or evaluation metrics. Use local embeddings and a local vector database so your documents never leave your infrastructure. Layer in contextual embeddings, hybrid search, and reranking one at a time, measuring the improvement at each step.

The gap between a naive RAG pipeline and an optimized one is 87% to 95% retrieval accuracy. That gap is the difference between a tool that occasionally gives wrong answers and one that reliably finds the right information. The techniques to close that gap are well-understood, well-benchmarked, and within reach of any team willing to do the work.