Hybrid RAG in Docker Compose: RustFS, Ollama, LanceDB, and a Frontier Model

You have documents — SOPs, batch records, technical reports, regulatory filings — and you want AI-powered search over them. Not the kind that sends your proprietary content to a third-party API for embedding. The kind where every byte stays on your infrastructure, the embeddings are generated locally, and you control the entire retrieval pipeline.

The architecture that works is hybrid RAG: combine dense vector search (semantic meaning) with sparse BM25 keyword matching (exact terms), fuse the results, and send only the relevant chunks to a frontier model for answer generation. This tutorial builds the entire stack in Docker Compose — four containers, one command to start. The complete source code for this reference implementation is available in the hybrid-rag-docker-example GitHub repository.

Why hybrid matters

Pure vector search fails on quality documents. “SOP-QA-007” and “SOP-QA-070” produce nearly identical embeddings. “USP <85>” (Bacterial Endotoxins) and “USP <87>” (Biological Reactivity) are semantically similar text but entirely different specifications. Vector search confuses them because embeddings capture meaning, not exact strings.

Pure BM25 keyword search fails in the opposite direction. Ask “what happens if storage temperature drops below threshold” and it will miss a chunk that says “cold room deviation event triggered when thermal reading falls beneath the minimum.” Same meaning, different words.

Hybrid search catches both. Dense retrieval handles the conceptual match. Sparse retrieval handles the exact term match. Reciprocal Rank Fusion merges the two ranked lists into a single result set that outperforms either method alone.

The architecture

┌────────────────────────── Docker Compose Network ───────────────────────────┐
│                                                                             │
│  ┌──────────────┐       ┌──────────────────┐       ┌──────────────────┐     │
│  │              │       │                  │       │                  │     │
│  │   RustFS     │◀──────│   RAG API        │──────▶│    Ollama        │     │
│  │  (S3 Store)  │       │  (Orchestrator)  │       │ qwen3-embedding  │     │
│  │   :9000      │       │    :8000         │       │    :11434        │     │
│  │              │       │                  │       │                  │     │
│  └──────────────┘       │  ┌────────────┐  │       └──────┬───────────┘     │
│                         │  │  LanceDB   │  │              │                 │
│                         │  │ (Vector +  │  │              │                 │
│                         │  │  BM25 FTS) │  │              │                 │
│                         │  └───────┬────┘  │       ┌──────▼───────────┐     │
│                         └──────────┴───────┘       │  Frontier Model  │     │
│                                    │               │ (OpenAI-compat)  │     │
│                                    │               │   via Ollama     │     │
│                              ┌─────▼────┐          └──────────────────┘     │
│                              │  Client  │                                   │
│                              └──────────┘                                   │
└─────────────────────────────────────────────────────────────────────────────┘

The flow:

1. Upload a document to RustFS (S3-compatible object storage, written in Rust)
2. RAG API downloads the file, parses it, chunks it into overlapping segments
3. Ollama generates 4096-dimensional embeddings using qwen3-embedding:8b
4. LanceDB stores chunks with both dense vectors and a Tantivy full-text search index
5. On query: LanceDB runs vector search AND BM25 keyword search simultaneously
6. Results are fused using Reciprocal Rank Fusion (RRF)
7. Assembled context is sent to a frontier model via OpenAI-compatible endpoint
8. The model generates an answer grounded in your source documents

Project structure

hybrid-rag/
├── docker-compose.yml
├── .env
├── orchestrator/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app/
│   │   ├── __init__.py
│   │   ├── main.py
│   │   ├── config.py
│   │   ├── embedding.py
│   │   ├── document_processor.py
│   │   ├── vector_store.py
│   │   ├── hybrid_search.py
│   │   └── llm_client.py
│   └── scripts/
│       └── init-ollama.sh
└── data/
    └── sample/
        └── sample.txt

Step 1: Docker Compose

This is the only file that ties everything together. Four services, one network, persistent volumes for all state.

version: '3.8'

services:
  rustfs:
    image: rustfs/rustfs:latest
    container_name: rustfs
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - RUSTFS_ROOT_USER=admin
      - RUSTFS_ROOT_PASSWORD=admin123456
    volumes:
      - rustfs_data:/data
    command: server /data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 15s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: unless-stopped

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
      - ./scripts/init-ollama.sh:/docker-entrypoint-init.d/init-ollama.sh:ro
    # Uncomment for NVIDIA GPU:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s
    restart: unless-stopped

  rag-orchestrator:
    build:
      context: ./orchestrator
      dockerfile: Dockerfile
    container_name: rag-orchestrator
    ports:
      - "8000:8000"
    environment:
      - RUSTFS_ENDPOINT=rustfs:9000
      - RUSTFS_ACCESS_KEY=admin
      - RUSTFS_SECRET_KEY=admin123456
      - RUSTFS_BUCKET=documents
      - OLLAMA_HOST=http://ollama:11434
      - EMBEDDING_MODEL=qwen3-embedding:8b
      - FRONTIER_MODEL_URL=http://ollama:11434/v1
      - FRONTIER_MODEL_NAME=qwen3:32b
      - LANCEDB_URI=/app/data/lancedb
      - CHUNK_SIZE=512
      - CHUNK_OVERLAP=50
      - TOP_K=5
      - DENSE_WEIGHT=0.7
    volumes:
      - lancedb_data:/app/data/lancedb
    depends_on:
      rustfs:
        condition: service_healthy
      ollama:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 30s
    restart: unless-stopped

volumes:
  rustfs_data:
  ollama_data:
  lancedb_data:

RustFS runs as UID 10001. If you hit permission errors on the data volume:

sudo chown -R 10001:10001 ./data/rustfs

Step 2: Ollama initialization script

The orchestrator needs two models: qwen3-embedding:8b for embeddings and qwen3:32b for generation. This script runs inside the Ollama container on first boot and pulls both.

#!/bin/bash
set -e

# Wait for Ollama API
MAX_RETRIES=60
RETRY_COUNT=0
while ! curl -sf http://localhost:11434/api/tags > /dev/null 2>&1; do
    RETRY_COUNT=$((RETRY_COUNT + 1))
    if [ $RETRY_COUNT -ge $MAX_RETRIES ]; then
        echo "ERROR: Ollama API not ready"
        exit 1
    fi
    sleep 5
done

# Pull embedding model
if ! ollama list | grep -q "^qwen3-embedding:8b"; then
    echo "Pulling qwen3-embedding:8b..."
    ollama pull qwen3-embedding:8b
fi

# Pull frontier model
if ! ollama list | grep -q "^qwen3:32b"; then
    echo "Pulling qwen3:32b..."
    ollama pull qwen3:32b
fi

echo "Models ready."
ollama list

Make it executable: chmod +x scripts/init-ollama.sh

For lower resource usage, swap qwen3:32b for qwen3:8b — the architecture stays identical.

Step 3: The orchestrator Dockerfile

FROM python:3.11-slim-bookworm

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential curl && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

Step 4: Python dependencies

fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.4
boto3==1.35.99
lancedb==0.20.0
pandas==2.2.3
pyarrow==18.1.0
requests==2.32.3
pypdf==5.1.0
python-multipart==0.0.20
numpy==2.2.1
python-dotenv==1.0.1
tqdm==4.67.1

Step 5: Configuration

# app/config.py
import os
from dataclasses import dataclass, field

@dataclass
class Config:
    # RustFS
    rustfs_endpoint: str = os.getenv("RUSTFS_ENDPOINT", "rustfs:9000")
    rustfs_access_key: str = os.getenv("RUSTFS_ACCESS_KEY", "admin")
    rustfs_secret_key: str = os.getenv("RUSTFS_SECRET_KEY", "admin123456")
    rustfs_bucket: str = os.getenv("RUSTFS_BUCKET", "documents")

    # Ollama
    ollama_host: str = os.getenv("OLLAMA_HOST", "http://ollama:11434")
    embedding_model: str = os.getenv("EMBEDDING_MODEL", "qwen3-embedding:8b")

    # Frontier Model (OpenAI-compatible)
    frontier_url: str = os.getenv("FRONTIER_MODEL_URL", "http://ollama:11434/v1")
    frontier_model: str = os.getenv("FRONTIER_MODEL_NAME", "qwen3:32b")
    frontier_api_key: str = os.getenv("FRONTIER_API_KEY", "ollama")

    # LanceDB
    lancedb_uri: str = os.getenv("LANCEDB_URI", "/app/data/lancedb")
    table_name: str = "documents"

    # Chunking
    chunk_size: int = int(os.getenv("CHUNK_SIZE", "512"))
    chunk_overlap: int = int(os.getenv("CHUNK_OVERLAP", "50"))

    # Retrieval
    top_k: int = int(os.getenv("TOP_K", "5"))
    dense_weight: float = float(os.getenv("DENSE_WEIGHT", "0.7"))
    rrf_k: int = 60

settings = Config()

Step 6: Document processing

Handles PDF, DOCX, TXT, and Markdown. Chunks text using recursive character splitting that respects paragraph and sentence boundaries.

# app/document_processor.py
import io
import re
import pypdf

def extract_text(file_content: bytes, filename: str) -> str:
    ext = filename.rsplit(".", 1)[-1].lower()
    if ext == "pdf":
        reader = pypdf.PdfReader(io.BytesIO(file_content))
        return "\n".join(page.extract_text() or "" for page in reader.pages)
    elif ext == "docx":
        from docx import Document
        doc = Document(io.BytesIO(file_content))
        return "\n".join(p.text for p in doc.paragraphs)
    else:
        return file_content.decode("utf-8", errors="replace")

def chunk_text(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
    text = text.strip()
    if len(text) <= chunk_size:
        return [text]

    separators = ["\n\n", "\n", ". ", " ", ""]
    return _split(text, separators, chunk_size, overlap)

def _split(text, separators, chunk_size, overlap):
    if len(text) <= chunk_size:
        return [text] if text.strip() else []

    sep = next((s for s in separators if s in text), separators[-1])
    parts = text.split(sep)
    chunks, current = [], ""

    for part in parts:
        candidate = (current + sep + part) if current else part
        if len(candidate) <= chunk_size:
            current = candidate
        else:
            if current:
                chunks.append(current)
            if len(part) > chunk_size and len(separators) > 1:
                chunks.extend(_split(part, separators[1:], chunk_size, overlap))
                current = ""
            else:
                current = part

    if current and current.strip():
        chunks.append(current)

    if overlap > 0 and len(chunks) > 1:
        overlapped = [chunks[0]]
        for i in range(1, len(chunks)):
            overlapped.append(chunks[i-1][-overlap:] + " " + chunks[i])
        chunks = overlapped

    return [c.strip() for c in chunks if c.strip()]

Step 7: Embedding generation

Calls Ollama’s embedding API. The qwen3-embedding:8b model produces 4096-dimensional vectors — high enough resolution to distinguish between semantically similar but functionally different document sections.

# app/embedding.py
import requests
from app.config import settings

def get_embedding(text: str) -> list[float]:
    resp = requests.post(
        f"{settings.ollama_host}/api/embeddings",
        json={"model": settings.embedding_model, "prompt": text.strip()},
        timeout=120
    )
    resp.raise_for_status()
    return resp.json()["embedding"]

def get_embeddings_batch(texts: list[str]) -> list[list[float]]:
    embeddings = []
    with requests.Session() as session:
        for text in texts:
            resp = session.post(
                f"{settings.ollama_host}/api/embeddings",
                json={"model": settings.embedding_model, "prompt": text.strip()},
                timeout=120
            )
            resp.raise_for_status()
            embeddings.append(resp.json()["embedding"])
    return embeddings

Step 8: LanceDB vector store

LanceDB stores both the dense vectors and a Tantivy-based full-text search index in a single table. No external dependencies — it runs embedded inside the orchestrator process.

# app/vector_store.py
import logging
import pandas as pd
import pyarrow as pa
import lancedb
from app.config import settings

logger = logging.getLogger(__name__)
_db = None
_table = None

def get_table():
    global _db, _table
    if _table is None:
        _db = lancedb.connect(settings.lancedb_uri)
        if settings.table_name in _db.table_names():
            _table = _db.open_table(settings.table_name)
        else:
            _table = None
    return _table

def init_table(dimension: int):
    global _db, _table
    _db = lancedb.connect(settings.lancedb_uri)
    if settings.table_name in _db.table_names():
        _table = _db.open_table(settings.table_name)
        logger.info("Opened table '%s' (%d rows)", settings.table_name, _table.count_rows())
    else:
        schema = pa.schema([
            pa.field("chunk_id", pa.string()),
            pa.field("text", pa.string()),
            pa.field("source", pa.string()),
            pa.field("vector", pa.list_(pa.float32(), list_size=dimension)),
        ])
        _table = _db.create_table(settings.table_name, schema=schema)
        logger.info("Created table '%s' with dim=%d", settings.table_name, dimension)
    return _table

def add_records(records: list[dict]):
    global _table
    if _table is None:
        raise RuntimeError("Table not initialized")
    _table.add(records)

Step 9: Hybrid search with Reciprocal Rank Fusion

This is where the magic happens. Two searches run in parallel — dense vector similarity and sparse BM25 keyword matching — then the results are fused using RRF.

# app/hybrid_search.py
import numpy as np
from rank_bm25 import BM25Okapi
from app.config import settings
from app.vector_store import get_table
from app.embedding import get_embeddings_batch

class HybridSearchEngine:
    def __init__(self):
        self.all_chunks = []
        self.bm25 = None
        self.ready = False

    def rebuild(self):
        table = get_table()
        if table is None:
            return
        df = table.to_pandas()
        self.all_chunks = df.to_dict("records")
        if self.all_chunks:
            tokenized = [r["text"].lower().split() for r in self.all_chunks]
            self.bm25 = BM25Okapi(tokenized)
        self.ready = True

    def vector_search(self, query_embedding, limit):
        table = get_table()
        return table.search(query_embedding).metric("cosine").limit(limit).to_list()

    def keyword_search(self, query, limit):
        if self.bm25 is None:
            return []
        scores = self.bm25.get_scores(query.lower().split())
        top_idx = np.argsort(scores)[::-1][:limit]
        return [{**self.all_chunks[i], "_bm25_score": float(scores[i])}
                for i in top_idx if scores[i] > 0]

    def hybrid_search(self, query, top_k=None):
        top_k = top_k or settings.top_k
        fetch_k = top_k * 3

        query_embedding = get_embeddings_batch([query])[0]
        vec_results = self.vector_search(query_embedding, fetch_k)
        kw_results = self.keyword_search(query, fetch_k)

        # RRF fusion
        rrf_scores = {}
        id_to_result = {}

        for rank, r in enumerate(vec_results):
            cid = r["chunk_id"]
            rrf_scores[cid] = rrf_scores.get(cid, 0.0) + 1.0 / (settings.rrf_k + rank + 1)
            id_to_result[cid] = r

        for rank, r in enumerate(kw_results):
            cid = r["chunk_id"]
            rrf_scores[cid] = rrf_scores.get(cid, 0.0) + 1.0 / (settings.rrf_k + rank + 1)
            if cid not in id_to_result:
                id_to_result[cid] = r

        sorted_ids = sorted(rrf_scores, key=rrf_scores.get, reverse=True)[:top_k]
        final = []
        for cid in sorted_ids:
            entry = id_to_result[cid].copy()
            entry["_rrf_score"] = round(rrf_scores[cid], 6)
            final.append(entry)
        return final

engine = HybridSearchEngine()

How RRF works:

RRF_score(d) = Σ  1 / (k + rank_i(d))

Where k = 60 (smoothing constant) and rank_i(d) is the document’s position in each result list. A chunk ranked #3 in vector search and #5 in keyword search gets:

score = 1/(60+3) + 1/(60+5) = 0.01587 + 0.01538 = 0.03125

This naturally favors documents that appear in both result lists — exactly what you want for hybrid retrieval.

Step 10: LLM client

The frontier model is called via OpenAI-compatible endpoint. This means you can point it at Ollama, vLLM, LiteLLM, OpenAI, OpenRouter, or any compatible proxy without changing application code.

# app/llm_client.py
from openai import OpenAI
from app.config import settings

client = OpenAI(
    base_url=settings.frontier_url,
    api_key=settings.frontier_api_key,
)

def generate_answer(query: str, context_chunks: list[str]) -> str:
    context = "\n\n---\n\n".join(
        f"[Source {i+1}]: {chunk}" for i, chunk in enumerate(context_chunks)
    )

    system_prompt = (
        "You are a precise assistant. Answer using ONLY the provided context. "
        "If the context doesn't contain enough information, say so clearly. "
        "Cite sources by their [number] reference."
    )

    user_prompt = f"Context:\n\n{context}\n\n---\n\nQuestion: {query}\n\nAnswer:"

    response = client.chat.completions.create(
        model=settings.frontier_model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        temperature=0.2,
        max_tokens=2048,
    )
    return response.choices[0].message.content

Step 11: FastAPI application

The main application ties everything together: file upload, ingestion, hybrid query, and health checks.

# app/main.py
import time
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI, UploadFile, File, HTTPException
from pydantic import BaseModel
import boto3
from botocore.client import Config as BotoConfig

from app.config import settings
from app.document_processor import extract_text, chunk_text
from app.embedding import get_embeddings_batch, get_embedding
from app.vector_store import init_table, add_records
from app.hybrid_search import engine
from app.llm_client import generate_answer

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

s3 = boto3.client(
    "s3",
    endpoint_url=f"http://{settings.rustfs_endpoint}",
    aws_access_key_id=settings.rustfs_access_key,
    aws_secret_access_key=settings.rustfs_secret_key,
    config=BotoConfig(signature_version="s3v4"),
    region_name="us-east-1",
)

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Ensure bucket exists
    try:
        s3.head_bucket(Bucket=settings.rustfs_bucket)
    except Exception:
        s3.create_bucket(Bucket=settings.rustfs_bucket)

    # Probe embedding dimension
    dim = len(get_embedding("dimension probe"))
    init_table(dim)
    engine.rebuild()
    logger.info("Ready. %d chunks indexed. Embedding dim=%d.", engine.chunk_count, dim)
    yield

app = FastAPI(title="Hybrid RAG Orchestrator", lifespan=lifespan)

class QueryRequest(BaseModel):
    query: str
    top_k: int = settings.top_k

class QueryResponse(BaseModel):
    answer: str
    sources: list[dict]
    timings: dict

@app.get("/health")
async def health():
    return {
        "status": "ok",
        "documents": engine.document_count,
        "chunks": engine.chunk_count,
        "embedding_model": settings.embedding_model,
        "llm_model": settings.frontier_model,
    }

@app.post("/api/upload")
async def upload_file(file: UploadFile = File(...)):
    data = await file.read()
    s3.put_object(Bucket=settings.rustfs_bucket, Key=file.filename, Body=data)
    return {"filename": file.filename, "size": len(data)}

@app.post("/api/ingest")
async def ingest():
    objects = s3.list_objects_v2(Bucket=settings.rustfs_bucket).get("Contents", [])
    total_chunks = 0
    files_processed = 0

    for obj in objects:
        key = obj["Key"]
        ext = key.rsplit(".", 1)[-1].lower()
        if ext not in {"pdf", "txt", "md", "docx"}:
            continue

        raw = s3.get_object(Bucket=settings.rustfs_bucket, Key=key)["Body"].read()
        text = extract_text(raw, key)
        chunks = chunk_text(text, settings.chunk_size, settings.chunk_overlap)

        if not chunks:
            continue

        embeddings = get_embeddings_batch(chunks)
        records = [
            {"chunk_id": f"{key}#{i}", "text": c, "source": key, "vector": e}
            for i, (c, e) in enumerate(zip(chunks, embeddings))
        ]
        add_records(records)
        total_chunks += len(records)
        files_processed += 1

    engine.rebuild()
    return {"files": files_processed, "chunks": total_chunks}

@app.post("/api/query", response_model=QueryResponse)
async def query(req: QueryRequest):
    if engine.chunk_count == 0:
        raise HTTPException(400, "No documents ingested. Upload and run /api/ingest first.")

    t0 = time.time()
    results = engine.hybrid_search(req.query, req.top_k)
    search_ms = round((time.time() - t0) * 1000)

    context_chunks = [r["text"] for r in results]
    sources = [
        {"source": r["source"], "text": r["text"][:200], "rrf_score": r.get("_rrf_score", 0)}
        for r in results
    ]

    t0 = time.time()
    answer = generate_answer(req.query, context_chunks)
    llm_ms = round((time.time() - t0) * 1000)

    return QueryResponse(
        answer=answer,
        sources=sources,
        timings={"search_ms": search_ms, "llm_ms": llm_ms, "total_ms": search_ms + llm_ms},
    )

Step 12: Run it

# Start all services
docker compose up -d --build

# Watch the orchestrator logs (wait for "Ready" message)
docker compose logs -f rag-orchestrator

# Upload a document
curl -X POST http://localhost:8000/api/upload -F "file=@sop-cleanroom-01.txt"

# Trigger ingestion (chunk, embed, index)
curl -X POST http://localhost:8000/api/ingest

# Query
curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What happens if CR-MODULE-99 reports a low temperature?"}'

Step 13: Verify with a test document

Create sop-cleanroom-01.txt:

SOP-CLEANROOM-01: Environmental Monitoring Parameters.
All cold room storage arrays designated under tier-1 biological compliance
must operate continuously at a threshold setting of 4.0°C. If sensor reading
array CR-MODULE-99 drops below 2.0°C, an absolute deviation event is
registered, and quality control management must be notified immediately.

Upload, ingest, then query with both exact-match terms and semantic phrasing:

# Exact keyword match test
curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the threshold for CR-MODULE-99?"}'

# Semantic match test
curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What happens if storage temperature drops too low?"}'

The first query relies heavily on BM25 to match the exact identifier “CR-MODULE-99.” The second relies on vector similarity to understand that “drops too low” means the same thing as “drops below 2.0°C.” Hybrid search handles both.

Tuning the retrieval weights

The DENSE_WEIGHT environment variable controls the balance:

Value	Behavior	Best for
0.3	70% keyword, 30% semantic	Documents with many IDs, codes, part numbers
0.5	Equal weight	General-purpose mixed content
0.7	70% semantic, 30% keyword	Narrative documents, policies, procedures
1.0	Pure vector search	Conceptual Q&A where exact terms don’t matter
0.0	Pure keyword search	Exact lookup by ID or code

For quality documents (SOPs, CAPAs, deviations), start at 0.7 and adjust based on query patterns. If users frequently search by document ID or part number, drop to 0.5.

Access points

Service	URL	Credentials
RAG API	http://localhost:8000/docs	None (Swagger UI)
RustFS Console	http://localhost:9001	admin / admin123456
Ollama API	http://localhost:11434	None

What to avoid

Do not use fixed-size chunking. Cutting every 500 characters splits sentences in half and separates headings from their content. Use structure-aware chunking that respects paragraph and sentence boundaries.

Do not skip the BM25 layer. Vector-only RAG will confuse document IDs, part numbers, and regulatory references that differ by one character. Hybrid retrieval is not optional for regulated content.

Do not use a cloud embedding model for the initial indexing. The whole point of this architecture is data sovereignty. Your raw document text never leaves your network. The embedding model runs in Ollama on your infrastructure.

Do not run the frontier model on CPU for production. CPU inference on a 32B model produces one token every few seconds. For testing, it works. For anything users will actually interact with, you need a GPU or a hosted API endpoint.

The bottom line

This stack gives you a fully local embedding pipeline with state-of-the-art hybrid retrieval and frontier-model generation quality — all containerized in four Docker services. Your documents never leave your infrastructure. The retrieval is transparent and tunable. The frontier model is swappable via a single environment variable.

For quality documents in regulated environments, hybrid RAG is not a nice-to-have. It is the minimum viable retrieval architecture. Pure vector search will miss your SOP numbers. Pure keyword search will miss your conceptual queries. Hybrid catches both.

---

Research notes: [[hybrid-rag-docker-tutorial-analysis]]