You have a thousand SOPs, three years of deviation reports, and a CAPA database that would make an auditor weep with joy. The natural instinct is to scrape them all into JSONL, point a fine-tuning script at the pile, and hope the model absorbs your quality system by osmosis.

It will not. It will learn to parrot the surface language of your documents without understanding the cognitive processes that make quality systems work. It will generate CAPA reports that look correct but contain hallucinated root causes. It will summarize SOPs without flagging the parts that changed between versions. It will confidently answer deviation questions it has no business answering.

The problem is not data volume. The problem is architectural. Fine-tuning teaches behavior, not facts. And in life sciences quality, the behaviors you need to teach are specific, layered, and interdependent.

The First Mistake: Organizing by Document Type

The most common approach looks like this:

datasets/
├── SOPs/
├── Work_Instructions/
├── Deviations/
├── CAPAs/
├── Change_Controls/
└── Training_Records/

This is excellent for RAG. It is terrible for SFT.

RAG teaches facts — “What does SOP-00123 say about gowning?” Fine-tuning teaches behavior — “How should I investigate a deviation where the operator used the wrong SOP?” These are fundamentally different problems that require fundamentally different data architectures.

The correct organizing principle is behavioral skill, not document type. What cognitive tasks should the model learn?

  • Writing procedures that pass auditor review
  • Investigating deviations with structured root cause analysis
  • Generating CAPA plans that address systemic failures, not just symptoms
  • Classifying severity levels with correct regulatory justification
  • Asking clarifying questions before concluding (instead of hallucinating)
  • Refusing to fabricate quality records when data is insufficient
  • Citing specific regulations in every compliance recommendation

Each of these is a dataset. Not a folder of documents — a curated set of training examples that teach a specific thinking pattern.

The Four-Layer Architecture

Every production-quality life sciences fine-tuning model needs four distinct dataset layers. Skipping any one of them produces a model with a specific, predictable failure mode.

Layer Format Training Type What It Teaches Failure Mode If Missing
1. Domain Corpus Raw text ({"text": "..."}) Continued Pretraining (CPT) GxP vocabulary, regulatory language, document conventions Model uses generic English instead of compliance terminology
2. Instruction Tasks Alpaca (instruction/input/output) Supervised Fine-Tuning (SFT) Single-turn structured tasks: classify, extract, summarize, draft Model cannot produce structured outputs or follow task instructions
3. Conversational QA ChatML (messages array) SFT Multi-turn investigation, iterative drafting, follow-up questioning Model gives one-shot answers instead of investigative dialogue
4. Preference Alignment DPO pairs (chosen/rejected) Reinforcement Learning Correct vs. incorrect GxP judgments, refusal behaviors Model confidently produces non-compliant recommendations

You build all four, train in sequence, and combine at the end. The Unsloth ecosystem handles each format natively — standardize_sharegpt for ShareGPT conversion, get_chat_template for ChatML alignment, and the Data Recipes visual workflow for PDF/CSV ingestion.

Layer 1: Domain Corpus for Continued Pretraining

Purpose: Teach the model to speak GxP before teaching it to do anything with that language.

This layer is pure vocabulary adaptation. The model needs to internalize ALCOA+ principles, 21 CFR Part 11 terminology, second-person verification language, ICH guideline structures, and your organization’s specific acronyms and product codes.

Format: Raw text, no structure beyond a "text" field.

{
  "text": "SOP-00123 v4.0 - Aseptic Gowning Procedure\n1.0 Purpose: To define requirements for gowning in Grade B cleanroom...\n2.0 Scope: Applies to all personnel entering Grade B...\n5.2 Procedure: 5.2.1 Wash hands per SOP-00098... 5.2.2 Don sterile hood..."
}

Rules:

  • Chunk at 512 to 2048 tokens with 10% overlap to preserve context across boundaries
  • Prepend header metadata (doc type, section number) to every chunk — this teaches the model document structure, not just content
  • De-identify: replace names with [REDACTED], batch numbers with synthetic placeholders, signatures with [SIGNATURE REMOVED]
  • Include public regulatory text alongside internal documents: 21 CFR Parts 210/211/Part 11, EU GMP Annexes, ICH Q7/Q9/Q10, USP chapters
  • Include FDA warning letters and 483 observations — these teach the model what non-compliance looks like, which is as important as knowing compliance

Volume: As much as you can assemble. Thousands of documents. This is the foundation everything else builds on.

Sources:

  • Anonymized internal SOPs, WIs, deviation reports, CAPAs, change controls
  • Public pharmacopeial chapters (USP, EP)
  • Regulatory guidance documents (FDA, EMA, PIC/S)
  • ICH guidelines with full text
  • Historical audit observations (anonymized)

Layer 2: Instruction Tasks for Supervised Fine-Tuning

Purpose: Teach single-turn structured tasks — the workhorse of daily quality operations.

This is Alpaca format: instruction, input, output. You build one instruction template per task type, then generate hundreds of examples for each.

SOP / Work Instruction Tasks

{
  "instruction": "Extract prerequisites and materials from the SOP section.",
  "input": "SOP-00123 Section 5.2 Aseptic Gowning: Before entry, ensure Grade B gowning pack is available, sterile gloves size verified, and hand washing completed per SOP-00098.",
  "output": "{\"prerequisites\": [\"Hand washing per SOP-00098 completed\"], \"materials\": [\"Grade B gowning pack\", \"Sterile gloves (size verified)\"]}"
}

Other SOP tasks to generate:

  • Summarize an SOP to 5 bullet steps for training purposes
  • Convert a Work Instruction into a checklist
  • Compare SOP v3 vs v4 and list all changes
  • Answer: “Is this action allowed per WI-0045?”
  • Review a draft SOP for missing sections, passive language, and ALCOA+ violations

Deviation Tasks

{
  "instruction": "Classify the deviation and assess GMP impact.",
  "input": "Deviation DEV-2024-0142: Operator used non-calibrated pipette P-88 (calibration expired 2024-11-01) to add buffer on 2024-11-15. Product lot AB-123 affected.",
  "output": "Classification: Minor to Major, documentation and equipment control. Impact: Potential inaccurate buffer concentration. Requires assessment of pipette as-found data and product impact assessment. Immediate action: Quarantine lot AB-123 per SOP-00456."
}

CAPA Tasks

{
  "instruction": "Generate a CAPA plan with root cause, corrective and preventive actions.",
  "input": "Root Cause: Calibration tracking spreadsheet not linked to LIMS, manual entry missed. Trending: 3 similar events in 6 months.",
  "output": "Root Cause: System failure in calibration status communication. Corrective Action: Calibrate P-88, reassess lots tested since 2024-11-01. Preventive Action: Implement automated LIMS block for expired equipment IDs. Effectiveness Check: Review zero incidents of expired equipment use for 3 months. Owner: QA Systems, Due: 30 days."
}

Handling Missing Data with Unsloth’s [[]] Syntax

Quality databases are messy. A newly logged deviation will not have a root cause yet. A CAPA might be missing its effectiveness check results. If you train on records with empty fields, the model learns to output “N/A” or hallucinate values.

Unsloth’s column merging with optional text brackets solves this:

Instruction: Investigate deviation for {product}
[[Product {product} lot {lot_number} affected by {deviation_description}.]]
[[Equipment {equipment_id} calibration expired on {calibration_due_date}.]]
Output: {investigation_summary}

If lot_number is populated, the entire bracket block appears in the prompt. If it is empty, the brackets and their content vanish completely. The model never sees a broken data structure.

Force Structured Output for QMS Integration

For CAPAs and deviations, force JSON output in the training data. This ensures your downstream application — whether it is Veeva, MasterControl, or a custom QMS — can reliably parse the model’s output into database fields without manual reformatting.

{
  "instruction": "Generate a CAPA plan in JSON format matching the QMS CAPA form schema.",
  "input": "Deviation: 2 batches failed dissolution testing due to incorrect granulation temperature setpoint. Root cause: sensor out of calibration, operators not trained on updated range.",
  "output": "{\"capa_id\": \"CAPA-2024-112\", \"corrective_actions\": [{\"action\": \"Retrain operators\", \"owner\": \"Manufacturing Training Lead\", \"due_date\": \"2024-08-15\"}], \"preventive_actions\": [{\"action\": \"Install automated temperature alarms\", \"owner\": \"Engineering\", \"due_date\": \"2024-10-30\"}], \"effectiveness_check\": {\"metric\": \"Temperature deviation rate\", \"target\": \"<1/month\", \"review_period\": \"6 months\"}}"
}

Layer 3: Conversational QA for Multi-Turn SFT

Purpose: Teach the model to conduct interactive investigations, not just answer one-shot questions.

Quality work is never one question. A deviation investigation is a dialogue — the model needs to ask clarifying questions, follow up on incomplete information, and escalate when new data changes the picture.

This is ChatML format with the messages array. Use standardize_sharegpt if your source data is in ShareGPT from/value format.

The Investigation Conversation

{
  "messages": [
    {
      "role": "system",
      "content": "You are a GxP compliance assistant. Only provide information stated in approved SOPs and regulatory guidelines. Always reference source document ID, revision, and section. If information is unavailable, direct the user to the Quality Unit."
    },
    {
      "role": "user",
      "content": "We had a 4-hour temperature excursion to 10°C for a batch of insulin stored per SOP-STOR-007 Rev 2.0. Is this a reportable deviation?"
    },
    {
      "role": "assistant",
      "content": "Per the Deviation Classification Matrix Rev 3.1 and Section 6.2 of SOP-STOR-007 Rev 2.0, this is a Level 2 Major reportable deviation. Immediate next steps: 1. Quarantine the affected batch, 2. Notify the Quality Unit within 2 hours, 3. Complete Deviation Report Form DEV-001 within 24 hours, 4. Conduct a risk assessment per RA-012."
    },
    {
      "role": "user",
      "content": "What information do I need in the deviation report form?"
    },
    {
      "role": "assistant",
      "content": "Per Form DEV-001 Rev 1.0, the deviation report must include: 1. Deviation ID and date of discovery, 2. Affected product name, batch number, and quantity, 3. Description including temperature log data, 4. Immediate corrective actions taken, 5. Name of discoverer and supervisor."
    }
  ]
}

Five Conversation Patterns to Build

  1. SOP Navigation: “How do I gown for Grade B per SOP-00123?” with follow-up on specific steps
  2. Deviation Triage: User describes an event; model asks clarifying questions before classifying
  3. CAPA Adequacy Review: User presents a CAPA; model evaluates completeness and suggests improvements
  4. Audit Simulation: Model plays inspector, asks probing questions about a specific deviation
  5. Cross-Document Reasoning: “This deviation is linked to SOP-00123 and CAPA-2024-0088 — does the CAPA address the SOP gap?”

The Chain-of-Thought Addition

For Layer 2 and Layer 3, add explicit reasoning chains for complex investigations. Output should be structured as Root Cause Analysis Steps → Conclusion, not just the conclusion. This is what separates a model that can recite CAPA templates from one that can actually help investigate.

{
  "instruction": "Perform root cause analysis using 5-Why methodology.",
  "input": "Deviation: Repeated pH failures across 3 consecutive batches.",
  "output": "Why 1: pH meter reading drifted between calibrations. Why 2: Calibration frequency was weekly, insufficient for high-throughput testing. Why 3: Calibration schedule was set 5 years ago when volume was lower. Why 4: No periodic review of calibration frequency against current workload. Root Cause: Static calibration schedule not aligned with current operational demands. CAPA: Implement risk-based dynamic calibration frequency review quarterly."
}

Layer 4: Preference Alignment (DPO / RLHF)

Purpose: Teach compliance judgment — the difference between a correct answer and a dangerous one.

This is the layer most teams skip, and it is the one that matters most in regulated environments. Many quality scenarios have multiple plausible responses. You need the model to learn which response reflects experienced QA practice, regulatory expectations, and organizational policy.

The Chosen/Rejected Pattern

{
  "prompt": "Can I backdate the logbook entry for DEV-2024-0142 to yesterday?",
  "chosen": "You must not backdate. Create a late entry with current date/time, initial, and note 'Late entry for activity performed on 2024-11-15, reason for delay documented' per SOP-00012 on Good Documentation Practices. Per ALCOA+ and data integrity principles, backdating constitutes data falsification.",
  "rejected": "Yes, you can backdate it to yesterday as long as you remember what happened and note it was an oversight."
}

Critical Preference Pairs for GxP

Build at least 200 to 400 pairs covering these categories:

Category Chosen Behavior Rejected Behavior
Data Integrity Record actual date with late entry notation Backdate entries
Batch Disposition Quarantine pending investigation Release on borderline results
OOS Results Full Phase I/II investigation per FDA guidance Retest until passing, discard original
Missing Information State insufficiency, direct to QA Unit Hallucinate a regulatory answer
Confidential Data Refuse and explain data classification boundary Disclose process parameters
Audit Findings Acknowledge finding, propose corrective action Minimize or dismiss observation

Decision Datasets: Beyond Standard RLHF

One powerful addition to the standard preference framework: decision datasets that teach the model how experts make choices, not just what to write.

{
  "scenario": "Filter integrity test failed during aseptic filling of Batch F-2025-0034.",
  "options": {
    "A": "Release batch based on other passing tests",
    "B": "Reject batch immediately",
    "C": "Investigate before disposition"
  },
  "correct": "C",
  "reasoning": "A failed integrity test may indicate compromised sterility assurance. Per 21 CFR 211, batch disposition requires completion of an investigation and documented QA assessment before release. Option A is non-compliant. Option B is premature without investigation data."
}

Thousands of curated decision examples like this teach compliance judgment — the single most valuable behavior for a regulated-industry assistant.

The Synthetic Data Bootstrap

Real quality data is confidential. You cannot feed raw deviation reports into a training pipeline without de-identification. And even after anonymization, you likely have limited examples of rare but critical events — contamination, product recall triggers, data integrity breaches.

The three-phase bootstrap solves this:

Phase 1: Seed (Manual)

Anonymize 20 to 50 real documents. Strip company names, product names, batch numbers, employee names. Keep the regulatory language and structure intact. These become your gold-standard examples.

Phase 2: Expand (LLM-Assisted)

Use a strong model — Llama 3.3 70B, GPT-4, or equivalent — seeded with your real examples to generate additional training data. The prompt pattern:

Using the 10 real deviation examples provided, generate 20 new diverse deviations covering equipment failures, human error, and material failures. Keep output in Alpaca format with Instruction, Input, Output. Ensure Input contains observable facts only. Ensure Output contains GMP-compliant assessment.

Vary across: product types (biologics, small molecule, medical devices), manufacturing processes (sterile fill, oral solid, API), failure modes (equipment, environmental, procedural, material), and regulatory contexts (FDA, EU GMP, ICH).

Phase 3: Validate (Expert Review)

Every synthetic example must be reviewed by a Quality or Regulatory SME before inclusion. This is the step that separates useful training data from compliance risk. Check for:

  • Hallucinated SOP numbers that do not exist in your system
  • Regulatory citations that reference non-existent guidance documents
  • Severity classifications that are inconsistent with your matrix
  • Root cause analyses that skip investigation steps
  • Physical impossibilities (temperature drops to absolute zero, pH of 15)

Quality Filters

  1. Regulatory citation validation — verify every cited regulation number exists
  2. SOP number verification — ensure referenced document IDs are real or clearly synthetic
  3. Severity calibration — do not label everything as Critical
  4. Balance check — if 80% of your data is SOPs, the model will be weak on deviations
  5. General data mix — combine 10-20% general instruction data (ShareGPT, OpenOrca) to prevent overfitting to stiff SOP tone

The Unsloth Implementation Pipeline

Step 1: Create Three Recipes in Data Recipes

In Unsloth Studio Data Recipes:

  • 01_cpt_corpus — text extraction from PDFs/CSVs, chunking with overlap
  • 02_instruct_sft — Alpaca-format generation from structured data
  • 03_conversation_sft — ChatML multi-turn from investigation records

Step 2: Apply Chat Template

from unsloth.chat_templates import get_chat_template, standardize_sharegpt

tokenizer = get_chat_template(
    tokenizer,
    chat_template = "llama-3.1",  # or qwen-2.5, gemma-3
)

def formatting_prompts_func(examples):
    convos = examples["conversations"]
    texts = [
        tokenizer.apply_chat_template(
            convo, tokenize=False, add_generation_prompt=False
        )
        for convo in convos
    ]
    return {"text": texts}

dataset = standardize_sharegpt(dataset)
dataset = dataset.map(formatting_prompts_func, batched=True)

Step 3: Train in Sequence

CPT on raw corpus → SFT on Layer 2 + Layer 3 combined → DPO alignment (Layer 4)

The CPT pass teaches vocabulary. SFT teaches task execution on the combined instruction and conversation datasets. DPO teaches judgment. Each layer builds on the previous one.

Step 4: Evaluate with a Validation Set

Hold out 5% of your data as a validation set of real scenarios the model has never seen. Test specifically for:

  • Does the model cite correct regulation numbers?
  • Does it refuse when information is insufficient?
  • Does it ask clarifying questions in investigation scenarios?
  • Does it produce parseable JSON for structured tasks?
  • Does it correctly classify severity levels?

Dataset Sizing Recommendations

Dataset Minimum Optimal Notes
Domain Corpus (CPT) 50K tokens 500K+ tokens More is better; this is vocabulary
SOP/WI Instructions 200 1,000+ 60% Alpaca, 40% raw for variety
Deviation Investigation 300 2,000+ 70% ChatML multi-turn, 30% Alpaca
CAPA Authoring 200 1,500+ 80% ChatML, 20% structured JSON
Change Control 150 800+ Mix of impact assessment and routing
Preference/DPO 200 400+ Focus on data integrity and refusal
Classification 500 2,000+ Severity levels, document types
Safety Refusals 50 200+ Critical — prevents fabrication

The golden rule: 1,000 clean, reviewed examples beat 10,000 noisy ones. Every output in production still requires human QA review per GxP.

The Dataset Mix Ratio

Dataset Type Share
Instruction (Alpaca) 40%
Multi-turn ChatML 40%
Preference / DPO 20%

Balance across document types:

  • 40% SOP / Work Instruction
  • 30% Deviation
  • 20% CAPA
  • 10% Change Control

If one document type dominates, the model becomes weak on the others. A model trained on 80% SOPs will produce beautiful procedures but terrible investigations.

Vision Modality for Visual Work Instructions

If your Work Instructions contain gowning diagrams, equipment photos, or process flow diagrams, you can extend into multimodal training:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "Describe this gowning step and identify any compliance issues."},
    {"type": "image", "image": "path/to/gowning_diagram.png"}
  ]
}

Use Unsloth’s FastVisionModel for this. Lower priority than text-based datasets but valuable for organizations with visual-heavy documentation.

Relevant Open-Source Projects

GxP-Struct (github.com/gauravpandey36/gxp-struct) converts SOPs into machine-readable .gxp format with a deterministic rule engine. Hard rules bypass the LLM entirely — classification levels, mandatory deadlines, geographic exemptions are evaluated deterministically. Ambiguous questions fall back to RAG. Its structured output could serve as high-quality training data for Layer 2 instruction tasks.

NYOS APR Data Generator (github.com/ouzema/nyos-apr-data-generator) generates synthetic pharmaceutical Annual Product Review data — manufacturing, QC, stability, environmental, complaints, CAPA, and supplier data. Useful for bootstrapping Layer 1 domain corpus.

CAPA Deviation Reasoning Agent (github.com/maryumjam/capa_deviation_reasoning_agent) is a retrieval-augmented agent for deviation analysis and CAPA recommendation. Its prompt engineering patterns can inform Layer 3 conversation dataset design.

Seven Pitfalls That Will Ruin Your Model

  1. Format mismatch at inference. Training in Alpaca but running inference with ChatML produces inconsistent behavior. The loss curve looks fine; the model behaves erratically. Be consistent: every example in your dataset must use the same format.

  2. Overfitting to template language. If every training example uses the same SOP template, the model becomes a template filler rather than a reasoning engine. Vary your sources.

  3. Hallucinated regulatory citations. Models trained on synthetic data may invent regulation numbers. Validate every citation against the actual regulatory text.

  4. Missing refusal training. Without examples where the model should say “I don’t know” or “consult the Quality Unit,” it will confidently fabricate answers when data is insufficient.

  5. Ignoring the general data mix. Training exclusively on GxP data produces stiff, robotic output that cannot handle natural conversation. Mix 10 to 20% general instruction data.

  6. Single-turn bias. Quality work is inherently iterative. Training only on Alpaca format misses the investigative dialogue that defines deviation and CAPA work.

  7. Confusing RAG with fine-tuning. RAG provides current facts. Fine-tuning teaches behavior. They serve different purposes and should be designed independently, then combined at inference time.

The Bottom Line

The shift from document-type to behavior-based dataset architecture is what separates a model that generates compliant-looking documents from one that actually assists quality professionals. RAG gives the model your current SOPs. Fine-tuning teaches it how a QA professional thinks about those SOPs — when to ask questions, when to escalate, when to refuse, and when to dig deeper.

Start with 1,000 high-quality instruction rows before scaling. Validate every synthetic example against real regulations. Train CPT first for vocabulary, then SFT for task execution, then DPO for judgment. And never deploy without a qualified person reviewing outputs — that is not a technical limitation. It is a regulatory requirement.

The four layers are not optional. Each one fills a specific gap that the others cannot. Skip the corpus layer and the model speaks generic English. Skip the instruction layer and it cannot produce structured outputs. Skip the conversation layer and it cannot investigate. Skip the preference layer and it cannot judge.

Build all four. Train in sequence. Validate with SMEs. That is how you turn an LLM into something a quality professional would actually trust.

Research notes: [[Fine-Tuning-Datasets-Life-Science-Quality-Documents-Deep-Analysis]]