Every life sciences company has the same problem. The same drug product lives in the ERP as one thing, in the LIMS as another, and in the regulatory system as a third. The same investigator appears in the CTMS under one name, in the CRM under another, and in the safety database under a third. Nobody knows which version is right. Everyone copies from whichever system they trust most. Regulatory submissions get delayed because the data doesn’t match. Batch recalls take days instead of hours because nobody can trace the full chain.

The industry answer for the last fifteen years has been to buy a commercial MDM suite — Informatica, Reltio, Stibo, SAP MDG. These are good products. They are also expensive ($500K–$2M annually in licensing), increasingly cloud-first (which conflicts with keeping regulated data in-house), and validated as black boxes (which means you pay consultants every time you need to change a matching rule).

There is a better way. You can build a fully GxP-compliant, multi-domain MDM platform on mature open-source software, deploy it on commodity hardware in your own data center, and pass any FDA or EMA audit — for roughly one-quarter the cost of a commercial suite over three years.

This is not a theoretical exercise. Every component in the architecture described here runs in production at organizations that handle regulated data at scale. The stack is PostgreSQL, Kafka, Airflow, Keycloak, and a handful of focused Python services. No vendor lock-in. No per-record licensing. No data leaving your building.

Why Life Sciences MDM Is Different

Generic MDM treats master data as a consistency problem. Life sciences MDM treats it as a compliance problem.

The differences are not subtle:

Every change must be traceable. When the FDA asks “who changed this product classification and why,” you need an immutable audit trail with timestamps, user IDs, and electronic signatures. This is 21 CFR Part 11, not a nice-to-have.

Matching rules carry regulatory risk. If your probabilistic matching engine merges two patients who are actually different people, you have a HIPAA violation. If it splits one patient into two, your safety signal detection is broken. The thresholds matter in ways they don’t in retail or financial services.

Data integrity is a legal requirement. ALCOA+ — Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, Available — is not a data quality framework. It is a regulatory expectation. Your MDM platform must satisfy every one of those principles by design, not by policy.

Survivorship rules are domain-specific. The rule “most recent update wins” works for a customer address in retail. It does not work when the question is whether a drug product’s regulatory status should be overwritten by a CRM user who hasn’t checked with Regulatory Affairs.

These requirements don’t disappear when you choose open source. They do change how you architect the platform.

The Consensus Architecture

After examining the full landscape of open-source tools, integration patterns, and life sciences regulatory requirements, a clear architectural consensus emerges. The stack has six layers.

┌─────────────────────────────────────────────────┐
│              CONSUMING SYSTEMS                  │
│  ERP │ LIMS │ CTMS │ CRM │ MES │ BI │ AI Agents │
└────────────────────┬────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────┐
│            API & DISTRIBUTION LAYER             │
│  Kong Gateway │ Kafka Events │ Batch Extracts   │
└────────────────────┬────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────┐
│              MDM CORE ENGINE                    │
│  Golden Records │ Matching │ Stewardship │ Audit│
│  (PostgreSQL + Python microservices)            │
└────────────────────┬────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────┐
│         DATA QUALITY & GOVERNANCE               │
│  Great Expectations │ OpenMetadata │ Lineage    │
└────────────────────┬────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────┐
│           INGESTION & STREAMING                 │
│  Debezium CDC │ Apache NiFi │ Airflow │ dbt     │
└────────────────────┬────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────┐
│              SOURCE SYSTEMS                     │
│  SAP ERP │ Veeva Vault │ LIMS │ CTMS │ LegacyDBs│
└─────────────────────────────────────────────────┘

Below these layers, cross-cutting services handle identity (Keycloak), secrets (HashiCorp Vault), monitoring (Prometheus + Grafana), and logging (Loki or ELK). Everything runs on Kubernetes — k3s for smaller deployments, RKE2 for larger ones.

The Technology Stack

Database: PostgreSQL 16

PostgreSQL is the unanimous choice for the MDM hub, and the reasons go beyond “it’s free.”

JSONB columns handle the reality that life sciences master data is semi-structured. A drug product in the US has NDC codes, dosage forms, and SPL labeling requirements. The same product in the EU has different regulatory attributes. A monolithic relational schema collapses under this variation. JSONB lets you store flexible attributes per domain without constantly altering tables.

pg_trgm and fuzzystrmatch give you built-in fuzzy matching — Levenshtein distance, Soundex, Metaphone — without an external matching engine for deterministic and simple probabilistic use cases.

Row-Level Security partitions access at the database level. QA sees quality data. Regulatory sees registration data. Commercial sees HCP data. This is not application-level filtering — it is enforced by the database itself.

pg_audit captures every DDL and DML statement for 21 CFR Part 11 compliance. Combined with append-only audit tables, you get an immutable record of every change ever made to any master record.

Logical replication provides read replicas for reporting and analytics without impacting the operational MDM database.

Event Bus: Apache Kafka

Every golden record change produces an event. Product approved. HCP merged. Vendor status changed. These events flow through Kafka topics that downstream systems subscribe to.

Nobody polls the MDM database. Nobody runs nightly batch extracts to check for changes. The MDM hub publishes; consuming systems subscribe. This is the difference between an integration architecture from 2010 and one that actually works.

Kafka also serves as the durable log for replay. If a downstream system misses an event, it can re-consume from its last committed offset. This is critical for GxP — you need to prove that every system received every relevant change.

Ingestion: Debezium + NiFi + Airflow

Three tools, three patterns, no overlap:

Debezium reads transaction logs from source databases (SAP, LIMS, CTMS) and pushes changes to Kafka in near-real-time via Change Data Capture. No performance hit on the source system. No custom extraction scripts.

Apache NiFi handles file-based ingestion — CSV drops from legacy systems, HL7 FHIR messages from clinical sites, reference data feeds from FDA and CMS. Visual flow design means you can build and modify pipelines without writing code.

Apache Airflow orchestrates scheduled workflows — nightly reconciliation, weekly NPI verification, quarterly reference data updates, monthly DQ scorecards. DAG-based scheduling with retries, monitoring, and alerting.

dbt (data build tool) handles the transformation layer — standardizing formats, applying business rules, building staging models. SQL-based, version-controlled, testable.

Data Quality: Great Expectations

Great Expectations provides the declarative rules engine that every other layer references. Before a record enters the golden store, it must pass validation:

  • NDC code must be exactly 10 digits in the correct format
  • NPI must be exactly 10 digits and verifiable against CMS NPPES
  • DEA number must match the regex pattern and have a valid check digit
  • If lifecycle status is “Marketed,” then NDC, dosage form, and route of administration are all required
  • If controlled substance, then DEA schedule must exist
  • Supplier must have a valid GMP certification date within the last 12 months

Each rule produces a pass/fail with a data quality score. Records below 70 trigger steward review. Records below 50 are quarantined — they never reach the golden store.

Identity: Keycloak + HashiCorp Vault

Keycloak bridges your existing Active Directory or LDAP to the MDM platform. SSO, MFA, RBAC — all managed centrally. When a data steward approves a golden record change, Keycloak authenticates them, confirms their role, and logs the session.

HashiCorp Vault manages secrets (API keys, database credentials, encryption keys) and provides transit encryption for audit log hash chains. Secrets rotate automatically. No credentials in environment variables or config files.

Search: OpenSearch

Master data in PostgreSQL is optimized for transactional integrity, not full-text search. OpenSearch indexes golden records so that a steward can search for “tablet Japan 500mg released” and get results in milliseconds. It also powers the fuzzy matching index for entity resolution.

Object Storage: MinIO

Certificates, COAs, validation reports, images, specifications — these don’t belong in PostgreSQL. MinIO provides S3-compatible object storage on your own hardware. Metadata stays in the database; the files live in MinIO. Encryption at rest, versioning, lifecycle policies — all the features of cloud object storage without the cloud.

Master Data Domains: What to Master and When

The biggest mistake in MDM is trying to master everything at once. Life sciences companies have a dozen potential domains. Start with two or three that deliver the highest return on governance investment.

Phase 1: Product/Material Master and Reference Data

Product Master is the natural starting point. It is highly structured, resides primarily in the ERP and regulatory systems, has clear unique identifiers (NDC, GTIN, CAS numbers), and delivers immediate ROI by eliminating duplicate product records that cause procurement errors and regulatory confusion.

Entities: Drug products, APIs, excipients, packaging components, medical devices. Key attributes: NDC code, dosage form, strength, route of administration, regulatory status, lifecycle stage, shelf life, storage conditions.

Reference Data runs in parallel. MedDRA, SNOMED CT, ICD-10, RxNorm, ATC codes, WHO Drug Dictionary, ISO country codes, UOM — this is the Rosetta Stone that all other domains reference. Centralizing it prevents the chaos where five departments use five different code sets for the same thing.

Phase 2: Organization (HCP/HCO) and Vendor/Supplier

HCP/HCO Master addresses the highest data quality pain in commercial operations. The same physician appears as three different records in the CRM because their name was entered differently each time. NPI-based deterministic matching collapses these immediately. Fuzzy matching on name, address, and specialty catches the rest.

Entities: Healthcare professionals, healthcare organizations, CROs, KOLs. Key attributes: NPI, DEA number, state license, specialty, affiliation hierarchy, KOL flag.

Vendor/Supplier Master ensures that procurement and quality share the same view of every supplier’s qualification status, audit history, and GxP certification.

Phase 3: Site, Study, and Batch

Site/Facility Master unifies manufacturing plants, QC labs, warehouses, and clinical sites.

Study/Protocol Master harmonizes clinical trial reference data across CTMS, EDC, and safety systems.

Batch/Lot Master connects manufacturing batch records to product master data for end-to-end traceability — critical for recall management.

Entity Resolution: How Matching Actually Works

Every MDM proposal, regardless of technology stack, converges on the same two-stage matching strategy.

Stage 1: Deterministic Matching (Auto-Merge)

These are high-confidence matches based on unique identifiers:

Domain Matching Key Confidence
Product NDC code exact match 100%
Material CAS number exact match 100%
HCP NPI exact match 100%
Vendor DUNS / Tax ID exact match 100%
Site GLN / DUNS exact match 100%

If two records share the same NDC, they are the same product. No ambiguity. No steward review needed.

Stage 2: Probabilistic Matching (Scored + Steward Review)

This is where it gets interesting. When records don’t share a unique identifier — misspelled names, partial addresses, different formatting — you need probabilistic scoring.

For HCP matching, a typical rule set looks like:

  • Last name similarity ≥ 0.90 (Levenshtein distance)
  • First name similarity ≥ 0.85
  • State = same
  • Specialty = same
  • Composite score ≥ 0.85 → auto-merge
  • Score 0.60–0.84 → steward review queue
  • Score < 0.60 → reject

The technology here ranges from PostgreSQL’s built-in fuzzystrmatch functions for simpler cases to dedicated probabilistic matching libraries like Splink (built by the UK Ministry of Justice, MIT-licensed) for complex multi-field matching with Fellegi-Sunter weights.

Start with PostgreSQL fuzzystrmatch. When the steward review queue grows too large because the matching isn’t precise enough, bring in Splink.

Survivorship Rules

When two records merge, which values survive? This is not a technical question — it is a business governance question that must be answered per domain and per attribute.

Common patterns:

  • Most recent verified source wins for contact data (phone, email, address)
  • Most complete record wins for descriptive fields
  • Regulatory source wins for classification and status fields
  • Manual steward decision always overrides automated rules

All survivorship rules must be version-controlled in Git, not buried in database stored procedures. When the FDA asks why a golden record has a particular value, you point to the rule version, the source records, and the timestamp.

The Compliance Layer (This Is Non-Negotiable)

21 CFR Part 11: Electronic Records and Signatures

The MDM platform is an electronic system that creates, modifies, maintains, and distributes electronic records. It falls squarely under 21 CFR Part 11.

Implementation:

Immutable audit trail. Every change to every master record is captured in an append-only audit table. No UPDATE. No DELETE. The audit record includes: record ID, field changed, old value, new value, changed by (user ID, IP address, session), timestamp, change reason, and workflow step.

Electronic signatures. When a steward approves a golden record change, they re-enter their credentials (authenticated via Keycloak) and attach a meaning: “I approve this golden record as the authoritative version.” The signature is recorded in the audit trail and cannot be repudiated.

System access controls. Unique user IDs, role-based access, session timeouts, device checks — all managed through Keycloak with LDAP/AD integration.

ALCOA+ Data Integrity

Principle How the MDM Platform Satisfies It
Attributable Every record change tagged with authenticated user ID + source system
Legible Standardized data formats; human-readable audit logs exportable to PDF
Contemporaneous Database server timestamps (NTP-synced), not application-layer clocks
Original Staging schema preserves raw source data exactly as received
Accurate Great Expectations validates on ingestion; cross-reference against authoritative external sources (NPPES, FDA NDC directory)
Complete Required field rules per domain; completeness scoring in DQ engine
Consistent Cross-field validation rules; referential integrity enforced at database level
Enduring WAL archiving, point-in-time recovery, 7+ year retention policy
Available Streaming replication, automated failover, quarterly DR drills

GAMP 5 Validation

Here is where open source has a hidden advantage. The MDM platform components fall into GAMP 5 categories:

  • Category 3 (Infrastructure): PostgreSQL, Kafka, Kubernetes, MinIO — these are standard infrastructure products. You document them in your infrastructure qualification but don’t need to re-validate their core functionality.
  • Category 4 (Configurable): Great Expectations rules, Airflow DAGs, matching configurations — these are configurable applications where you validate the configuration, not the underlying software.
  • Category 5 (Custom): Your custom Python matching microservices, the stewardship UI, the API layer — these require full IQ/OQ/PQ.

Compare this to a commercial MDM suite, which is a Category 4 or 5 black box where every change requires re-validation of the entire platform. With open source, you validate only the components you build or configure, and you can automate most of the test cases.

Security Architecture

Network segmentation puts MDM servers in a dedicated VLAN. Firewall rules allow only approved ports from known subnets. TLS 1.3 for all external communications. An internal CA manages certificates.

At the database level, PostgreSQL Row-Level Security enforces domain-level access. A QA analyst can read product master data but cannot edit regulatory fields. A regulatory affairs user can edit registration data but cannot approve quality decisions. The auditor role gets read-only access to everything plus the full audit trail.

Column-level encryption protects PII/PHI fields. Patient data is tokenized before it hits the MDM hub — the platform matches on tokens, and only authorized clinical systems can reverse the token back to identifiable data. PHI never resides in the MDM golden store.

The Phased Roadmap

Phase 0: Discovery (3–6 Weeks)

This is the most important phase and the one most teams try to skip.

  • Inventory source systems. Catalog every system that holds master-like data. Document the data owner, the data model, the interface method (API, CDC, file drop), and the data quality baseline.
  • Profile data quality. Run Great Expectations or a profiling tool against the top 3 source systems. Measure duplication rates, completeness, conformance to standards. Expect 15–30% duplication in unmanaged master data.
  • Pick the pilot domain. Product/Material if your ERP data is relatively clean and the business case is clear. HCP/HCO if commercial teams are actively suffering from duplicate records.
  • Define the golden record schema. What attributes belong in the golden record? What stays in the source system? What is the canonical format for each attribute?
  • Draft the validation strategy. IQ/OQ/PQ templates, risk-based approach, traceability matrix.

Phase 1: Foundation and Pilot (3–6 Months)

  • Provision infrastructure. Three to five servers, Kubernetes cluster, PostgreSQL HA (primary + 2 replicas), Kafka (3-broker cluster), MinIO, Keycloak, Kong.
  • Deploy security baseline. TLS everywhere, encryption at rest, RBAC, audit logging.
  • Build ingestion for 3–5 source systems. Debezium CDC for databases, NiFi for files, Airflow for scheduled extracts.
  • Implement deterministic matching. Start with exact-match rules on unique identifiers (NDC, NPI, CAS, DUNS).
  • Build the stewardship UI. A React application where data stewards see the match queue, review side-by-side diffs, and approve or reject merges with electronic signatures.
  • Run IQ/OQ. Validate the core platform components.
  • Go live with the pilot domain. First golden records flowing to downstream systems.

Phase 2: Expand and Govern (6–12 Months)

  • Add 2–3 more domains. HCP/HCO, Vendor, Site.
  • Implement probabilistic matching. Layer Splink or custom Python scoring on top of the deterministic rules.
  • Deploy data quality dashboards. Great Expectations results flowing into Grafana for executive visibility.
  • Integrate downstream systems. Kafka event subscriptions for ERP, CRM, LIMS, data warehouse.
  • Establish domain governance councils. Business-led groups that own definitions, matching rules, and stewardship SLAs.

Phase 3: Optimize and Validate (12–18 Months)

  • Add remaining domains. Study, Sample, Batch.
  • Implement ML-assisted matching. Anomaly detection on master data changes. Self-tuning matching thresholds.
  • Deploy data catalog. OpenMetadata for lineage, glossary, and impact analysis.
  • Complete GAMP 5 validation. Full IQ/OQ/PQ documentation for all domains.
  • Storage tiering. Hot (NVMe, 2 years), warm (SAS, 5 years), cold (tape/LTO, 15–25 years).

Cost: The Real Numbers

Software Licensing: $0

Component Commercial Equivalent Annual License Savings
MDM Platform Informatica MDM / Reltio / Stibo $500K–$2M
ETL Tool Informatica PowerCenter / Talend Enterprise $200K–$500K
Data Catalog Collibra / Alation $150K–$400K
Container Platform Red Hat OpenShift $100K–$300K
Total Annual Savings $950K–$3.2M

Infrastructure: ~$100K One-Time

Item Cost
3x Database Servers (x86, 128GB RAM, 4TB NVMe) $45,000
3x Application Servers (64GB RAM) $27,000
1x DR Server (lower spec) $12,000
NAS Storage (20TB, ZFS) $8,000
Network (switches, firewall rules) $5,000
Total ~$97,000

3-Year TCO

Approach 3-Year Total
Commercial MDM Suite $3M–$8M+
Open-Source In-House $500K–$700K
Savings $2.5M–$7.3M

The primary cost is people, not software. Budget for 2–3 senior data engineers, 1 architect, 1 data steward, and 1 validation specialist. Hire engineers who know Kafka, PostgreSQL, and Python — not consultants who know how to configure Informatica.

What to Do Monday Morning

  1. Form the Data Governance Council. Get Regulatory, Quality, Commercial, IT, and R&D in the same room. Appoint a chair (CDO or VP Quality). The hardest part of MDM is not the technology — it is getting the business to agree on what “golden” means and who gets to decide.

  2. Profile your top 3 source systems. Run a duplication analysis on your ERP, CRM, and LIMS. Measure the problem. Present the numbers to the executive team. “We have 23% duplication in our product master and 31% in our HCP data” is the business case that opens the budget.

  3. Provision a sandbox. Three VMs, PostgreSQL, Kafka, Airflow. Get a working pipeline from one source system to a staging table in a weekend. Prove the stack works before you scale it.

  4. Pick your pilot domain. Product/Material if you want structured, low-risk quick wins. HCP/HCO if the commercial teams need relief now. Either works — the platform is domain-agnostic.

  5. Start small, deliver fast, then expand. Golden records for one domain in three months. Three domains in twelve months. Full platform in eighteen months. Each phase delivers measurable value that justifies the next phase’s investment.

The companies that succeed with MDM are not the ones with the biggest budgets or the fanciest tools. They are the ones that treat it as a governance project with a technology platform — not a technology project with a governance afterthought. Build the governance first. The technology will follow.