Data Engineering for Pharma: What's Different About Building Data Platforms for Regulated Industries

After building data platforms for a global pharmaceutical services company operating in 130+ countries, I can tell you: the tools are the same. Databricks. Delta Lake. Terraform. PySpark. Medallion architecture. Everything in the modern data stack works in pharma exactly as it works in retail or manufacturing.

What changes is the priority ordering. And that changes everything about how you design.

Data Quality Is Non-Negotiable

In retail, a bad record means a wrong number on a dashboard. Someone notices, files a ticket, it gets fixed in the next pipeline run. The business impact is a slightly inaccurate weekly report.

In pharma, a bad record in supply chain data can mean a patient doesn’t receive medication. A data quality issue in a managed access program — where the company provides medications to patients who can’t access them through normal channels — has direct human consequences. This isn’t hyperbole. These programs operate across 130+ countries, and the data infrastructure is what connects patient need to drug availability.

This changes how you think about data quality from an architectural concern to a regulatory and operational requirement. The quarantine table pattern — where records failing validation are routed to a separate table with error metadata instead of being silently dropped or coerced — isn’t defensive programming. It’s the minimum bar.

Every record that fails validation must be reviewed. Not “flagged for optional review.” Reviewed. The quarantine table needs to capture what failed, which rule triggered the failure, the severity classification, and enough context for a data steward to determine whether the issue is a source system problem, a transformation bug, or a legitimate edge case.

We implemented this as a config-driven data quality engine where validation rules are defined per source system in structured configuration files. Null checks, range validations, referential integrity, format checks, and business-specific rules — all externalized from pipeline code. When a new regulatory requirement introduces a new validation constraint, the rule is added to the config. No pipeline redeployment.

The key design decision: the quarantine stream runs in parallel with the clean stream. The pipeline doesn’t block on quality failures. Clean records flow forward to silver and gold layers for downstream consumption. Failed records accumulate in quarantine for review. This means the platform is always producing output — the quarantine backlog doesn’t halt operations.

Audit Trails Are the Primary Data Structure

Pharmaceutical regulators don’t ask “what’s the current state of your data?” They ask “show me every transformation applied to this record from the moment it entered your system to the moment it appeared in this report.”

This is a fundamentally different question, and it changes how you design your medallion architecture.

In most implementations, the bronze layer is treated as a staging area. Data lands, gets processed, moves to silver, and the bronze tables can be compacted, archived, or even dropped after a retention period. The “real” data lives in silver and gold.

In pharma, the bronze layer is the audit-grade source of truth. It’s the authoritative record of what the source system sent, when it sent it, and in what format. You don’t compact bronze. You don’t drop it. You maintain it as the starting point of a traceable lineage chain that a regulator can follow from source to report.

This has practical implications:

• Bronze table design — every bronze record gets ingestion metadata: source system identifier, ingestion timestamp (wall clock, not source timestamp), batch or job identifier, and a record hash. These columns cost nothing to add and save enormous amounts of debugging and audit time.
• Silver transformation logging — every transformation applied at the bronze-to-silver transition is logged. Not just “this record was transformed” but “these specific rules were applied, these fields were modified, this was the before state and this is the after state.”
• Gold derivation tracking — aggregations and business logic in the gold layer maintain references back to the silver records they were derived from. A number on a report can be decomposed into its constituent records.

This sounds like a lot of overhead. It is. But it’s not optional overhead — it’s the core deliverable. The platform’s purpose is to produce auditable data, and the audit trail is what makes the data trustworthy.

Multi-Country Data Residency Is an Architectural Constraint

Global pharma operations span 130+ countries. Regulatory requirements vary by country and by data type. Patient-related data in the EU falls under GDPR. Clinical trial data has country-specific reporting requirements. Pharmacovigilance data (adverse event reporting) has its own regulatory framework per jurisdiction.

The practical implication: you can’t put everything in one data lake in us-east-1 and call it a day.

The platform architecture needs regional data boundaries. Certain data stays in specific geographic regions — not just because of regulation, but because of data sovereignty requirements that the company’s legal and compliance teams have determined apply to their operations.

At the same time, the company needs consolidated global reporting. The VP of Supply Chain needs to see worldwide inventory levels. The regulatory affairs team needs cross-country pharmacovigilance summaries. The managed access team needs a global view of program utilization.

This creates a tension: data must stay local, but reporting must be global. We resolved this with a tiered approach:

• Source data stays regional — bronze and silver layers respect geographic boundaries
• Aggregated and anonymized data flows to a global layer — gold-level aggregations that don’t contain patient-level or restricted data can be consolidated for global reporting
• Access controls enforce boundaries — Unity Catalog’s fine-grained permissions ensure that even users with global reporting access can’t drill into restricted regional data

The architecture isn’t elegant. It’s an honest reflection of the regulatory reality. When you operate in 130+ countries, there is no clean solution — only solutions that respect the constraints.

Config-Driven Is the Only Way

Pharmaceutical regulations change. New reporting requirements are introduced. Existing requirements are modified. Countries adopt new data protection frameworks. Industry bodies update standards.

When this happens, you can’t wait for an engineering sprint to update pipeline code, test it, and deploy it. The regulatory team needs to modify validation rules, schema mappings, and reporting logic on their timeline, not yours.

This is why we externalize everything we can into configuration:

• Data quality rules — defined in config files per source system, applied by the DQ engine at runtime
• Schema definitions — expected schemas for each source system, used for validation and transformation
• Transformation logic — parameterized transforms that read their behavior from configuration
• Reporting templates — report structures defined in config, populated by the platform

The pipeline code itself changes infrequently. What changes are the configurations that the pipeline code interprets. This separation means the regulatory team can modify the system’s behavior through config changes that go through a lighter review process than code changes.

This isn’t a pharma-specific insight — we use config-driven patterns in every industry. But in pharma, the business case for config-driven architecture is strongest because the rate of regulatory change is highest.

What Stays the Same

The tools are identical. Databricks, Delta Lake, Unity Catalog, Terraform, PySpark — everything in the modern data stack works for pharma. The medallion architecture works. The design patterns work.

What changes is the priority ordering:

• Correctness over speed — a pipeline that processes every record correctly in 4 hours beats a pipeline that processes 99.5% correctly in 20 minutes
• Auditability over simplicity — the extra metadata columns, the transformation logs, the lineage tracking add complexity. That complexity is the product
• Governance over agility — access controls, data residency, and audit trails slow down development. That’s the trade-off for operating in a regulated industry

If you’re building a data platform for pharma and you’re using the same design priorities you’d use for a consumer tech company, something is wrong. The tools are the same. The priorities are not.