You know that 3 AM page: “The executive dashboard is empty. What happened?”

Your data pipeline failed. Again. Manual rollback time. Stakeholder emails. Root cause analysis. 3 hours of your life gone.

For months, I lived this cycle:

• 12% delivery error rate (1 in 8 releases broke)
• 5 pipeline failures per week
• 3 hours to fix each incident
• Manual rollbacks every time

Then I built DPAT (Declarative Pipeline Assurance Tool), gated our CI/CD, and enforced data contracts.

The result:

• Errors dropped 12% → 2% (82% reduction)
• Fixes went 3 hours → 30 minutes
• Failures: 5/week → 0-1/week

This post shows you the complete implementation - YAML data quality checks, CI/CD gates, and data contract enforcement.

Why this matters

Here’s the data pipeline reality:

Before (the chaos):

• Ship fast, break things
• No quality gates
• Manual testing (when you remember)
• Discover issues in production
• 3 AM firefights

The cost:

• 12% of releases fail
• 3 hours per fix (minimum)
• Lost stakeholder trust
• Your sanity

What I needed:

• Automated quality gates
• Fast failure detection
• Automatic rollback
• Clear ownership
• Less stress

This is the system that solved it.

Part 1 - DPAT: declarative data quality checks in YAML

Define YAML checks like this:

1
2
3
4
5
6
7
8
9

checks:
  - name: "order_date_not_null"
    column: order_date
    type: not_null
    severity: error
  - name: "daily_volume_range"
    type: row_count
    expected_range: [ 1_000_000, 1_100_000 ]
    severity: error

Then wired that into our Spark ETL:

1
2

val results = DPAT.runChecks(df, "orders", date = runDate)
assert(results.isSuccess)

Part 2 - Gated CRQ: quality gates in CI/CD

1
2
3
4
5
6

- name: Run DPAT checks
  run: sbt "runDPAT --date ${{ env.RUN_DATE }}"

- name: Gated CRQ
  if: steps.runDPAT.outcome == 'failure'
  run: exit 1

Failed checks stop the build and prevent broken data releases.

Part 3 - Data contracts: clear ownership and schema enforcement

We published data schema/contracts like:

Since enforcing those schemas, compatibility bugs dropped by 45%.

The results: before vs after

That’s how we cut delivery errors by 82%.

Architecture: how it all fits together

1
2
3
4

[Source System] → [Spark ETL Job] → DPAT Checks → Data Contract Validator → Data Lake
                            │
                            ↓ (on fail)
                  [CI/CD Rollback & Alerts]

The flow:

1. Spark ETL processes data
2. DPAT runs quality checks (nulls, volumes, ranges)
3. Data contracts validate schemas
4. On success → write to Data Lake
5. On failure → rollback + alert

Conclusion

Building quality gates transformed our data platform.

Before:

• 12% failure rate
• 3-hour firefights
• Manual rollbacks
• Lost trust

After:

• 2% failure rate
• 30-minute fixes
• Automated everything
• Confidence restored

The key? Shift left - catch issues before production, not after.

How to apply this

If you’re facing similar issues:

1. Build declarative checks - YAML-based quality rules
2. Gate your CI/CD - Failed checks = blocked deployment
3. Enforce data contracts - Clear ownership and schemas
4. Automate rollbacks - Don’t rely on manual intervention

Start small - one pipeline, one set of checks. Then expand.

TL;DR

• The problem: 12% delivery error rate, 5 failures/week, 3 hours per fix, manual rollbacks, 3 AM pages
• The solution: DPAT (YAML quality checks) + Gated CRQ (CI/CD gates) + Data contracts (schema enforcement)
• DPAT: Declare data quality rules in YAML (not_null, row_count ranges, custom validations)
• Implementation: Spark ETL runs DPAT checks, asserts success before writing to Data Lake
• Gated CRQ: CI/CD pipeline fails if DPAT checks fail, prevents broken releases
• Data contracts: Published schemas with clear ownership, version control, compatibility validation
• Architecture: Source → Spark → DPAT → Contract Validator → Lake (with rollback on failure)
• Results: 12% → 2% error rate (82% drop), 5 → 0-1 failures/week, 3hr → 30min resolution time
• Impact: Compatibility bugs dropped 45%, better confidence, automated enforcement, fewer firefights
• Key lesson: Shift left - catch issues in CI/CD, not production
• How to start: Build declarative checks, gate deployments, enforce contracts, automate rollbacks