You know that perfect moment when your pipeline runs on time and dashboards look great?

Then you check Slack: “Why are all customer emails NULL?”

Your heart sinks. The job succeeded. Data wrote successfully. But 18% of rows are garbage.

For months, I fought this with inline checks:

1
2
3

.filter($"email".isNotNull)
.filter($"amount" > 0)
.filter($"date" <= current_date())

Copy-pasted across 50 pipelines. No reuse. No tests. No automation.

Then I built DQF (Data Quality Framework) + DPAT (Declarative Pipeline Assurance Tool).

The result:

• Constraint-based validation (.isNotNull.matchesRegex(...))
• YAML declarative checks (expected_range: [50000, 60000])
• Automated CI/CD gates
• JSON reports + Slack alerts
• No more inline WHERE clauses

This post shows you the complete implementation.

Why this matters

Here’s the data quality reality:

Before (inline chaos):

• Quality checks scattered across code
• Copy-paste WHERE clauses everywhere
• No central validation
• Discover issues in production
• Manual testing

The cost:

• NULL data in dashboards
• Stakeholder trust destroyed
• Hours debugging “why did this pass?”
• No reusability
• No automation

What I needed:

• Reusable constraint library
• Declarative test definitions
• Automated execution
• Clear failure reporting
• CI/CD integration

This is the framework that solved it.

Part 1 - What the framework does

Think of this as scalatest meets dbt tests meets common sense:

• ✅ Schema checks: missing columns, data types, required fields
• ✅ Value constraints: uniqueness, nullability, allowed ranges
• ✅ Volume & frequency checks: “is my partition even here?”
• ✅ Aggregation deltas: “is this spike normal?”

Part 2 - DPAT: YAML-based declarative checks

Here’s a real example from our YAML-based DPAT engine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

checks:
  - name: transactions_have_valid_amounts
    type: range
    column: transaction_amount
    min: 0
    max: 99999
    severity: error

  - name: daily_volume_sanity
    type: row_count
    expected_range: [50000, 60000]
    severity: warning

You define expectations. It tests them. CI/CD blocks if it must.

Part 3 - Constraint DSL in Scala

Here’s how we express constraints in the DQ library:

1
2
3
4
5

Constraint
  .on("user_id")
  .isNotNull
  .and("email").matchesRegex(".+@.+\\..+")
  .and("signup_date").isInPast

You can attach them inline with dataframes or bind them to Spark Datasets with implicits.

Part 4 - Failure reporting (JSON + dashboards)

All test results emit to JSON, Slack webhooks, and Looker dashboards.

1
2
3
4
5
6

{
  "check": "transactions_have_valid_amounts",
  "status": "FAILED",
  "failures": 328,
  "details": "328 rows exceeded max=99999"
}

Or you can just scroll through a dq_results.html report with filters by severity.

Part 5 - Integration patterns

You can wire this into:

• Airflow DAGs (as standalone Python ops via REST)
• Spark ETL pipelines (Scala SDK)
• Shell scripts (CLI + exit codes)
• CI/CD (GitHub Actions / GitLab CI)

Future: open source plans

This started internally, but parts of it are being modularized. The goal is to OSS:

• Reusable constraint library
• YAML-based test definition DSL
• CLI test runner + report UI

Conclusion

Building a data quality framework transformed how we ship pipelines.

Before:

• Inline WHERE clauses everywhere
• No reusability
• Discover issues in production
• Manual testing

After:

• Constraint-based validation
• YAML declarative checks
• Automated CI/CD gates
• Catch issues before deployment

The key? Make quality checks first-class citizens, not afterthoughts.

If you’re still writing inline checks, stop. Your future self will thank you.

TL;DR

• The problem: Inline quality checks scattered across code, copy-pasted WHERE clauses, NULL data in production, no reusability
• The solution: DQF (constraint library) + DPAT (YAML-based declarative checks) + CI/CD integration
• DQF features: Schema validation, value constraints, volume checks, aggregation deltas, “is my partition here?”
• DPAT example: YAML defines range, row_count, not_null checks with severity: error/warning
• Constraint DSL: .on("user_id").isNotNull.and("email").matchesRegex(...).and("signup_date").isInPast
• Scala integration: Attach constraints inline with DataFrames or bind to Datasets with implicits
• Failure reporting: JSON output, Slack webhooks, Looker dashboards, HTML reports with severity filters
• Integration points: Airflow DAGs (REST API), Spark ETL (Scala SDK), shell scripts (CLI), CI/CD (GitHub Actions)
• Architecture: Define checks in YAML → DPAT engine validates → JSON results → CI/CD blocks if failed
• Future: Open source plans for constraint library, YAML DSL, CLI runner, report UI
• Bottom line: Stop writing inline quality checks - build reusable, testable, automated validation
• Key insight: Quality gates as first-class citizens, not WHERE clause afterthoughts