Session title*

Type-safe Data Engineering - Compile-time contracts & Fiber safety

Description*

Data pipelines rarely fail because of exotic algorithms. They fail because schemas drift quietly and side effects leak loudly. This session shows how to push both problems left: make the compiler act as a gate and make the runtime behave predictably. Part one encodes schema intent as compile-time data contracts in Scala 3. Macros build a structural view of types and compare changes against explicit policies like backward, forward, full, or none. Illegal changes surface as precise, human-readable errors before deploy, not at 2 a.m. Part two layers fiber-safe execution so concurrency, cancellation, and retries are supervised rather than improvised. Effects are scoped and cleaned up, so a retry does not mean duplicate writes or leaked handles.

What you can expect:

• Compiler as gate: contracts and sinks required, or it will not compile.
• Runtime as chaperone: fiber-safe effects, predictable cancellation, idempotent retries.
• Algebra stays constant: swap Spark, Flink, and Kafka runners via a trait, not a rewrite.
• Scaffolding behaves: no code generation until types align.

Along the way we keep transformations pure in your code, push side effects to the edges, and align with real-world evolution rules in table formats and schema registries. The result is a style of data engineering that reads clearly in code reviews and ports across engines without rewrites. Attendees leave with patterns, small steps to adopt in existing repos, and a Monday-ready checklist to cut works on my machine and surprised in production at the same time.

Outline of the presentation?*

0. Cold open - the 2 a.m. pager

Schema rename slips in, nothing crashes, finance discovers nulls later. Set stakes and name the villain: drift and effect leaks. Outcome audience knows why “compiler as gate” matters.

1. Thesis - fail early, run safely

Make the compiler refuse illegal structures and keep concurrency fiber-safe so retries and cancellation are clean.

2. Boundaries that keep you honest

Compile-time vs runtime - what each is good for. DX vs process - fast local red to green vs CI policy gates. Align expectations with real-world mess.

3. Contracts at compile time

Policy lattice: Exact, Backward, Forward, Full. Scala 3 macros derive schema AST, compare under policy, emit evidence or a human diff. Show a tiny compile error that points to the exact field.

4. Red to green demo

Start with a mismatch that fails under Exact, relax to Backward for a safe migration, turn tests into compile-fail gates.

5. Composable effects without smuggling side effects

Kleisli as the spine: sources to transforms to sinks as A => F[B] that actually compose. Pure inside, effects at the edges.

6. Fiber-safe execution

Structured concurrency, Resource lifecycles, predictable cancellation, no leaked handles, no duplicate writes on retries. Works with Cats Effect or ZIO.

7. Engine portability seam

Spark, Flink, Kafka as pluggable runners. Keep contracts and composition, swap the interpreter.

8. Edges with the outside world

Table evolution in Iceberg, schema compatibility in registries. Contracts in code, explicit evolution at the boundary.

9. Tradeoffs and rollout

Where macros are worth it, when to stop at runtime checks, how to adopt incrementally. A 4-week playbook.

10. Close and Q&A

Recap and a checklist repo to try on Monday.

Practical takeaways*

Attendees will leave with-

1. Practical patterns, tradeoffs, and code examples for building data systems.
2. Data systems that will be safer, more predictable, & more maintainable.
3. Encode schema intent as policies and let the compiler reject illegal changes before deploy.
4. Compose effectful stages with Category theory-Kleisli so pipelines stay readable, testable, & side effects stay at the edges.
5. Keep engines pluggable by separating pipeline algebra from runners for Spark, Flink, & Kafka.

Track*

Engineering

Keywords (comma separated)

Scala 3, Compile-time macros, Data Engineering, Type safety, Kleisli, Cats Effect, ZIO, Spark, Flink, Kafka, Schema evolution, Apache Iceberg, Schema Registry

Type of Talk*

Technical

Level of talk*

Intermediate / Practitioner

Have you delivered this talk before? Where?*

Yes. https://scala.io/sessions/paris-2025/compile-time-contracts-fiber-safe-data-pipelines

The recording of this talk can be made public*

Yes.

Do you want to have a speaker coach session?*

Yes.

Requirements (Please let us know if you have any technical requirments or date restrictions. We will schedule your talk accordingly)

We do not require any special setup beyond internet and live code presentation support.

For organizers only - supporting links (not for reviewers):

• https://vitthalmirji.com/2025/10/category-theory-for-data-engineers-the-kleisli-pattern-that-makes-data-pipelines-compose/
• https://vitthalmirji.com/2025/09/compile-time-data-contracts-catch-schema-mismatches-at-compile-time-in-scala-3/
• https://vitthalmirji.com/2025/06/functional-utilities-for-data-engineering-stop-repeating-yourself-in-scala-pipelines/