<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Vitthal Mirji</title><link>https://vitthalmirji.com/tags/metaprogramming/</link><description>Technical articles on Scala, data engineering, functional programming, and software architecture by Vitthal Mirji.</description><language>en-us</language><copyright>Vitthal Mirji</copyright><managingEditor>vitthalmirji@gmail.com (Vitthal Mirji)</managingEditor><lastBuildDate>Fri, 06 Nov 2026 13:45:55 +0000</lastBuildDate><atom:link href="https://vitthalmirji.com/tags/metaprogramming/index.xml" rel="self" type="application/rss+xml"/><image><url>https://vitthalmirji.com/images/default-og.png</url><title>Vitthal Mirji</title><link>https://vitthalmirji.com/</link></image><item><title>DET week 2 homework 1: concision pass on compile-time contracts with redlines</title><link>https://vitthalmirji.com/2025/12/det-week-2-homework-1-concision-pass-on-compile-time-contracts-with-redlines/</link><guid isPermaLink="true">https://vitthalmirji.com/2025/12/det-week-2-homework-1-concision-pass-on-compile-time-contracts-with-redlines/</guid><pubDate>Wed, 03 Dec 2025 00:00:00 +0000</pubDate><dc:creator>vitthal-mirji</dc:creator><category>technical-writing</category><category>det-workshop</category><category>concision</category><category>detail-audit</category><category>detail-jobs</category><category>word-reduction</category><category>scala</category><category>scala-3</category><category>compile-time</category><category>data-contracts</category><category>metaprogramming</category><category>yaakov-bressler</category><category>homework</category><category>workshop-assignment</category><category>week-2</category><description>Week 2 concision homework that redlines the compile-time contracts article. Cut over 25% by striking details with no job, keep before and after visible, and explain each cut so the reader sees what stayed. Shows trade-offs behind each cut.</description><content:encoded><![CDATA[<h2 id="workshop-context-det-week-2---option-1-concision">Workshop context (DET week 2 - Option 1: Concision)</h2>
<ul>
<li>Source article: <a href="/compile-time-data-contracts">Compile-time data contracts</a>
</li>
<li>Task: cut &gt;25% of the compile-time contracts article by removing details without a job (Orient, Credibility, Texture, Mechanical, Contrast, Anchor).</li>
<li>Rule: no hard deletes. Before stays visible via strike-through; after stays as plain text.</li>
<li>Reference process: Week 1 redline model + week-2 detail rules.</li>
</ul>
<h2 id="legend">Legend</h2>
<ul>
<li>Plain text = kept in the concise reading path (after).</li>
<li><del>Strike-through</del> = removed from main path (before, still visible).</li>
<li>Admonitions = why a removal/keep decision was made.</li>
</ul>
<h2 id="concision-flow">Concision flow</h2>
<pre class="mermaid">flowchart TD
    A[Copy full article] --&gt; B[Tag each block with job]
    B --&gt; C{Job present?}
    C --&gt;|Yes| D[Keep]
    C --&gt;|No| E[Strike + admonition]
    D --&gt; F[Re-read for &gt;25% cut]
    E --&gt; F
    F --&gt; G[Publish redline with before/after visible]
    style E fill:#ffe6e6,stroke:#d33
    style D fill:#e6ffea,stroke:#2c9
</pre><p>What if broken pipelines never launched - because the compiler stopped them?</p>
<blockquote>
  <p>&ldquo;If it compiles, contracts align.&rdquo;</p>

</blockquote><p>I&rsquo;ve been working on this problem for a while now. You know how it goes - you&rsquo;re running a data pipeline, everything&rsquo;s fine, then someone changes a field name upstream and boom, your job crashes at 2 AM. The on-call engineer (probably you) gets paged, and you spend the next hour figuring out what went wrong.</p>
<p>Once, an upstream team silently renamed a column <code>amount</code> → <code>amt</code>. Our pipeline didn&rsquo;t crash immediately - it happily wrote nulls for weeks. By the time we caught it, we had to backfill millions of records. That&rsquo;s the day I realized: runtime validation catches problems late; compile-time contracts catch them earlier.</p>
<p>This post shows you how to catch these issues at compile time. Not at runtime. Not even at test time. At compile time. Your code literally won&rsquo;t build if schemas drift.</p>
<p>We&rsquo;ll build this from scratch - first understand how Scala 3 macros work, then write a minimal contract system that proves schema compatibility before your code even runs. I&rsquo;ll also show the Scala 2 version because, let&rsquo;s be honest, many teams are still on Scala 2.</p>
<p>All code here runs in a vanilla sbt project. You can grab the working code in github from <code>https://github.com/vim89/compile-time-data-contracts</code> and run it yourself.</p>
<h2 id="why-this-matters">Why this matters</h2>
<p>Look, I get it. Writing tests is important. But tests are samples. They check what you think might break. The compiler, though? It&rsquo;s exhaustive for structure. It can prove your schemas match - every field, every type, every time.</p>
<p>Here&rsquo;s the thing - schemas change all the time in real projects. And when they do:</p>
<ul>
<li>Your build should fail early, not your production job</li>
<li>Keep transformations pure and effects at the edges (so retries don&rsquo;t duplicate writes)</li>
<li>Let the compiler do the heavy lifting</li>
</ul>
<p><strong>Important upfront:</strong> This focuses on preventing schema drift within your controlled boundaries - the systems you build, deploy, and version. It doesn&rsquo;t replace runtime validation for external data sources or third-party APIs. Think of it as catching a large class of schema mismatches during development and deployment, not eliminating all drift everywhere. You&rsquo;ll still need runtime checks for data coming from systems you don&rsquo;t control.
<div class="admonition warning">
  <div class="admonition-title">Scope boundary</div>
  <div class="admonition-content">
    Compile-time contracts catch drift in your controlled codebase - the transforms you write, the case classes you define, the schemas you version together. They won&rsquo;t catch upstream API changes, late-arriving schema registry updates, or external data sources that change independently. Use compile-time for what you control, runtime validation for everything else. Defense in depth, not either/or.
  </div>
</div></p>
<h2 id="what-well-build">What we&rsquo;ll build</h2>
<p>We&rsquo;re going to build a small compile-time validation system:</p>
<ol>
<li>A way to describe case class structure at compile time (we&rsquo;ll call it &ldquo;shape&rdquo;)</li>
<li>A policy system - Exact, Backward, Forward compatibility modes</li>
<li>A macro that proves &ldquo;these two shapes conform under policy P&rdquo; - or refuses to compile</li>
<li>A phantom-typed pipeline builder that enforces correct construction order</li>
</ol>
<p>First, Scala 3 (using inline + quotes), then Scala 2 (using blackbox macros). I&rsquo;ll point out the differences as we go.</p>
<h3 id="run-the-code-in-github">Run the code in github</h3>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone https://github.com/vim89/compile-time-data-contracts.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> compile-time-data-contracts
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Scala 3</span>
</span></span><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition info">
  <div class="admonition-title">Information</div>
  <div class="admonition-content">
    The code in github includes working examples you can run immediately. All compile-fail cases are documented inline - uncomment them to see the compiler errors.
  </div>
</div>
<hr>
<p><del>## Part 1 - Scala 3: the mental model (inline + quotes)</del></p>
<p><del>Scala 3 macros are different from Scala 2. Better, actually. They&rsquo;re safer and easier to reason about.</del></p>
<p><del>Two main ideas:</del>
<del>- <code>inline def</code>: tells the compiler &ldquo;expand this at compile time&rdquo;</del>
<del>- <code>quotes</code> + <code>Expr[T]</code>: gives you a typed AST that you can inspect</del></p>
<div class="admonition tip">
  <div class="admonition-title">Quick win</div>
  <div class="admonition-content">
    If you&rsquo;re new to Scala 3 macros, start by reading macro-generated code in the REPL: sbt&gt; console, then scala&gt; import scala.quoted.*, then inspect what your macros actually expand to. Understanding the generated code is 80% of debugging macro issues. The compiler&rsquo;s doing the work - you just need to see what it&rsquo;s actually producing.
  </div>
</div>
<div class="admonition warning">
  <div class="admonition-title">❌ Removed: Part 1 - Scala 3: the mental model</div>
  <div class="admonition-content">
    Jobless/duplicate for concision objective.
  </div>
</div>
<h2 id="part-2---describe-shapes-at-compile-time-scala-3">Part 2 - Describe shapes at compile time (Scala 3)</h2>
<p>Now we need to describe case class structure. In real projects, you might use Shapeless, Magnolia, or Scala 3&rsquo;s Mirror. I&rsquo;ll show a compact Mirror-based version here because it&rsquo;s self-contained.</p>
<p>The idea: a typeclass <code>Shape[A]</code> that knows the fields of <code>A</code>.</p>
<pre class="mermaid">classDiagram
    class Field {
      +name: String
      +tpe: String
      +hasDefault: Boolean
      +isOptional: Boolean
    }
    class Shape {
      +fields(): List~Field~
    }
    Shape &#34;1&#34; --&gt; &#34;*&#34; Field : returns
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Shape.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.deriving.Mirror</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">tpe</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">hasDefault</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Shape</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>     <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">A</span><span class="o">]]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span> <span class="c1">// element optionality handled elsewhere
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="n">given</span> <span class="n">derived</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">using</span> <span class="n">m</span><span class="k">:</span> <span class="kt">Mirror.Of</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">m</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">p</span><span class="k">:</span> <span class="kt">Mirror.ProductOf</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">labels</span> <span class="k">=</span> <span class="n">constValueTuple</span><span class="o">[</span><span class="kt">p.MirroredElemLabels</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">types</span>  <span class="k">=</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">p.MirroredElemTypes</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">zipped</span> <span class="k">=</span> <span class="n">zip</span><span class="o">(</span><span class="n">labels</span><span class="o">,</span> <span class="n">types</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">          <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">            <span class="n">zipped</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">,</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span> <span class="k">=</span> <span class="n">tpe</span><span class="o">.</span><span class="n">startsWith</span><span class="o">(</span><span class="s">&#34;Option[&#34;</span><span class="o">))</span> <span class="o">}.</span><span class="n">toList</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">import</span> <span class="nn">scala.compiletime.</span><span class="o">{</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">constValueTuple</span><span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">T</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">inline</span> <span class="n">erasedValue</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">h</span><span class="o">]</span> <span class="k">:</span><span class="kt">:</span> <span class="k">type</span><span class="kt">Names</span><span class="o">[</span><span class="kt">t</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">constValue</span><span class="o">[</span><span class="err">&#34;&#34;</span> <span class="kt">+</span> <span class="kt">T</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">zip</span><span class="o">[</span><span class="kt">L</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span>, <span class="kt">R</span> <span class="k">&lt;:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">labels</span><span class="k">:</span> <span class="kt">L</span><span class="o">,</span> <span class="n">types</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">labels</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">constValue</span><span class="o">[</span><span class="kt">h</span><span class="o">].</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">-&gt;</span> <span class="n">types</span><span class="o">.</span><span class="n">head</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">zip</span><span class="o">[</span><span class="kt">t</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">types</span><span class="o">.</span><span class="n">tail</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Yeah, it&rsquo;s a bit dense. But here&rsquo;s what it does:</p>
<ul>
<li>Uses Scala 3&rsquo;s Mirror to reflect on case classes at compile time</li>
<li>Extracts field names and types</li>
<li>Marks optional fields (those wrapped in <code>Option</code>)</li>
<li>Returns a <code>List[Field]</code> describing the structure</li>
</ul>
<pre class="mermaid">flowchart LR
    A[&#34;Case class A(...)&#34;] --&gt; M[&#34;Mirror.Of[A]&#34;]
    M --&gt; L[&#34;MirroredElemLabels&#34;]
    M --&gt; T[&#34;MirroredElemTypes&#34;]
    L --&gt; Z[&#34;zip(labels, types)&#34;]
    T --&gt; Z
    Z --&gt; S[&#34;Shape[A].fields: List[Field]&#34;]
</pre><p>Usage:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ShapeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">note</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkShape</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="n">println</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">fields</span><span class="o">)</span> <span class="o">//</span> <span class="nc">List</span><span class="o">(</span><span class="nc">Field</span><span class="o">(</span><span class="s">&#34;id&#34;</span><span class="o">,</span><span class="s">&#34;Long&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;email&#34;</span><span class="o">,</span><span class="s">&#34;String&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;note&#34;</span><span class="o">,</span><span class="s">&#34;Option[String]&#34;</span><span class="o">,</span><span class="kc">false</span><span class="o">,</span><span class="kc">true</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Now we can ask the compiler &ldquo;what fields does this case class have?&rdquo; at compile time. That&rsquo;s the foundation.</p>
<blockquote>
  <p>If you prefer libraries, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works great for both Scala 2 and 3.</p>

</blockquote><hr>
<h2 id="part-3---policies-exact--backward--forward">Part 3 - Policies: Exact / Backward / Forward</h2>
<p>Here&rsquo;s where we define compatibility rules. Think of these as your migration strategies:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Policy.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Policy</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Policy</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">Exact</span>     <span class="kt">extends</span> <span class="kt">Policy</span>
</span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Backward</span>  <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// producer can add optional/defaulted fields
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Forward</span>   <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// consumer tolerates extra fields
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Exact</span>    <span class="k">extends</span> <span class="nc">Exact</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Backward</span> <span class="k">extends</span> <span class="nc">Backward</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Forward</span>  <span class="k">extends</span> <span class="nc">Forward</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Why three policies?</p>
<ul>
<li><strong>Exact</strong>: Schemas must match exactly. No surprises. Use this for critical paths.</li>
<li><strong>Backward</strong>: Producer can add optional fields. Useful when rolling out new features without breaking consumers.</li>
<li><strong>Forward</strong>: Consumer ignores extra fields. Useful when the producer might send more than you need.</li>
</ul>
<pre class="mermaid">flowchart LR
    EX[&#34;Policy.Exact&#34;]
    BW[&#34;Policy.Backward&#34;]
    FW[&#34;Policy.Forward&#34;]

    EX --- BW
    EX --- FW

    EX:::tight --&gt;|no diffs allowed| OK1[&#34;✔ match only&#34;]
    BW:::relax --&gt;|allow missing optional in producer\nallow extra fields in producer| OK2[&#34;✔ migration adds&#34;]
    FW:::relax --&gt;|allow extra fields in consumer side| OK3[&#34;✔ tolerant reader&#34;]

    classDef tight fill:#eef,stroke:#446;
    classDef relax fill:#efe,stroke:#484;
</pre><p>Visual (save/share):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/policy-lattice.svg"
            alt="Policy lattice"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    A tiny policy lattice - Exact / Backward / Forward.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<p>In practice: start with Exact, relax to Backward/Forward during migrations, then tighten back to Exact once stable.
<div class="admonition success">
  <div class="admonition-title">Migration strategy</div>
  <div class="admonition-content">
    The policy progression that works in production: Exact for stable pipelines, Backward during producer rollouts (allows new optional fields), Forward during consumer updates (tolerates extras), then back to Exact after migration completes. Document the expected timeline - &ldquo;Backward policy through Q2 migration, then tightening to Exact.&rdquo; Prevents policy drift from becoming permanent.
  </div>
</div></p>
<hr>
<p><del>## Part 4 - How this fits with existing tools</del></p>
<p><del>Before we dive deeper, you might be thinking: &ldquo;We already have Avro, Protobuf, schema registries, and Great Expectations. Why add compile-time contracts?&quot;</del></p>
<p><del>Fair question. Here&rsquo;s how they compare:</del></p>
<p><del><strong>Schema Registry (Confluent, AWS Glue)</strong></del>
<del>- <strong>What it does</strong>: Centralized schema storage with version control</del>
<del>- <strong>When it helps</strong>: Runtime validation across services, schema evolution tracking</del>
<del>- <strong>The gap</strong>: You find mismatches when the job runs, not when you compile</del>
<del>- <strong>Use together</strong>: Schema registry for cross-service contracts; compile-time for intra-repo checks</del></p>
<p><del><strong>Avro / Protobuf</strong></del>
<del>- <strong>What they do</strong>: Binary serialization with embedded schemas</del>
<del>- <strong>When they help</strong>: Network efficiency, strong contracts between services</del>
<del>- <strong>The gap</strong>: Schema compatibility is checked at runtime or via external tooling</del>
<del>- <strong>Use together</strong>: Avro/Protobuf for wire format; compile-time checks to ensure your case classes match those schemas</del></p>
<p><del><strong>Great Expectations / Deequ</strong></del>
<del>- <strong>What they do</strong>: Runtime data quality validation (nulls, ranges, distributions)</del>
<del>- <strong>When they help</strong>: Catching bad data values, statistical anomalies</del>
<del>- <strong>The gap</strong>: They validate data content, not schema structure at compile time</del>
<del>- <strong>Use together</strong>: Compile-time for structure; Great Expectations for data quality</del></p>
<p><del><strong>The compile-time advantage</strong>: Catches drift in your controlled codebase before deployment. If your transform expects field <code>X</code> but the source produces field <code>Y</code>, your build fails - not your production job. Think of it as an additional safety layer for systems you build and version together.</del></p>
<p><del>Real scenario: You have Avro schemas in a registry, but your Scala pipeline reads them into case classes. Compile-time contracts verify those case classes match the expected contract before you deploy. If upstream changes the Avro schema, your build breaks - you don&rsquo;t wait for the job to crash.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ Removed: Part 4 - How this fits</div>
  <div class="admonition-content">
    Jobless/duplicate for concision objective.
  </div>
</div>
<hr>
<h2 id="part-5---understanding-typerepr-the-compilers-view-of-types">Part 5 - Understanding TypeRepr: The compiler&rsquo;s view of types</h2>
<p>Before we dive into the macro, you need to understand <code>TypeRepr</code> - Scala 3&rsquo;s way of representing types during compilation.</p>
<p>Think of it like this: When you write <code>List[String]</code>, the compiler doesn&rsquo;t just see text. It sees a structured tree:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">AppliedType(List, [String])
</span></span><span class="line"><span class="cl">   │               │
</span></span><span class="line"><span class="cl">   │               └─ TypeRepr for String
</span></span><span class="line"><span class="cl">   └─ Type constructor</span></span></code></pre></td></tr></table>
</div>
</div><pre class="mermaid">flowchart TB
    A[&#34;List[String]&#34;] --&gt; B[&#34;AppliedType(List,[String])&#34;]
    B --&gt; C[&#34;tycon = List&#34;]
    B --&gt; D[&#34;args = [String]&#34;]
</pre><p><strong>Key TypeRepr operations:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span> <span class="o">=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 1. Subtype checking: Is List[String] a subtype of Seq[?]?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>  <span class="c1">// true
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 2. Type equality: Is this exactly String?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// false
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 3. Pattern matching: Break it apart
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="n">tycon</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// tycon = List
</span></span></span><span class="line"><span class="cl">    <span class="o">//</span> <span class="n">args</span> <span class="k">=</span> <span class="o">[</span><span class="kt">String</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> To compare schemas, we need to ask questions like:</p>
<ul>
<li>&ldquo;Is this field an Option?&rdquo;</li>
<li>&ldquo;What&rsquo;s inside this List?&rdquo;</li>
<li>&ldquo;Does this case class have a primary constructor?&rdquo;</li>
</ul>
<p><code>TypeRepr</code> lets us answer these at compile time.</p>
<hr>
<h2 id="part-5---the-typeinspector-pattern-one-question-at-a-time">Part 5 - The TypeInspector pattern: One question at a time</h2>
<p>Now let&rsquo;s build the utilities. Each function answers ONE question about a type. Start simple:</p>
<h3 id="inspector-1-is-this-a-case-class">Inspector 1: Is this a case class?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>  <span class="c1">// Get the symbol (the &#34;name&#34; of the type)
</span></span></span><span class="line"><span class="cl">  <span class="n">s</span><span class="o">.</span><span class="n">isClassDef</span> <span class="o">&amp;&amp;</span> <span class="n">s</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">Case</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>typeSymbol</code> - Every type has a symbol (like a unique ID)</li>
<li><code>isClassDef</code> - Is it a class? (not a trait, object, etc.)</li>
<li><code>flags.is(Flags.Case)</code> - Does it have the <code>case</code> modifier?</li>
</ul>
<p><strong>Try it mentally:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>  <span class="c1">// ✅ isClassDef=true, has Case flag
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">UserTrait</span>            <span class="c1">// ❌ isClassDef=false
</span></span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">UserObject</span>          <span class="o">//</span> <span class="o">❌</span> <span class="n">not</span> <span class="n">a</span> <span class="n">class</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-2-what-are-the-type-arguments">Inspector 2: What are the type arguments?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span> <span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">args</span>  <span class="c1">// Extract the arguments
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">_</span>                    <span class="k">=&gt;</span> <span class="nc">Nil</span>   <span class="o">//</span> <span class="nc">No</span> <span class="n">arguments</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>AppliedType</code> - Pattern for generic types like <code>List[String]</code> or <code>Map[String, Int]</code></li>
<li><code>args</code> - The type parameters: <code>[String]</code> or <code>[String, Int]</code></li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>        <span class="c1">// AppliedType(List, [String])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>    <span class="c1">// AppliedType(Map, [String, Int])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String, Int]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>              <span class="c1">// Not an AppliedType
</span></span></span><span class="line"><span class="cl">                    <span class="o">//</span> <span class="n">appliedArgs</span> <span class="n">returns</span> <span class="nc">Nil</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-3-is-this-an-option">Inspector 3: Is this an Option?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>  <span class="c1">// Is it a subtype of Option[Something]?
</span></span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span>           <span class="c1">// Yes! Extract the Something
</span></span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span>                             <span class="o">//</span> <span class="nc">Nope</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>&lt;:&lt;</code> - Subtype check: &ldquo;Is this type compatible with Option[?]?&rdquo;</li>
<li><code>Option[?]</code> - The <code>?</code> means &ldquo;Option of anything&rdquo;</li>
<li><code>appliedArgs(t).headOption</code> - Get the first (and only) type argument</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// ✅ Is subtype of Option[?]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// headOption returns Some(String)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Some</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>       <span class="c1">// ✅ Some is subtype of Option
</span></span></span><span class="line"><span class="cl">                <span class="c1">// Returns Some(Int)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>          <span class="c1">// ❌ Not an Option
</span></span></span><span class="line"><span class="cl">                <span class="o">//</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> Field-level optionality. When we see <code>age: Option[Int]</code>, we need to know:</p>
<ol>
<li>The field itself is optional</li>
<li>The underlying type is <code>Int</code></li>
</ol>
<h3 id="inspector-4-is-this-a-sequence">Inspector 4: Is this a sequence?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">isSeqLike</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Vector</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">isSeqLike</span> <span class="n">then</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span> <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s ANY kind of sequence type</li>
<li>If yes, extract what&rsquo;s inside (the element type)</li>
</ul>
<p><strong>Why check multiple types?</strong> Because Scala has many collection types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>    <span class="c1">// ✅ Is List[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Seq</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>        <span class="c1">// ✅ Is Seq[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Vector</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="c1">// ✅ Is Vector[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span>     <span class="c1">// ✅ Is Array[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>     <span class="o">//</span> <span class="o">✅</span> <span class="nc">Is</span> <span class="nc">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p>All these should become <code>SequenceShape(element)</code>.</p>
<h3 id="inspector-5-is-this-a-map">Inspector 5: Is this a Map?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[(</span><span class="kt">TypeRepr</span>, <span class="kt">TypeRepr</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Map</span><span class="o">[</span><span class="kt">?</span>, <span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">k</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">v</span> <span class="kt">::</span> <span class="kt">Nil</span> <span class="o">=&gt;</span> <span class="nc">Some</span><span class="o">((</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">))</span>  <span class="c1">// Got key and value
</span></span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Map requires two type args: </span><span class="si">${</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s a <code>Map[Something, Something]</code></li>
<li>Extract BOTH type arguments: key and value</li>
<li>If Map has wrong number of args, abort (compiler error)</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>  <span class="c1">// ✅ Returns Some((String, Int))
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Long</span>, <span class="kt">User</span><span class="o">]</span>   <span class="c1">// ✅ Returns Some((Long, User))
</span></span></span><span class="line"><span class="cl"><span class="nc">String</span>            <span class="o">//</span> <span class="o">❌</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-6-can-this-be-a-map-key">Inspector 6: Can this be a Map key?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isAtomicKey</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Short</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Boolean</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why this matters:</strong> Maps in Spark StructType only support primitive keys. You can&rsquo;t have <code>Map[User, String]</code> - User isn&rsquo;t a primitive.</p>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">User</span><span class="o">]</span>  <span class="c1">// ✅ String is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Data</span><span class="o">]</span>     <span class="c1">// ✅ Int is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">User</span>, <span class="kt">String</span><span class="o">]</span>  <span class="o">//</span> <span class="o">❌</span> <span class="nc">User</span> <span class="n">is</span> <span class="nc">NOT</span> <span class="n">atomic</span> <span class="o">-</span> <span class="n">compile</span> <span class="n">error</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="the-pattern-recap">The pattern recap:</h3>
<p>Each inspector has ONE job:</p>
<ol>
<li><strong>isCaseClass</strong> - &ldquo;Is this a case class?&rdquo;</li>
<li><strong>appliedArgs</strong> - &ldquo;What are the generic type arguments?&rdquo;</li>
<li><strong>optionArg</strong> - &ldquo;Is this an Option, and what&rsquo;s inside?&rdquo;</li>
<li><strong>seqArg</strong> - &ldquo;Is this a sequence, and what&rsquo;s the element type?&rdquo;</li>
<li><strong>mapArgs</strong> - &ldquo;Is this a Map, and what are key/value types?&rdquo;</li>
<li><strong>isAtomicKey</strong> - &ldquo;Can this be a Map key?&rdquo;</li>
</ol>
<p>Compose these to answer complex questions. That&rsquo;s the TypeInspector pattern.</p>
<hr>
<h2 id="part-6---building-shapes-recursively-the-interpreter-pattern">Part 6 - Building shapes recursively: The interpreter pattern</h2>
<p>Now we use those inspectors to build a <code>TypeShape</code> - our internal representation of a type&rsquo;s structure.</p>
<p>This is recursive. For <code>List[Option[User]]</code>, we build:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">SequenceShape(
</span></span><span class="line"><span class="cl">  OptionalShape(
</span></span><span class="line"><span class="cl">    StructShape([field1, field2, ...])
</span></span><span class="line"><span class="cl">  )
</span></span><span class="line"><span class="cl">)</span></span></code></pre></td></tr></table>
</div>
</div><p>Let me show you the algorithm step by step.</p>
<h3 id="the-shape-building-algorithm">The shape building algorithm:</h3>
<pre class="mermaid">flowchart TD
    T[&#34;typeShapeOf(t)&#34;] --&gt; O{&#34;Option?&#34;}
    O -- &#34;Yes&#34; --&gt; OI[&#34;inner = optionArg(t)&#34;] --&gt; R1[&#34;typeShapeOf(inner)&#34;]
    O -- &#34;No&#34; --&gt; S{&#34;Sequence?&#34;}
    S -- &#34;Yes&#34; --&gt; SE[&#34;elem = seqArg(t)&#34;] --&gt; R2[&#34;SequenceShape(typeShapeOf(elem))&#34;]
    S -- &#34;No&#34; --&gt; M{&#34;Map?&#34;}
    M -- &#34;Yes&#34; --&gt; MKV[&#34;(k,v) = mapArgs(t)&#34;]
    MKV --&gt; K{&#34;isAtomicKey(k)?&#34;}
    K -- &#34;No&#34; --&gt; ERR[&#34;abort: unsupported key&#34;]
    K -- &#34;Yes&#34; --&gt; MV[&#34;MapShape(PrimitiveShape(k), typeShapeOf(v))&#34;]
    M -- &#34;No&#34; --&gt; CC{&#34;Case class?&#34;}
    CC -- &#34;Yes&#34; --&gt; FS[&#34;StructShape(collect FieldShape...)&#34;]
    CC -- &#34;No&#34; --&gt; P[&#34;PrimitiveShape(t.show)&#34;]
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">TypeShape</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Step 1: Is it an Option?
</span></span></span><span class="line"><span class="cl">  <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Step 2: Is it a sequence?
</span></span></span><span class="line"><span class="cl">    <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">      <span class="c1">// Step 3: Is it a Map?
</span></span></span><span class="line"><span class="cl">      <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="o">!</span><span class="n">isAtomicKey</span><span class="o">(</span><span class="n">k</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Unsupported Map key: </span><span class="si">${</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="nc">MapShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="o">),</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">v</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1">// Step 4: Is it a case class?
</span></span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// ... handle case class
</span></span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// Step 5: Fallback - primitive
</span></span></span><span class="line"><span class="cl">          <span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>The order matters!</strong> We check Option first, then sequences, then maps, then case classes, finally primitives.</p>
<h3 id="walk-through-an-example-listoptionstring">Walk through an example: <code>List[Option[String]]</code></h3>
<pre class="mermaid">flowchart TD
    A[&#34;List[Option[String]]&#34;] --&gt;|typeShapeOf| B{&#34;Option?&#34;}
    B -- &#34;No&#34; --&gt; C{&#34;Sequence?&#34;}
    C -- &#34;Yes&#34; --&gt; D[&#34;elem = Option[String]&#34;]
    D --&gt; E[&#34;typeShapeOf(Option[String])&#34;]
    E --&gt; F{&#34;Option?&#34;}
    F -- &#34;Yes&#34; --&gt; G[&#34;inner = String&#34;]
    G --&gt; H[&#34;typeShapeOf(String)&#34;]
    H --&gt; I{&#34;Case class?&#34;}
    I -- &#34;No&#34; --&gt; J[&#34;PrimitiveShape(String)&#34;]
    J --&gt; K[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><ul>
<li>
<p><strong>Step 1:</strong> Is <code>List[Option[String]]</code> an Option?</p>
<ul>
<li>No. <code>optionArg</code> returns <code>None</code>.</li>
<li>Continue to Step 2.</li>
</ul>
</li>
<li>
<p><strong>Step 2:</strong> Is it a sequence?</p>
<ul>
<li>Yes! <code>seqArg</code> returns <code>Some(Option[String])</code>.</li>
<li>Build <code>SequenceShape(typeShapeOf(Option[String]))</code>.</li>
<li><strong>Recurse</strong> on <code>Option[String]</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 1:</strong> Is <code>Option[String]</code> an Option?</p>
<ul>
<li>Yes! <code>optionArg</code> returns <code>Some(String)</code>.</li>
<li><strong>Recurse</strong> on <code>String</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 2:</strong> Is <code>String</code> an Option?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 3:</strong> Is <code>String</code> a sequence?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 4:</strong> Is <code>String</code> a Map?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 5:</strong> Is <code>String</code> a case class?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 6:</strong> Fallback to primitive.</p>
<ul>
<li>Return <code>PrimitiveShape(&quot;String&quot;)</code>.</li>
</ul>
</li>
</ul>
<p><strong>Unwind the recursion:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="o">(</span><span class="n">from</span> <span class="nc">Option</span> <span class="n">recursion</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, where did the <code>Option</code> go? It got consumed! The <code>typeShapeOf</code> for <code>Option</code> just recurses on the inner type. This is the edge case I mentioned - element-level optionality gets lost.</p>
<h3 id="handling-case-classes">Handling case classes:</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">sym</span>    <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>                      <span class="c1">// Get the class symbol
</span></span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">params</span> <span class="k">=</span> <span class="n">sym</span><span class="o">.</span><span class="n">primaryConstructor</span><span class="o">.</span><span class="n">paramSymss</span><span class="o">.</span><span class="n">flatten</span>  <span class="c1">// Get constructor params
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">fields</span> <span class="k">=</span> <span class="n">params</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">p</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">name</span>       <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span>                      <span class="c1">// Field name
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">ptpe</span>       <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">memberType</span><span class="o">(</span><span class="n">p</span><span class="o">)</span>             <span class="c1">// Field type
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">HasDefault</span><span class="o">)</span> <span class="c1">// Has default value?
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nc">StructShape</span><span class="o">(</span><span class="n">fields</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Let&rsquo;s break this down line by line:</strong></p>
<ol>
<li><strong><code>t.typeSymbol</code></strong> - Get the symbol for the case class</li>
<li><strong><code>sym.primaryConstructor</code></strong> - Case classes have a primary constructor</li>
<li><strong><code>.paramSymss.flatten</code></strong> - Get all parameters (flatten handles multiple param lists)</li>
<li><strong>For each parameter <code>p</code>:</strong>
<ul>
<li><strong><code>p.name</code></strong> - The field name: <code>&quot;id&quot;</code>, <code>&quot;email&quot;</code>, etc.</li>
<li><strong><code>t.memberType(p)</code></strong> - The field&rsquo;s type as TypeRepr</li>
<li><strong><code>p.flags.is(Flags.HasDefault)</code></strong> - Check if it has a default value like <code>age: Int = 0</code></li>
<li><strong><code>optionArg(ptpe).fold(...)</code></strong> - Check if the field type is Option[T]:
<ul>
<li>If yes: extract T, set <code>isOpt = true</code></li>
<li>If no: use the type as-is, set <code>isOpt = false</code></li>
</ul>
</li>
<li><strong><code>typeShapeOf(uT)</code></strong> - <strong>Recurse!</strong> Build the shape for the field&rsquo;s underlying type</li>
</ul>
</li>
</ol>
<h3 id="example-case-class-userid-long-email-string-age-optionint--none">Example: <code>case class User(id: Long, email: String, age: Option[Int] = None)</code></h3>
<pre class="mermaid">flowchart TD
    U[&#34;User(id: Long, email: String, age: Option[Int] = None)&#34;]
    U --&gt; F1[&#34;Field &#39;id&#39; : Long&#34;]
    F1 --&gt; S1[&#34;PrimitiveShape(Long)&#34;]
    U --&gt; F2[&#34;Field &#39;email&#39; : String&#34;]
    F2 --&gt; S2[&#34;PrimitiveShape(String)&#34;]
    U --&gt; F3[&#34;Field &#39;age&#39; : Option[Int] (default None)&#34;]
    F3 --&gt; O1{&#34;Option?&#34;}
    O1 -- &#34;Yes&#34; --&gt; A1[&#34;inner = Int&#34;]
    A1 --&gt; S3[&#34;PrimitiveShape(Int)&#34;]
    S1 --&gt; COLLECT[&#34;Collect fields&#34;]
    S2 --&gt; COLLECT
    S3 --&gt; COLLECT
    COLLECT --&gt; FINAL[&#34;StructShape([ id:PrimitiveShape(Long), email:PrimitiveShape(String), age:PrimitiveShape(Int) (optional, default) ])&#34;]
</pre><p><strong>Processing:</strong></p>
<ul>
<li>
<p><strong>Field 1: id</strong></p>
<ul>
<li><code>name</code> = &ldquo;id&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Long</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(Long)</code> = None → <code>uT = Long</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(Long)</code> = <code>PrimitiveShape(&quot;Long&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;id&quot;, PrimitiveShape(&quot;Long&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 2: email</strong></p>
<ul>
<li><code>name</code> = &ldquo;email&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>String</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(String)</code> = None → <code>uT = String</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(String)</code> = <code>PrimitiveShape(&quot;String&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;email&quot;, PrimitiveShape(&quot;String&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 3: age</strong></p>
<ul>
<li><code>name</code> = &ldquo;age&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Option[Int]</code></li>
<li><code>hasDefault</code> = <strong>true</strong> (has <code>= None</code>)</li>
<li><code>optionArg(Option[Int])</code> = Some(<code>Int</code>) → <code>uT = Int</code>, <code>isOpt = true</code></li>
<li><code>typeShapeOf(Int)</code> = <code>PrimitiveShape(&quot;Int&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;age&quot;, PrimitiveShape(&quot;Int&quot;), hasDefault=true, isOptional=true)</code></li>
</ul>
</li>
<li>
<p><strong>Final shape:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">StructShape</span><span class="o">([</span>
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">id</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Long</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">email</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">String</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">age</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Int</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">true</span>, <span class="kt">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">])</span></span></span></code></pre></td></tr></table>
</div>
</div></li>
</ul>
<h3 id="why-this-matters-1">Why this matters:</h3>
<p>Now we have a normalized representation of the case class. We can compare two <code>StructShape</code>s to see if they match, regardless of the original Scala syntax.</p>
<p>The recursive structure handles arbitrarily nested types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Order</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>           <span class="c1">// Recurses into List, then into LineItem
</span></span></span><span class="line"><span class="cl">  <span class="n">metadata</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">],</span>   <span class="c1">// Recurses into Map, then String (twice)
</span></span></span><span class="line"><span class="cl">  <span class="n">address</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">]</span>         <span class="c1">// Recurses into Option, then Address
</span></span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Each recursion builds a piece of the tree. The tree is the schema.</p>
<hr>
<h2 id="part-7---field-vs-element-optionality-a-critical-distinction">Part 7 - Field vs Element optionality: A critical distinction</h2>
<p>Here&rsquo;s something subtle that most guides miss. The code in github handles two kinds of optionality:</p>
<h3 id="field-level-optionality-option-on-the-field-itself">Field-level optionality (Option on the field itself)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                                   <span class="o">^^^^^^^^^^^</span> <span class="n">field</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The macro detects this and sets <code>isOptional = true</code> on the FieldShape:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="element-level-optionality-option-inside-a-collection">Element-level optionality (Option inside a collection)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Data</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                          <span class="o">^^^^^^^^^^^^^^</span> <span class="n">element</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This creates a nested shape: <code>SequenceShape(OptionShape(PrimitiveShape(&quot;String&quot;)))</code></p>
<p>Wait, that&rsquo;s not in the code in github! Let me check&hellip; Actually, the code in github&rsquo;s <code>typeShapeOf</code> handles this by recursing:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>So <code>List[Option[String]]</code> becomes:</p>
<ol>
<li><code>seqArg</code> detects List → extract <code>Option[String]</code></li>
<li>Recurse on <code>Option[String]</code> → <code>typeShapeOf(Option[String])</code></li>
<li><code>optionArg</code> detects Option → extract <code>String</code></li>
<li>Base case: <code>PrimitiveShape(&quot;String&quot;)</code></li>
</ol>
<p>But the final shape is <code>SequenceShape(PrimitiveShape(&quot;String&quot;))</code> - the intermediate Option is consumed!</p>
<p><strong>Why this matters for contracts</strong>: When you write:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Producer</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span></span></span></code></pre></td></tr></table>
</div>
</div><p>These should probably NOT conform under Exact. The producer allows null elements, the contract doesn&rsquo;t. But the current code in github might allow it because the Option gets unwrapped.</p>
<p>This is the kind of edge case you discover when shipping to production. The fix would be to add an <code>OptionalShape</code> wrapper in the model.
<div class="admonition caution">
  <div class="admonition-title">Optionality edge case</div>
  <div class="admonition-content">
    The element-level optionality issue (List[Option[String]] getting flattened) is real and subtle. If your schemas use nullable collections, test that the compile-time check actually catches the difference between List[String] and List[Option[String]]. The current implementation may allow mismatches here. Add explicit test cases for your nullable collection patterns before relying on them in production.
  </div>
</div></p>
<p>Visual (field vs element optionality):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/optionality.svg"
            alt="Field vs Element Optionality"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    Field optionality vs element optionality - they&#39;re not the same.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<pre class="mermaid">flowchart LR
    A[&#34;List[Option[String]]&#34;] --&gt; B{&#34;Element optionality?&#34;}
    B -- &#34;intended shape&#34; --&gt; C[&#34;SequenceShape(OptionalShape(PrimitiveShape(String)))&#34;]
    B -- &#34;current code in github\n(consumes Option)&#34; --&gt; D[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><hr>
<h2 id="part-8---compiletime--evidence-the-full-picture">Part 8 - Compile‑time <strong>&ldquo;conforms&rdquo;</strong> evidence (the full picture)</h2>
<p>Now let&rsquo;s put it all together. Here&rsquo;s the complete macro from the code in github:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Conforms.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.quoted.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">class</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">object</span> <span class="nc">Conforms</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">materialize</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">$</span><span class="o">{</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">P:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span>, <span class="kt">Boolean</span>, <span class="kt">Boolean</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">fs</span> <span class="k">=</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span>
</span></span><span class="line"><span class="cl">        <span class="n">fs</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">hasDefault</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">isOptional</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Compare fields by name (unordered, case-sensitive for now)
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">outMap</span>      <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contractMap</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">missing</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">outMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s">:</span><span class="si">$t$</span><span class="s">{if opt then &#34;</span> <span class="o">(</span><span class="n">optional</span><span class="o">)</span><span class="s">&#34; else &#34;&#34;}&#34;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">extra</span>   <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span>      <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">contractMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mism</span>    <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span> <span class="o">!=</span> <span class="n">t</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> expected </span><span class="si">$t</span><span class="s">, found </span><span class="si">${</span><span class="n">outMap</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">_1</span><span class="si">}</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">!=</span> <span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> optionality mismatch&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">okMissing</span><span class="o">,</span> <span class="n">okExtra</span><span class="o">,</span> <span class="n">okMism</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Exact</span><span class="o">]</span>    <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Backward</span><span class="o">]</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">.</span><span class="n">filterNot</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="s">&#34;(optional)&#34;</span><span class="o">)),</span> <span class="nc">Nil</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Forward</span><span class="o">]</span>  <span class="k">=&gt;</span> <span class="o">(</span><span class="nc">Nil</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span>                  <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">okMissing</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okExtra</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okMism</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="s">s&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">        |Compile‑time contract drift (policy: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span><span class="si">}</span><span class="s">).
</span></span></span><span class="line"><span class="cl"><span class="s">        |Out: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span><span class="si">}</span><span class="s"> vs Contract: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Missing: </span><span class="si">${</span><span class="n">okMissing</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Extra: </span><span class="si">${</span><span class="n">okExtra</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Mismatched: </span><span class="si">${</span><span class="n">okMism</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |&#34;&#34;&#34;</span><span class="o">.</span><span class="n">stripMargin</span>
</span></span><span class="line"><span class="cl">      <span class="n">quotes</span><span class="o">.</span><span class="n">reflect</span><span class="o">.</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="n">msg</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="k">new</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Let me break down what&rsquo;s happening:</p>
<ol>
<li>
<p><strong>Extract shapes</strong>: We summon <code>Shape[Out]</code> and <code>Shape[Contract]</code> and get their field lists. Note the <code>.valueOrAbort</code> - this forces compile-time evaluation.</p>
</li>
<li>
<p><strong>Compare</strong>: We build maps of field names → (type, optionality) and find differences.</p>
</li>
<li>
<p><strong>Apply policy</strong>:</p>
<ul>
<li>Exact: all differences are errors</li>
<li>Backward: missing optional fields are OK, extras are OK</li>
<li>Forward: extras are OK (but missing required fields are not)</li>
</ul>
</li>
<li>
<p><strong>Abort or succeed</strong>: If there are violations, we call <code>report.errorAndAbort</code> with a detailed message showing exactly what&rsquo;s wrong. Otherwise, we return the evidence.</p>
</li>
</ol>
<p>The beauty here: <strong>this runs during compilation</strong>. If schemas don&rsquo;t match, your code won&rsquo;t compile. Period.
<div class="admonition danger">
  <div class="admonition-title">Build breakage alert</div>
  <div class="admonition-content">
    Compile-time contract checks will break your build when schemas drift. That&rsquo;s the point, but coordinate with your team. When upstream adds a field, every downstream pipeline build fails until they update contracts or transforms. This forces alignment, but can block deployments if not managed. Use CI branch builds to preview impact before merging schema changes.
  </div>
</div></p>
<p>Use it like this:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ConformsSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutOk</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutMissing</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkConformance</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutOk</span>,      <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>   <span class="c1">// ✅ compiles
</span></span></span><span class="line"><span class="cl">  <span class="c1">// summon[Conforms[OutMissing, Contract, Policy.Exact]] // ❌ compile‑time error (Missing: email:String)
</span></span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span> <span class="o">//</span> <span class="o">✅</span> <span class="n">compiles</span> <span class="o">(</span><span class="n">migration</span> <span class="n">mode</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>That&rsquo;s it. The entire core. In production, you&rsquo;d want to handle the element optionality issue I mentioned, but the structure stays the same.</p>
<hr>
<h2 id="part-9---developer-ergonomics-what-using-this-actually-feels-like">Part 9 - Developer ergonomics: What using this actually feels like</h2>
<p>Let&rsquo;s talk about the practical side - what happens when you actually use this in your daily work?</p>
<h3 id="compile-times">Compile times</h3>
<p><strong>Question: Does this slow down compilation?</strong></p>
<p>In practice, not noticeably. The macro runs once per <code>summon[Conforms[...]]</code> call during compilation. For a typical pipeline with 5-10 contract checks, you&rsquo;re adding maybe 100-200ms to your build. Compare that to the hours you&rsquo;d spend debugging a production schema mismatch.</p>
<h3 id="error-messages">Error messages</h3>
<p>When schemas drift, you get this:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[error] Compile-time contract drift (policy: Policy.Exact).
</span></span><span class="line"><span class="cl">[error] Out: CustomerProducer vs Contract: CustomerContract
</span></span><span class="line"><span class="cl">[error] Extra: segment
</span></span><span class="line"><span class="cl">[error] Missing:
</span></span><span class="line"><span class="cl">[error] Mismatched:</span></span></code></pre></td></tr></table>
</div>
</div><p>Clear, actionable, and it stops your build. No cryptic macro errors, no stack traces. Just: &ldquo;Hey, you have an extra field called <code>segment</code>.&rdquo;</p>
<h3 id="integration-patterns">Integration patterns</h3>
<p><strong>With Spark:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Your Spark job
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">df</span> <span class="k">=</span> <span class="n">spark</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">typed</span> <span class="k">=</span> <span class="n">df</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">]</span>  <span class="c1">// Spark encoder
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Compile-time check before transformation
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerProducer</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">output</span> <span class="k">=</span> <span class="n">typed</span><span class="o">.</span><span class="n">transform</span><span class="o">(</span><span class="n">dropSegment</span><span class="o">).</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>With schema registries:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Avro schema in registry → case class → contract check
</span></span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerAvro</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">amt</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerAvro</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span> <span class="nc">Build</span> <span class="n">fails</span> <span class="k">if</span> <span class="nc">Avro</span> <span class="n">schema</span> <span class="n">drift</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>CI/CD integration:</strong>
Just run <code>sbt compile</code>. If contracts drift, the build fails. No special tooling needed. Works with Jenkins, GitHub Actions, GitLab CI - anything that runs sbt.</p>
<h3 id="onboarding">Onboarding</h3>
<p><strong>New team members ask</strong>: &ldquo;Why won&rsquo;t this compile?&rdquo;</p>
<p><strong>Answer</strong>: &ldquo;Check the error. You&rsquo;re trying to use <code>CustomerProducer</code> but the contract expects <code>CustomerContract</code>. Either adjust your transform or update the contract.&rdquo;</p>
<p>That&rsquo;s it. The compiler guides them. After one or two cases, they get it.</p>
<h3 id="when-to-use-vs-skip">When to use vs skip</h3>
<p><strong>Use compile-time contracts when</strong>:</p>
<ul>
<li>Multiple teams work on the same pipeline codebase</li>
<li>Schema changes break production regularly</li>
<li>You want fast feedback (compile vs deploy-and-test)</li>
</ul>
<p><strong>Skip when</strong>:</p>
<ul>
<li>One-off scripts or exploratory work</li>
<li>External APIs you can&rsquo;t control (use runtime validation)</li>
<li>Schema is genuinely dynamic (JSON with unknown keys)</li>
</ul>
<hr>
<h2 id="part-10---nested-types-maps-lists-and-deep-structures">Part 10 - Nested types: Maps, Lists, and deep structures</h2>
<p>The code in github handles nested types beautifully. Look at this example from <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Deep / Nested structures
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">LineItem</span><span class="o">(</span><span class="n">sku</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">qty</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">attrs</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Address</span><span class="o">(</span><span class="n">street</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">zip</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderOut</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderContract</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>        <span class="c1">// List vs Seq - both are sequences
</span></span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>      <span class="c1">// Nested case class
</span></span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Nil</span>       <span class="c1">// Set vs Seq, has default
</span></span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">evDeepOk</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">OrderOut</span>, <span class="kt">OrderContract</span>, <span class="kt">SchemaPolicy.Backward.</span><span class="k">type</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening here:</strong></p>
<ol>
<li>
<p><strong>Nested case classes</strong> - <code>Address</code> inside <code>OrderOut</code>. The macro recurses: when it sees <code>Option[Address]</code>, it unwraps to <code>Address</code>, then checks if <code>Address</code> is a case class, and recurses again to get its fields.</p>
</li>
<li>
<p><strong>Collections</strong> - <code>List[LineItem]</code> vs <code>Seq[LineItem]</code>. Both match because <code>seqArg</code> treats them equivalently. The macro sees &ldquo;sequence of LineItem&rdquo; and recurses on <code>LineItem</code>.</p>
</li>
<li>
<p><strong>Maps</strong> - <code>Map[String, String]</code> in <code>attrs</code>. The macro checks key type (must be atomic), then recurses on value type.</p>
</li>
<li>
<p><strong>Default values</strong> - <code>tags: Seq[String] = Nil</code> has a default. Under Backward policy, if <code>OrderOut</code> is missing <code>tags</code>, it&rsquo;s OK because the contract has a default.</p>
</li>
</ol>
<p>This is why the TypeInspector pattern matters - it handles arbitrary nesting without special cases.</p>
<hr>
<h2 id="part-10---policy-modifiers-in-two-minutes-ordered-ci-byposition">Part 10 - Policy modifiers in two minutes (Ordered, CI, By‑Position)</h2>
<p>Sometimes you need stricter comparison. Or different rules. Here are three quick policy extensions:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/PolicyMods.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">PolicyMod</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">ExactOrdered</span>   <span class="kt">extends</span> <span class="kt">PolicyMod</span> <span class="c1">// names + order + types must match
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactCI</span>        <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// case‑insensitive field names
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// ignore names; types by index
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactOrdered</span>    <span class="k">extends</span> <span class="nc">ExactOrdered</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactCI</span>         <span class="k">extends</span> <span class="nc">ExactCI</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">ExactByPosition</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">ConformsMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">import</span> <span class="kt">scala.quoted.*</span>
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span> <span class="k">&lt;:</span> <span class="kt">PolicyMod</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">M:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">fields</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">))</span> <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mismatches</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">M</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">n</span> <span class="k">=</span> <span class="o">(</span><span class="n">s</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">toLowerCase</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">om</span> <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">cm</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">keySet</span> <span class="n">union</span> <span class="n">om</span><span class="o">.</span><span class="n">keySet</span><span class="o">).</span><span class="n">toList</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span> <span class="n">k</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">          <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">),</span> <span class="n">om</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">))</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="n">t1</span><span class="o">),</span> <span class="nc">Some</span><span class="o">(</span><span class="n">t2</span><span class="o">))</span> <span class="k">if</span> <span class="n">t1</span> <span class="o">!=</span> <span class="n">t2</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;</span><span class="si">$k</span><span class="s"> expected </span><span class="si">$t1</span><span class="s">, found </span><span class="si">$t2</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">),</span> <span class="nc">None</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;missing </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">None</span><span class="o">,</span> <span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">))</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;extra </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="k">_</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="k">_</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">!=</span> <span class="n">tC</span> <span class="k">=&gt;</span> <span class="s">s&#34;@</span><span class="si">$i</span><span class="s"> expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactOrdered</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="n">nO</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="n">nC</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">nameOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">nO</span> <span class="o">==</span> <span class="n">nC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(name) expected </span><span class="si">$nC</span><span class="s">, found </span><span class="si">$nO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">typeOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">==</span> <span class="n">tC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(type) expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="n">nameOk</span> <span class="o">++</span> <span class="n">typeOk</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;Unknown policy mod&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">mismatches</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Modifier drift: </span><span class="si">${</span><span class="n">mismatches</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="o">()</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Try these:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">com.example.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">A</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">B</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span>, <span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="c1">// ✅ same types by index
</span></span></span><span class="line"><span class="cl"><span class="c1">// ConformsMod[A, B, PolicyMod.ExactOrdered] // ❌ order matters → name mismatch
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIa</span><span class="o">(</span><span class="nc">ID</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIb</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">CIa</span>, <span class="kt">CIb</span>, <span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="o">//</span> <span class="o">✅</span> <span class="k">case</span><span class="err">‑</span><span class="n">insensitive</span> <span class="n">names</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Why these three? Because real systems need them:</p>
<ul>
<li><strong>ExactOrdered</strong>: For formats where position matters (some CSV parsers, protobuf with field numbers)</li>
<li><strong>ExactCI</strong>: Because someone always uses <code>UserId</code> when the contract says <code>userId</code></li>
<li><strong>ExactByPosition</strong>: For legacy systems that ignore field names entirely</li>
</ul>
<hr>
<h2 id="part-11---phantom-types-building-impossible-to-misuse-apis">Part 11 - Phantom types: Building impossible-to-misuse APIs</h2>
<p>Here&rsquo;s where things get interesting. The code, also uses phantom types, type-indexing / typestate builder patterns to build a pipeline builder that&rsquo;s <strong>impossible to misuse</strong>.
What are phantom types? They&rsquo;re type parameters that don&rsquo;t appear in the class body but control what you can do with an instance. Think of them as compile-time state machines.</p>
<pre class="mermaid">stateDiagram-v2
    [*] --&gt; Empty
    Empty --&gt; WithSource: addSource
    WithSource --&gt; WithTransform: transformAs[Next]
    WithTransform --&gt; Complete: addSink[Contract,P] + conforms
    Complete --&gt; [*]: build

    note right of WithTransform
      Evidence constraints:
      - S &lt;: WithSource to transform
      - S &lt;: WithTransform to sink
      - S = Complete to build
    end note
</pre><p>From the codebase <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Phantom type states
</span></span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Empty</span>         <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithSource</span>    <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithTransform</span> <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Complete</span>      <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">S</span> <span class="k">&lt;:</span> <span class="kt">BuilderState</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="nc">private</span> <span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">steps</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">PipelineStep</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add source when Empty
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSource</span><span class="o">[</span><span class="kt">C</span><span class="o">](</span><span class="n">src</span><span class="k">:</span> <span class="kt">TypedSource</span><span class="o">[</span><span class="kt">C</span><span class="o">])(</span><span class="n">using</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">])</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only transform after adding source
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">f</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span><span class="o">)(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span><span class="o">,</span>  <span class="c1">// This constraint enforces order!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">Next</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add sink after transform
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSink</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">](</span><span class="n">sink</span><span class="k">:</span> <span class="kt">TypedSink</span><span class="o">[</span><span class="kt">R</span><span class="o">])(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev0</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithTransform</span><span class="o">,</span>  <span class="c1">// Must have transform
</span></span></span><span class="line"><span class="cl">    <span class="n">ev1</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">CurContract</span>, <span class="kt">R</span>, <span class="kt">P</span><span class="o">],</span>  <span class="c1">// Compile-time contract check!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">R</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only build when Complete
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">build</span><span class="o">(</span><span class="n">using</span> <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="o">=</span><span class="k">:</span><span class="o">=</span> <span class="nc">Complete</span><span class="o">)</span><span class="k">:</span> <span class="kt">SparkSession</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="o">(</span><span class="n">spark</span><span class="k">:</span> <span class="kt">SparkSession</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PipelineBuilder</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">apply</span><span class="o">[</span><span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="nc">Nil</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="how-this-works">How this works:</h3>
<ol>
<li>
<p><strong>State transitions as types</strong> - Each method returns a new type parameter <code>S</code>. The compiler tracks which state you&rsquo;re in.</p>
</li>
<li>
<p><strong>Evidence constraints</strong> - <code>ev: S &lt;:&lt; WithSource</code> means &ldquo;S must be a subtype of WithSource&rdquo;. If it&rsquo;s not, this method doesn&rsquo;t exist.</p>
</li>
<li>
<p><strong>Compile-time state machine</strong> - You literally cannot call methods in the wrong order:</p>
</li>
</ol>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// ✅ This compiles
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;good&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">src</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// ❌ This doesn&#39;t compile - can&#39;t transform before adding source
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;bad&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">ERROR</span><span class="k">:</span> <span class="kt">No</span> <span class="kt">implicit</span> <span class="kt">evidence</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span></span></span></code></pre></td></tr></table>
</div>
</div><ol start="4">
<li><strong>Contract checking embedded</strong> - Notice <code>ev1: SchemaConforms[CurContract, R, P]</code> in <code>addSink</code>? That&rsquo;s where the compile-time contract check happens. The pipeline literally won&rsquo;t build if schemas drift.</li>
</ol>
<p>This pattern is called the <strong>Phantom Type Builder Pattern</strong>. For more details, see:</p>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala (Rhetorical Musings)</a>
</li>
</ul>
<p><strong>Why this matters</strong>: In production data pipelines, the order of operations matters. Read → Transform → Validate → Write. With phantom types, the compiler enforces this order. You can&rsquo;t accidentally write before transforming. You can&rsquo;t transform without a source.</p>
<p>This is the difference between &ldquo;here&rsquo;s how macros work&rdquo; and &ldquo;here&rsquo;s how to build production contract systems.&rdquo;</p>
<hr>
<h2 id="part-12---schema-evolution-and-versioning-handling-change-over-time">Part 12 - Schema evolution and versioning: Handling change over time</h2>
<p>Here&rsquo;s the reality: schemas evolve. Fields get added, deprecated, renamed. Teams roll out changes incrementally. The question isn&rsquo;t whether schemas will change - it&rsquo;s how you manage those changes without breaking production.</p>
<h3 id="the-schema-evolution-problem">The schema evolution problem</h3>
<p><strong>Scenario</strong>: You have <code>CustomerV1</code> running in production. Marketing wants to add a <code>segment</code> field for targeting. But you have 10 pipelines reading <code>CustomerV1</code>. How do you evolve without breaking everything?</p>
<p><strong>Wrong approach</strong>: Add <code>segment</code> to <code>CustomerV1</code>, deploy, hope for the best. Result: Some pipelines break because they don&rsquo;t handle the new field.</p>
<p><strong>Right approach</strong>: Version your contracts and use policies to manage transitions.</p>
<h3 id="versioning-strategy">Versioning strategy</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala - Current production
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala - New version with segment
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Notice <code>segment</code> is:</p>
<ol>
<li><strong>Optional</strong> (<code>Option[String]</code>)</li>
<li><strong>Has a default</strong> (<code>= None</code>)</li>
</ol>
<p>This makes it backward-compatible - old code can work with new data.</p>
<h3 id="migration-phases">Migration phases</h3>
<p><strong>Phase 1: Add the field (Backward policy)</strong></p>
<p>Deploy new producers writing <code>CustomerV2</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">producer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;write-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">rawData</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addSegment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">Policy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Backward allows optional fields
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Old consumers still read <code>CustomerV1</code>. They ignore the <code>segment</code> field. No breakage.</p>
<p><strong>Phase 2: Migrate consumers</strong></p>
<p>Update each consumer one by one:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">consumer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;read-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">source</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">EnrichedCustomer</span><span class="o">](</span><span class="n">useSegment</span><span class="o">)</span>  <span class="c1">// Now uses segment field
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">EnrichedCustomer</span>, <span class="kt">Policy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The compile-time contract ensures: &ldquo;Does this consumer handle the new schema?&rdquo; If you forget to update the case class, it won&rsquo;t compile.</p>
<p><strong>Phase 3: Deprecate V1</strong></p>
<p>Once all pipelines use <code>CustomerV2</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Mark V1 as deprecated
</span></span></span><span class="line"><span class="cl"><span class="nd">@deprecated</span><span class="o">(</span><span class="s">&#34;Use CustomerV2&#34;</span><span class="o">,</span> <span class="s">&#34;2025-10-01&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The compiler warns any remaining V1 usage. After a grace period, delete V1 entirely.</p>
<h3 id="handling-breaking-changes">Handling breaking changes</h3>
<p>What if you need to rename a field? Say, <code>email</code> → <code>emailAddress</code>?</p>
<p><strong>Non-breaking approach</strong>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add the new field, keep the old
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>              <span class="c1">// Old field
</span></span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New field
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, that&rsquo;s duplication. Better:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add alias constructor
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New canonical name
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">CustomerV3</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">fromV2</span><span class="o">(</span><span class="kt">v2:</span> <span class="kt">CustomerV2</span><span class="o">)</span><span class="kt">:</span> <span class="kt">CustomerV3</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">CustomerV3</span><span class="o">(</span><span class="n">v2</span><span class="o">.</span><span class="n">id</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">email</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">segment</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Step 2</strong>: Migrate producers to write <code>emailAddress</code>
<strong>Step 3</strong>: Migrate consumers to read <code>emailAddress</code>
<strong>Step 4</strong>: Remove the <code>fromV2</code> constructor</p>
<p>At each step, compile-time contracts verify: &ldquo;Does this transformation produce the expected schema?&rdquo;</p>
<h3 id="coexistence-during-migration">Coexistence during migration</h3>
<p>During migration, you have both V2 and V3 running. How to handle?</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V3</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Transformation handles both
</span></span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="n">normalize</span><span class="o">(</span><span class="n">schema</span><span class="k">:</span> <span class="kt">CustomerSchema</span><span class="o">)</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=</span> <span class="n">schema</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">V2</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">V3</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">v3</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=&gt;</span> <span class="n">v3</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The contract checks both paths:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V2</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span>  <span class="c1">// V2 → V3 allowed
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V3</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>     <span class="o">//</span> <span class="n">V3</span> <span class="o">→</span> <span class="n">V3</span> <span class="n">exact</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="real-world-migration-backward--exact">Real-world migration: Backward → Exact</h3>
<p>Let&rsquo;s look at a complete migration scenario from the code in github. This is exactly how you&rsquo;d roll out schema changes in production.</p>
<p>From <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="cm">/** Sink contract: the target schema we promise to write. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerContract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Producer: imagine an upstream source adds an extra field `segment`. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerProducer</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Declared &#34;Next&#34; schema after a transform (dropping `segment`). */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerNext</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Step 1: Migration phase - use Backward policy
</span></span></span><span class="line"><span class="cl"><span class="c1">// Producer has extra field, but contract allows it during migration
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">srcB</span> <span class="k">=</span> <span class="nc">TypedSource</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">](</span><span class="s">&#34;csv&#34;</span><span class="o">,</span> <span class="n">inPath</span><span class="o">,</span> <span class="nc">Map</span><span class="o">(</span><span class="s">&#34;header&#34;</span> <span class="o">-&gt;</span> <span class="s">&#34;true&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">sinkB</span> <span class="k">=</span> <span class="nc">TypedSink</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="n">tmpOutB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">dropExtras</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span> <span class="k">_</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="n">$</span><span class="s">&#34;id&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;email&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;age&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planB</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;CSV -&gt; Parquet B: transformAs[CustomerNext], Exact&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">srcB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerNext</span><span class="o">](</span><span class="s">&#34;drop segment&#34;</span><span class="o">)(</span><span class="n">dropExtras</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sinkB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="whats-happening-here">What&rsquo;s happening here:</h3>
<p><strong>Phase 1: Producer adds field</strong></p>
<ul>
<li>Upstream adds <code>segment</code> field to <code>CustomerProducer</code></li>
<li>Our contract is still <code>CustomerContract</code> (no <code>segment</code>)</li>
<li>We use <code>transformAs[CustomerNext]</code> to explicitly drop the extra field</li>
<li>Then check <code>CustomerNext</code> conforms to <code>CustomerContract</code> under <strong>Exact</strong></li>
</ul>
<p><strong>Why this works:</strong></p>
<ol>
<li>The transform explicitly declares the output schema (<code>CustomerNext</code>)</li>
<li>The compiler checks: Does <code>CustomerNext</code> match <code>CustomerContract</code>? Yes (both have id, email, age)</li>
<li>If we later change <code>CustomerNext</code> by accident, compilation fails</li>
</ol>
<p><strong>Phase 2: Stabilization</strong>
Once the migration is done:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Later: Everyone uses CustomerContract, no transform needed
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planStable</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;stable&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">contractSource</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">noTransform</span>  <span class="c1">// Direct pass-through
</span></span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This is real-world stuff. The code shows you exactly how to migrate schemas safely.</p>
<pre class="mermaid">flowchart TD
    P1[&#34;Producer adds field &#39;segment&#39;&#34;] --&gt; T1[&#34;transformAs[CustomerNext]\n(drop extras)&#34;]
    T1 --&gt; V1[&#34;summon Conforms[CustomerNext,CustomerContract,Policy.Exact]&#34;]
    V1 --&gt; W1[&#34;write sink&#34;]

    subgraph Phase 1 [&#34;Migration window&#34;]
      direction TB
      P1 --&gt; T1 --&gt; V1 --&gt; W1
    end

    W1 --&gt; P2[&#34;Stabilize: sources match contract&#34;]
    P2 --&gt; V2[&#34;Conforms[CustomerContract,CustomerContract,Policy.Exact]&#34;]
    V2 --&gt; W2[&#34;noTransform + write&#34;]
</pre><hr>
<h2 id="part-13---testing-compiletime-for-real-copypaste-tests">Part 13 - Testing compile‑time, for real (copy/paste tests)</h2>
<p>How do you test that code fails to compile? With <code>assertDoesNotCompile</code>:</p>
<pre class="mermaid">sequenceDiagram
    participant Dev
    participant CI as &#34;GitHub Actions&#34;
    participant SBT as &#34;sbt testOnly *CompileTimeSpec&#34;
    Dev-&gt;&gt;CI: open PR
    CI-&gt;&gt;SBT: run compile-fail suite
    SBT-&gt;&gt;SBT: assertCompiles / assertDoesNotCompile
    alt all pass
        SBT--&gt;&gt;CI: success
        CI--&gt;&gt;Dev: green check
    else drift detected
        SBT--&gt;&gt;CI: failure with errorAndAbort msg
        CI--&gt;&gt;Dev: red X + compiler diff
    end
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/CompileTimeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">org.scalatest.funsuite.AnyFunSuite</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CompileTimeSpec</span> <span class="k">extends</span> <span class="nc">AnyFunSuite</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">test</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Exact</span> <span class="kt">should</span> <span class="kt">fail</span> <span class="kt">when</span> <span class="kt">a</span> <span class="kt">required</span> <span class="kt">field</span> <span class="kt">is</span> <span class="kt">missing</span><span class="err">&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="kt">assertDoesNotCompile</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span><span class="err">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Backward should pass for the same mismatch&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OutMissing, Contract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;By‑Position should accept re‑ordered names with same types&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactByPosition]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Ordered should reject the same re‑ordering&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactOrdered]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Pipeline builder enforces correct order&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">      PipelineBuilder[Contract](&#34;bad&#34;)
</span></span></span><span class="line"><span class="cl"><span class="s">        .transformAs[Next](identity)  // Can&#39;t transform without source
</span></span></span><span class="line"><span class="cl"><span class="s">    &#34;&#34;&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Nested types conform correctly&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OrderOut, OrderContract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>You can wire this into CI:</p>
<div class="highlight highlight-yaml" data-lang="yaml"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/compile-fail.yml (concept)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">compile-fail</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">pull_request]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cf</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-java@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">distribution</span><span class="p">:</span><span class="w"> </span><span class="nt">temurin, java-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;17&#39;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Compile‑fail suite</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">sbt -v &#34;testOnly *CompileTimeSpec&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This is gold. Your CI literally proves that broken schemas can&rsquo;t be deployed.</p>
<hr>
<p><del>## Part 14 - Scala 2: how to do the same (and what&rsquo;s different)</del></p>
<p><del>Scala 2 uses def macros with a <code>Context</code> (blackbox/whitebox). The idea is identical, but the implementation uses <code>c.universe</code> trees instead of <code>Expr</code>/quotes.</del></p>
<p><del>Key differences:</del>
<del>- Scala 3: <code>inline</code> + <code>${ ... }</code> splices; Scala 2: <code>macro def</code> with a <code>Context</code></del>
<del>- Error reporting: <code>report.errorAndAbort</code> (Scala 3) vs <code>c.abort</code> (Scala 2)</del>
<del>- Trees: <code>Expr[T]</code> (Scala 3) vs raw <code>Tree</code> (Scala 2)</del>
<del>- Type inspection: <code>TypeRepr</code> (Scala 3) vs <code>Type</code> (Scala 2)</del></p>
<p><del>Scala 3 macros are safer. The quoted API prevents many common macro bugs. But Scala 2 macros work fine if you&rsquo;re careful.</del></p>
<p><del>For derivation, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works across both versions.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ Removed: Part 14 - Scala 2</div>
  <div class="admonition-content">
    Jobless/duplicate for concision objective.
  </div>
</div>
<hr>
<h2 id="part-15---integration-with-spark-runtime-defense-in-depth">Part 15 - Integration with Spark: Runtime defense-in-depth</h2>
<p>The code in github also shows how to mirror compile-time policies with Spark&rsquo;s built-in structural comparators for runtime validation.</p>
<p>From <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Derive Spark StructType from case class at compile time
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">struct</span><span class="k">:</span> <span class="kt">StructType</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">SparkSchema</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">derived</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="k">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Macro that converts case class → StructType
</span></span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]]</span> <span class="k">=</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Runtime policy mapping to Spark comparators
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyRuntime</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsIgnoreCaseAndNullability</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactByPosition.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurally</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="n">ignoreNullability</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactOrdered.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurallyByName</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="k">_</span> <span class="o">==</span> <span class="k">_</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Two-layer validation strategy</strong>:</p>
<ol>
<li><strong>Compile-time</strong> (macros) - Catches schema drift before deployment</li>
<li><strong>Runtime</strong> (Spark comparators) - Defensive check for external data sources</li>
</ol>
<pre class="mermaid">flowchart TD
    F[&#34;found: StructType&#34;] --- E[&#34;expected: StructType&#34;]
    F --&gt; EX[&#34;Policy.Exact&#34;] --&gt; C1[&#34;equalsIgnoreCaseAndNullability(found,expected)&#34;]
    F --&gt; EP[&#34;Policy.ExactByPosition&#34;] --&gt; C2[&#34;equalsStructurally(found,expected,true)&#34;]
    F --&gt; EO[&#34;Policy.ExactOrdered&#34;] --&gt; C3[&#34;equalsStructurallyByName(found,expected,resolver)&#34;]
</pre><p>For more on Spark&rsquo;s structural comparators, see the <a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">DataType API docs</a>
.</p>
<hr>
<h2 id="part-16---a-tiny-migration-playbook-you-can-adopt-tomorrow">Part 16 - A tiny migration playbook you can adopt tomorrow</h2>
<p>Here&rsquo;s how I use these policies in practice:</p>
<ul>
<li><strong>Critical paths → Exact</strong>. No surprises. If schemas don&rsquo;t match exactly, builds fail.</li>
<li><strong>Producer adding optional fields → Backward</strong> during rollout. Allows new optional fields on the producer side.</li>
<li><strong>Consumer that can ignore extras → Forward</strong> during rollout. Lets the consumer tolerate extra fields.</li>
<li><strong>After stabilization → tighten back to Exact</strong>.</li>
</ul>
<p>And remember:</p>
<ul>
<li>Keep transformations pure (no side effects inside map/filter)</li>
<li>Place I/O at edges (read once, write once)</li>
<li>Make side-effects idempotent (so retries don&rsquo;t duplicate writes)</li>
</ul>
<blockquote>
  <p>Compile-time prevents drift; runtime manages reality. Tests catch behavior, not shapes.</p>

</blockquote><hr>
<h2 id="part-17---production-patterns-what-i-learned-shipping-this">Part 17 - Production patterns: What I learned shipping this</h2>
<h3 id="pattern-1-organize-contracts-by-version">Pattern 1: Organize contracts by version</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// pipelines/CustomerMigration.scala
</span></span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">contracts._</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">migration</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;v1-to-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV1</span><span class="o">](</span><span class="n">sourceV1</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addNameField</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sinkV2</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-2-use-companion-objects-for-schema-caching">Pattern 2: Use companion objects for schema caching</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">User</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>  <span class="o">//</span> <span class="nc">Compute</span> <span class="n">once</span><span class="o">,</span> <span class="n">reuse</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-3-document-policy-choices-inline">Pattern 3: Document policy choices inline</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Good: Explicit reasoning
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Allow optional fields during Q2 migration
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Bad: No context
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Full</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">Why</span> <span class="nc">Full</span><span class="o">?</span> <span class="nc">When</span> <span class="n">will</span> <span class="n">we</span> <span class="n">tighten</span><span class="o">?</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-4-test-your-compile-fail-cases">Pattern 4: Test your compile-fail cases</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Keep these in version control as documentation
</span></span></span><span class="line"><span class="cl"><span class="n">test</span><span class="o">(</span><span class="s">&#34;CustomerV1 should not conform to CustomerV2 under Exact&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;summon[Conforms[CustomerV1, CustomerV2, Policy.Exact]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><hr>
<h2 id="appendix---full-demo-project-skeleton-copy--run">Appendix - Full demo project skeleton (copy &amp; run)</h2>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">├── build.sbt
</span></span><span class="line"><span class="cl">└── src
</span></span><span class="line"><span class="cl">    ├── main
</span></span><span class="line"><span class="cl">    │   └── scala
</span></span><span class="line"><span class="cl">    │       └── com/example/
</span></span><span class="line"><span class="cl">    │           ├── IntroMacro.scala
</span></span><span class="line"><span class="cl">    │           ├── Shape.scala
</span></span><span class="line"><span class="cl">    │           ├── Policy.scala
</span></span><span class="line"><span class="cl">    │           ├── Conforms.scala
</span></span><span class="line"><span class="cl">    │           └── SparkCore.scala
</span></span><span class="line"><span class="cl">    └── test
</span></span><span class="line"><span class="cl">        └── scala
</span></span><span class="line"><span class="cl">            └── com/example/
</span></span><span class="line"><span class="cl">                ├── ShapeSpec.scala
</span></span><span class="line"><span class="cl">                ├── ConformsSpec.scala
</span></span><span class="line"><span class="cl">                └── CompileTimeSpec.scala</span></span></code></pre></td></tr></table>
</div>
</div><p><strong>build.sbt (Scala 3)</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">scalaVersion</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;3.3.3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organization</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;com.example&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.apache.spark&#34;</span> <span class="o">%%</span> <span class="s">&#34;spark-sql&#34;</span> <span class="o">%</span> <span class="s">&#34;3.5.1&#34;</span> <span class="o">%</span> <span class="s">&#34;provided&#34;</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.scalatest&#34;</span> <span class="o">%%</span> <span class="s">&#34;scalatest&#34;</span> <span class="o">%</span> <span class="s">&#34;3.2.17&#34;</span> <span class="o">%</span> <span class="nc">Test</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Run it:</p>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><blockquote>
  <p>Tip: For Scala 2, add a sibling module with scala‑reflect and copy the S2 file from above.</p>

</blockquote><hr>
<h2 id="references">References</h2>
<h3 id="scala-3-macros--metaprogramming">Scala 3 Macros &amp; Metaprogramming</h3>
<ul>
<li><a href="https://rockthejvm.com/courses/scala-macros-and-metaprogramming" rel="noopener external" target="_blank">Rock the JVM - Scala Macros and Metaprogramming (course)</a>
</li>
<li><a href="https://rockthejvm.com/articles/scala-3-macros-comprehensive-guide" rel="noopener external" target="_blank">Rock the JVM - Scala 3 Macros comprehensive guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 Macros - Official overview</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/reflection.html" rel="noopener external" target="_blank">Scala 3 Reflection guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">Scala 3 Macro best practices</a>
</li>
</ul>
<h3 id="apache-spark-integration">Apache Spark Integration</h3>
<ul>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">Spark DataType structural comparators</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/DatasetHolder.html" rel="noopener external" target="_blank">Spark Dataset implicits (toDF/toDS)</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/sql-data-sources-csv.html" rel="noopener external" target="_blank">Spark CSV data source</a>
</li>
<li><a href="https://index.scala-lang.org/vincenzobaz/spark-scala3-encoders" rel="noopener external" target="_blank">Scala 3 encoders for Spark (community)</a>
</li>
</ul>
<h3 id="phantom-types--type-level-programming">Phantom Types &amp; Type-Level Programming</h3>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala - Builder Pattern (Rhetorical Musings)</a>
</li>
</ul>
<h3 id="other-resources">Other Resources</h3>
<ul>
<li><a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia - Generic derivation library</a>
</li>
</ul>
<hr>
<h2 id="what-compile-time-contracts-cant-catch">What compile-time contracts can&rsquo;t catch</h2>
<p>Let&rsquo;s be honest about the boundaries. Compile-time contracts handle a lot, but they&rsquo;re not magic. Here&rsquo;s what they <strong>can&rsquo;t</strong> do:</p>
<table>
	<thead>
			<tr>
					<th>Category</th>
					<th>What It Can&rsquo;t Catch</th>
					<th>Why</th>
					<th>Solution</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><strong>External data drift</strong></td>
					<td>• Third-party APIs that change without telling you; • Kafka topics from teams you don&rsquo;t coordinate with; • S3 buckets written by external vendors</td>
					<td>You don&rsquo;t control their build process</td>
					<td>Use runtime validation (schema registries, data contracts)</td>
			</tr>
			<tr>
					<td><strong>Late-arriving schema changes</strong></td>
					<td>• Schema registry updated after your deploy; • Database columns added while your job runs</td>
					<td>Compile-time happens before deploy</td>
					<td>Runtime checks with version monitoring</td>
			</tr>
			<tr>
					<td><strong>Data quality issues</strong></td>
					<td>• Nulls where you expect values (even if field is non-optional); • Out-of-range numbers (age = -5); • Malformed strings (email without @)</td>
					<td>Compile-time checks structure, not content</td>
					<td>Great Expectations, Deequ, or custom validators</td>
			</tr>
			<tr>
					<td><strong>Non-breaking additions</strong></td>
					<td>• Upstream adds optional field you don&rsquo;t use yet</td>
					<td>This is often fine! <code>Forward</code> policy handles it</td>
					<td>Policy-based awareness or stricter monitoring</td>
			</tr>
			<tr>
					<td><strong>Partial batch failures</strong></td>
					<td>• 1000 records match schema, 5 don&rsquo;t</td>
					<td>Compile-time is binary (compiles or doesn&rsquo;t)</td>
					<td>Runtime validation with error tables/quarantine</td>
			</tr>
	</tbody>
</table>
<p><strong>The takeaway</strong>: Compile-time contracts catch drift <strong>within your controlled codebase</strong> during development and deployment. For everything else - external sources, runtime changes, data quality - layer in runtime validation. Think defense-in-depth: compile-time as the first gate, runtime as the safety net.</p>
<hr>
<p><strong>TL;DR</strong></p>
<ul>
<li>Compile-time evidence + policy types make schema intent explicit and enforceable</li>
<li>TypeInspector pattern organizes type inspection into composable utilities</li>
<li>Phantom types with type-indexing/type-state builders create impossible-to-misuse APIs through compile-time state machines</li>
<li>Field vs element optionality matters for nested types</li>
<li>Nested structures (Maps, Lists, case classes) work through recursive shape building</li>
<li>Real migrations: use Backward during rollout, tighten to Exact after stabilization</li>
<li>Spark&rsquo;s comparators provide runtime defense-in-depth</li>
<li>Schema evolution needs versioning, coexistence strategies, and policy-based transitions</li>
<li>Developer ergonomics matter: clear errors, fast compile times, easy onboarding</li>
<li>Compile-time doesn&rsquo;t replace runtime validation - it complements it for controlled systems</li>
<li>This catches many drift classes early, but you still need runtime checks for external data</li>
<li>Compile-time contracts don&rsquo;t make pipelines unbreakable - they make breakage predictable</li>
</ul>
]]></content:encoded><enclosure url="https://vitthalmirji.com/images/og/2025-09-compile-time-data-contracts.png" type="image/png"/></item><item><title>DET week 2 homework 2: detail jobs audit for compile-time contracts tagged now</title><link>https://vitthalmirji.com/2025/12/det-week-2-homework-2-detail-jobs-audit-for-compile-time-contracts-tagged-now/</link><guid isPermaLink="true">https://vitthalmirji.com/2025/12/det-week-2-homework-2-detail-jobs-audit-for-compile-time-contracts-tagged-now/</guid><pubDate>Wed, 03 Dec 2025 00:00:00 +0000</pubDate><dc:creator>vitthal-mirji</dc:creator><category>technical-writing</category><category>det-workshop</category><category>supporting-details</category><category>detail-jobs</category><category>audit</category><category>concision</category><category>scala</category><category>scala-3</category><category>compile-time</category><category>data-contracts</category><category>metaprogramming</category><category>yaakov-bressler</category><category>homework</category><category>workshop-assignment</category><category>week-2</category><description>Week 2 homework option 2: tag every detail with its job and strike anything without a job, keeping before and after visible. Shows how each line earns its place and how to keep credibility, texture, and orientation without filler.</description><content:encoded><![CDATA[<h2 id="workshop-context-det-week-2---option-2-detail-jobs">Workshop context (DET week 2 - Option 2: Detail jobs)</h2>
<ul>
<li>Source article: <a href="/compile-time-data-contracts">Compile-time data contracts</a>
</li>
<li>Task: tag every detail with its job (Orient, Credibility, Texture, Mechanical, Contrast, Anchor). Strike anything without a job.</li>
<li>Rule: no hard deletes. Before stays visible via strike-through; after stays as plain text.</li>
<li>Reference: Week 1 redline method + Week 2 detail-job rules.</li>
</ul>
<h2 id="legend">Legend</h2>
<ul>
<li>Plain text = kept in main path (after).</li>
<li><del>Strike-through</del> = removed from main path (before, still visible).</li>
<li>Admonitions = why a decision was made.</li>
</ul>
<h2 id="audit-flow-mermaid">Audit flow (mermaid)</h2>
<pre class="mermaid">flowchart TD
    A[Copy full article] --&gt; B[Label each block with job]
    B --&gt; C{Job present?}
    C --&gt;|Yes| D[Keep]
    C --&gt;|No| E[Strike + note]
    D --&gt; F[Ensure each job appears at least once]
    E --&gt; F
    F --&gt; G[Publish redline with before/after visible]
    style E fill:#ffe6e6,stroke:#d33
    style D fill:#e6ffea,stroke:#2c9
</pre><h2 id="detail-inventory-jobs--actions">Detail inventory (jobs + actions)</h2>
<table>
	<thead>
			<tr>
					<th>#</th>
					<th>Detail</th>
					<th>Job(s)</th>
					<th>Action</th>
					<th>Why</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>1</td>
					<td>2 AM page story</td>
					<td>Texture + Anchor</td>
					<td>Keep (shortened)</td>
					<td>Single anchor to hook readers.</td>
			</tr>
			<tr>
					<td>2</td>
					<td>Quick start commands</td>
					<td>Orient + Mechanical</td>
					<td>Keep</td>
					<td>Proof path; reproducibility.</td>
			</tr>
			<tr>
					<td>3</td>
					<td>Macro primer (inline + quotes)</td>
					<td>Orient (duplicate)</td>
					<td>Remove (struck)</td>
					<td>Audience already macro-aware.</td>
			</tr>
			<tr>
					<td>4</td>
					<td>Policy definitions</td>
					<td>Orient + Mechanical</td>
					<td>Keep</td>
					<td>Core API.</td>
			</tr>
			<tr>
					<td>5</td>
					<td>TypeInspector pattern</td>
					<td>Mechanical + Contrast</td>
					<td>Keep</td>
					<td>Shows decisions.</td>
			</tr>
			<tr>
					<td>6</td>
					<td>Integrations section</td>
					<td>Orient (duplicate)</td>
					<td>Remove (struck)</td>
					<td>Duplicates production patterns.</td>
			</tr>
			<tr>
					<td>7</td>
					<td>Scala 2 section</td>
					<td>Orient (edge)</td>
					<td>Remove (struck)</td>
					<td>Non-primary path for this homework.</td>
			</tr>
			<tr>
					<td>8</td>
					<td>Production patterns</td>
					<td>Credibility + Mechanical</td>
					<td>Keep</td>
					<td>Real-world proof.</td>
			</tr>
			<tr>
					<td>9</td>
					<td>Runtime vs compile-time caution</td>
					<td>Credibility + Contrast</td>
					<td>Keep</td>
					<td>Prevents misuse.</td>
			</tr>
	</tbody>
</table>
<p>What if broken pipelines never launched - because the compiler stopped them?</p>
<blockquote>
  <p>&ldquo;If it compiles, contracts align.&rdquo;</p>

</blockquote><p>I&rsquo;ve been working on this problem for a while now. You know how it goes - you&rsquo;re running a data pipeline, everything&rsquo;s fine, then someone changes a field name upstream and boom, your job crashes at 2 AM. The on-call engineer (probably you) gets paged, and you spend the next hour figuring out what went wrong.</p>
<p>Once, an upstream team silently renamed a column <code>amount</code> → <code>amt</code>. Our pipeline didn&rsquo;t crash immediately - it happily wrote nulls for weeks. By the time we caught it, we had to backfill millions of records. That&rsquo;s the day I realized: runtime validation catches problems late; compile-time contracts catch them earlier.</p>
<p>This post shows you how to catch these issues at compile time. Not at runtime. Not even at test time. At compile time. Your code literally won&rsquo;t build if schemas drift.</p>
<p>We&rsquo;ll build this from scratch - first understand how Scala 3 macros work, then write a minimal contract system that proves schema compatibility before your code even runs. I&rsquo;ll also show the Scala 2 version because, let&rsquo;s be honest, many teams are still on Scala 2.</p>
<p>All code here runs in a vanilla sbt project. You can grab the working code in github from <code>https://github.com/vim89/compile-time-data-contracts</code> and run it yourself.</p>
<h2 id="why-this-matters">Why this matters</h2>
<p>Look, I get it. Writing tests is important. But tests are samples. They check what you think might break. The compiler, though? It&rsquo;s exhaustive for structure. It can prove your schemas match - every field, every type, every time.</p>
<p>Here&rsquo;s the thing - schemas change all the time in real projects. And when they do:</p>
<ul>
<li>Your build should fail early, not your production job</li>
<li>Keep transformations pure and effects at the edges (so retries don&rsquo;t duplicate writes)</li>
<li>Let the compiler do the heavy lifting</li>
</ul>
<p><strong>Important upfront:</strong> This focuses on preventing schema drift within your controlled boundaries - the systems you build, deploy, and version. It doesn&rsquo;t replace runtime validation for external data sources or third-party APIs. Think of it as catching a large class of schema mismatches during development and deployment, not eliminating all drift everywhere. You&rsquo;ll still need runtime checks for data coming from systems you don&rsquo;t control.
<div class="admonition warning">
  <div class="admonition-title">Scope boundary</div>
  <div class="admonition-content">
    Compile-time contracts catch drift in your controlled codebase - the transforms you write, the case classes you define, the schemas you version together. They won&rsquo;t catch upstream API changes, late-arriving schema registry updates, or external data sources that change independently. Use compile-time for what you control, runtime validation for everything else. Defense in depth, not either/or.
  </div>
</div></p>
<h2 id="what-well-build">What we&rsquo;ll build</h2>
<p>We&rsquo;re going to build a small compile-time validation system:</p>
<ol>
<li>A way to describe case class structure at compile time (we&rsquo;ll call it &ldquo;shape&rdquo;)</li>
<li>A policy system - Exact, Backward, Forward compatibility modes</li>
<li>A macro that proves &ldquo;these two shapes conform under policy P&rdquo; - or refuses to compile</li>
<li>A phantom-typed pipeline builder that enforces correct construction order</li>
</ol>
<p>First, Scala 3 (using inline + quotes), then Scala 2 (using blackbox macros). I&rsquo;ll point out the differences as we go.</p>
<h3 id="run-the-code-in-github">Run the code in github</h3>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone https://github.com/vim89/compile-time-data-contracts.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> compile-time-data-contracts
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Scala 3</span>
</span></span><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition info">
  <div class="admonition-title">Information</div>
  <div class="admonition-content">
    The code in github includes working examples you can run immediately. All compile-fail cases are documented inline - uncomment them to see the compiler errors.
  </div>
</div>
<hr>
<p><del>## Part 1 - Scala 3: the mental model (inline + quotes)</del></p>
<p><del>Scala 3 macros are different from Scala 2. Better, actually. They&rsquo;re safer and easier to reason about.</del></p>
<p><del>Two main ideas:</del>
<del>- <code>inline def</code>: tells the compiler &ldquo;expand this at compile time&rdquo;</del>
<del>- <code>quotes</code> + <code>Expr[T]</code>: gives you a typed AST that you can inspect</del></p>
<p><del><code>mermaid~~ ~~sequenceDiagram~~ ~~    actor Dev~~ ~~    participant Compiler~~ ~~    participant Macro as &quot;Conforms.impl&quot;~~ ~~    participant Reflect as &quot;TypeRepr/quotes&quot;~~ ~~    Dev-&gt;&gt;Compiler: compile~~ ~~    Compiler-&gt;&gt;Macro: splice ${ ... }~~ ~~    Macro-&gt;&gt;Reflect: inspect types~~ ~~    Reflect--&gt;&gt;Macro: shapes for Out and Contract~~ ~~    alt conforming~~ ~~        Macro--&gt;&gt;Compiler: emit Conforms[Out,Contract,P]~~ ~~        Compiler--&gt;&gt;Dev: build succeeds~~ ~~    else drift~~ ~~        Macro--&gt;&gt;Compiler: report.errorAndAbort(msg)~~ ~~        Compiler--&gt;&gt;Dev: compile error with drift details~~ ~~    end~~ ~~</code></del></p>
<p><del>Here&rsquo;s a minimal macro to see how it works:</del></p>
<p><del>```scala</del>
<del>// build.sbt: use Scala 3</del>
<del>// scalaVersion := &ldquo;3.3.3&rdquo;</del></p>
<p><del>// src/main/scala/com/example/IntroMacro.scala</del>
<del>package com.example</del>
<del>import scala.quoted.*</del></p>
<p><del>object IntroMacro:</del>
~~  inline def show(inline x: Any): String = ${ showImpl(&lsquo;x) }~~
~~  private def showImpl(x: Expr[Any])(using Quotes): Expr[String] =~~
~~    Expr(s&quot;You passed: ${x.show}&quot;)~~
<del>```</del></p>
<p><del>What&rsquo;s happening here?</del>
<del>- The <code>inline</code> method is what users call</del>
<del>- The <code>${ ... }</code> splice calls the implementation</del>
<del>- The implementation gets <code>'x</code> (quoted code) as <code>Expr[Any]</code></del>
<del>- We return <code>Expr[String]</code> - the AST for what the compiler will embed</del></p>
<p><del>Try it:</del></p>
<p><del><code>scala~~ ~~// src/main/scala/com/example/Demo.scala~~ ~~package com.example~~ ~~@main def run(): Unit =~~ ~~  println(IntroMacro.show(1 + 2)) // prints: You passed: 1.+(2)~~ ~~</code></del></p>
<p><del>See how it printed the AST of <code>1 + 2</code>? That&rsquo;s the power here - we can inspect code structure at compile time.</del></p>
<p><del>Check out the <a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 macros overview</a>
 and <a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">best practices</a>
 for more details.</del>
<del><div class="admonition tip">
  <div class="admonition-title">Quick win</div>
  <div class="admonition-content">
    ~~
<del>If you&rsquo;re new to Scala 3 macros, start by reading macro-generated code in the REPL: sbt&gt; console, then scala&gt; import scala.quoted.*, then inspect what your macros actually expand to. Understanding the generated code is 80% of debugging macro issues. The compiler&rsquo;s doing the work - you just need to see what it&rsquo;s actually producing.</del>
~~
  </div>
</div></del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ Removed: Part 1 - Scala 3: the mental model</div>
  <div class="admonition-content">
    Jobless/duplicate for detail-job objective.
  </div>
</div>
<h2 id="part-2---describe-shapes-at-compile-time-scala-3">Part 2 - Describe shapes at compile time (Scala 3)</h2>
<p>Now we need to describe case class structure. In real projects, you might use Shapeless, Magnolia, or Scala 3&rsquo;s Mirror. I&rsquo;ll show a compact Mirror-based version here because it&rsquo;s self-contained.</p>
<p>The idea: a typeclass <code>Shape[A]</code> that knows the fields of <code>A</code>.</p>
<pre class="mermaid">classDiagram
    class Field {
      +name: String
      +tpe: String
      +hasDefault: Boolean
      +isOptional: Boolean
    }
    class Shape {
      +fields(): List~Field~
    }
    Shape &#34;1&#34; --&gt; &#34;*&#34; Field : returns
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Shape.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.deriving.Mirror</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">tpe</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">hasDefault</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Shape</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>     <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">A</span><span class="o">]]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span> <span class="c1">// element optionality handled elsewhere
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="n">given</span> <span class="n">derived</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">using</span> <span class="n">m</span><span class="k">:</span> <span class="kt">Mirror.Of</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">m</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">p</span><span class="k">:</span> <span class="kt">Mirror.ProductOf</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">labels</span> <span class="k">=</span> <span class="n">constValueTuple</span><span class="o">[</span><span class="kt">p.MirroredElemLabels</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">types</span>  <span class="k">=</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">p.MirroredElemTypes</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">zipped</span> <span class="k">=</span> <span class="n">zip</span><span class="o">(</span><span class="n">labels</span><span class="o">,</span> <span class="n">types</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">          <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">            <span class="n">zipped</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">,</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span> <span class="k">=</span> <span class="n">tpe</span><span class="o">.</span><span class="n">startsWith</span><span class="o">(</span><span class="s">&#34;Option[&#34;</span><span class="o">))</span> <span class="o">}.</span><span class="n">toList</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">import</span> <span class="nn">scala.compiletime.</span><span class="o">{</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">constValueTuple</span><span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">T</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">inline</span> <span class="n">erasedValue</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">h</span><span class="o">]</span> <span class="k">:</span><span class="kt">:</span> <span class="k">type</span><span class="kt">Names</span><span class="o">[</span><span class="kt">t</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">constValue</span><span class="o">[</span><span class="err">&#34;&#34;</span> <span class="kt">+</span> <span class="kt">T</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">zip</span><span class="o">[</span><span class="kt">L</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span>, <span class="kt">R</span> <span class="k">&lt;:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">labels</span><span class="k">:</span> <span class="kt">L</span><span class="o">,</span> <span class="n">types</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">labels</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">constValue</span><span class="o">[</span><span class="kt">h</span><span class="o">].</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">-&gt;</span> <span class="n">types</span><span class="o">.</span><span class="n">head</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">zip</span><span class="o">[</span><span class="kt">t</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">types</span><span class="o">.</span><span class="n">tail</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Yeah, it&rsquo;s a bit dense. But here&rsquo;s what it does:</p>
<ul>
<li>Uses Scala 3&rsquo;s Mirror to reflect on case classes at compile time</li>
<li>Extracts field names and types</li>
<li>Marks optional fields (those wrapped in <code>Option</code>)</li>
<li>Returns a <code>List[Field]</code> describing the structure</li>
</ul>
<pre class="mermaid">flowchart LR
    A[&#34;Case class A(...)&#34;] --&gt; M[&#34;Mirror.Of[A]&#34;]
    M --&gt; L[&#34;MirroredElemLabels&#34;]
    M --&gt; T[&#34;MirroredElemTypes&#34;]
    L --&gt; Z[&#34;zip(labels, types)&#34;]
    T --&gt; Z
    Z --&gt; S[&#34;Shape[A].fields: List[Field]&#34;]
</pre><p>Usage:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ShapeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">note</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkShape</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="n">println</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">fields</span><span class="o">)</span> <span class="o">//</span> <span class="nc">List</span><span class="o">(</span><span class="nc">Field</span><span class="o">(</span><span class="s">&#34;id&#34;</span><span class="o">,</span><span class="s">&#34;Long&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;email&#34;</span><span class="o">,</span><span class="s">&#34;String&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;note&#34;</span><span class="o">,</span><span class="s">&#34;Option[String]&#34;</span><span class="o">,</span><span class="kc">false</span><span class="o">,</span><span class="kc">true</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Now we can ask the compiler &ldquo;what fields does this case class have?&rdquo; at compile time. That&rsquo;s the foundation.</p>
<blockquote>
  <p>If you prefer libraries, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works great for both Scala 2 and 3.</p>

</blockquote><hr>
<h2 id="part-3---policies-exact--backward--forward">Part 3 - Policies: Exact / Backward / Forward</h2>
<p>Here&rsquo;s where we define compatibility rules. Think of these as your migration strategies:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Policy.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Policy</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Policy</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">Exact</span>     <span class="kt">extends</span> <span class="kt">Policy</span>
</span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Backward</span>  <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// producer can add optional/defaulted fields
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Forward</span>   <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// consumer tolerates extra fields
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Exact</span>    <span class="k">extends</span> <span class="nc">Exact</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Backward</span> <span class="k">extends</span> <span class="nc">Backward</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Forward</span>  <span class="k">extends</span> <span class="nc">Forward</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Why three policies?</p>
<ul>
<li><strong>Exact</strong>: Schemas must match exactly. No surprises. Use this for critical paths.</li>
<li><strong>Backward</strong>: Producer can add optional fields. Useful when rolling out new features without breaking consumers.</li>
<li><strong>Forward</strong>: Consumer ignores extra fields. Useful when the producer might send more than you need.</li>
</ul>
<pre class="mermaid">flowchart LR
    EX[&#34;Policy.Exact&#34;]
    BW[&#34;Policy.Backward&#34;]
    FW[&#34;Policy.Forward&#34;]

    EX --- BW
    EX --- FW

    EX:::tight --&gt;|no diffs allowed| OK1[&#34;✔ match only&#34;]
    BW:::relax --&gt;|allow missing optional in producer\nallow extra fields in producer| OK2[&#34;✔ migration adds&#34;]
    FW:::relax --&gt;|allow extra fields in consumer side| OK3[&#34;✔ tolerant reader&#34;]

    classDef tight fill:#eef,stroke:#446;
    classDef relax fill:#efe,stroke:#484;
</pre><p>Visual (save/share):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/policy-lattice.svg"
            alt="Policy lattice"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    A tiny policy lattice - Exact / Backward / Forward.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<p>In practice: start with Exact, relax to Backward/Forward during migrations, then tighten back to Exact once stable.
<div class="admonition success">
  <div class="admonition-title">Migration strategy</div>
  <div class="admonition-content">
    The policy progression that works in production: Exact for stable pipelines, Backward during producer rollouts (allows new optional fields), Forward during consumer updates (tolerates extras), then back to Exact after migration completes. Document the expected timeline - &ldquo;Backward policy through Q2 migration, then tightening to Exact.&rdquo; Prevents policy drift from becoming permanent.
  </div>
</div></p>
<hr>
<p><del>## Part 4 - How this fits with existing tools</del></p>
<p><del>Before we dive deeper, you might be thinking: &ldquo;We already have Avro, Protobuf, schema registries, and Great Expectations. Why add compile-time contracts?&quot;</del></p>
<p><del>Fair question. Here&rsquo;s how they compare:</del></p>
<p><del><strong>Schema Registry (Confluent, AWS Glue)</strong></del>
<del>- <strong>What it does</strong>: Centralized schema storage with version control</del>
<del>- <strong>When it helps</strong>: Runtime validation across services, schema evolution tracking</del>
<del>- <strong>The gap</strong>: You find mismatches when the job runs, not when you compile</del>
<del>- <strong>Use together</strong>: Schema registry for cross-service contracts; compile-time for intra-repo checks</del></p>
<p><del><strong>Avro / Protobuf</strong></del>
<del>- <strong>What they do</strong>: Binary serialization with embedded schemas</del>
<del>- <strong>When they help</strong>: Network efficiency, strong contracts between services</del>
<del>- <strong>The gap</strong>: Schema compatibility is checked at runtime or via external tooling</del>
<del>- <strong>Use together</strong>: Avro/Protobuf for wire format; compile-time checks to ensure your case classes match those schemas</del></p>
<p><del><strong>Great Expectations / Deequ</strong></del>
<del>- <strong>What they do</strong>: Runtime data quality validation (nulls, ranges, distributions)</del>
<del>- <strong>When they help</strong>: Catching bad data values, statistical anomalies</del>
<del>- <strong>The gap</strong>: They validate data content, not schema structure at compile time</del>
<del>- <strong>Use together</strong>: Compile-time for structure; Great Expectations for data quality</del></p>
<p><del><strong>The compile-time advantage</strong>: Catches drift in your controlled codebase before deployment. If your transform expects field <code>X</code> but the source produces field <code>Y</code>, your build fails - not your production job. Think of it as an additional safety layer for systems you build and version together.</del></p>
<p><del>Real scenario: You have Avro schemas in a registry, but your Scala pipeline reads them into case classes. Compile-time contracts verify those case classes match the expected contract before you deploy. If upstream changes the Avro schema, your build breaks - you don&rsquo;t wait for the job to crash.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ Removed: Part 4 - How this fits</div>
  <div class="admonition-content">
    Jobless/duplicate for detail-job objective.
  </div>
</div>
<hr>
<h2 id="part-5---understanding-typerepr-the-compilers-view-of-types">Part 5 - Understanding TypeRepr: The compiler&rsquo;s view of types</h2>
<p>Before we dive into the macro, you need to understand <code>TypeRepr</code> - Scala 3&rsquo;s way of representing types during compilation.</p>
<p>Think of it like this: When you write <code>List[String]</code>, the compiler doesn&rsquo;t just see text. It sees a structured tree:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">AppliedType(List, [String])
</span></span><span class="line"><span class="cl">   │               │
</span></span><span class="line"><span class="cl">   │               └─ TypeRepr for String
</span></span><span class="line"><span class="cl">   └─ Type constructor</span></span></code></pre></td></tr></table>
</div>
</div><pre class="mermaid">flowchart TB
    A[&#34;List[String]&#34;] --&gt; B[&#34;AppliedType(List,[String])&#34;]
    B --&gt; C[&#34;tycon = List&#34;]
    B --&gt; D[&#34;args = [String]&#34;]
</pre><p><strong>Key TypeRepr operations:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span> <span class="o">=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 1. Subtype checking: Is List[String] a subtype of Seq[?]?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>  <span class="c1">// true
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 2. Type equality: Is this exactly String?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// false
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 3. Pattern matching: Break it apart
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="n">tycon</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// tycon = List
</span></span></span><span class="line"><span class="cl">    <span class="o">//</span> <span class="n">args</span> <span class="k">=</span> <span class="o">[</span><span class="kt">String</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> To compare schemas, we need to ask questions like:</p>
<ul>
<li>&ldquo;Is this field an Option?&rdquo;</li>
<li>&ldquo;What&rsquo;s inside this List?&rdquo;</li>
<li>&ldquo;Does this case class have a primary constructor?&rdquo;</li>
</ul>
<p><code>TypeRepr</code> lets us answer these at compile time.</p>
<hr>
<h2 id="part-5---the-typeinspector-pattern-one-question-at-a-time">Part 5 - The TypeInspector pattern: One question at a time</h2>
<p>Now let&rsquo;s build the utilities. Each function answers ONE question about a type. Start simple:</p>
<h3 id="inspector-1-is-this-a-case-class">Inspector 1: Is this a case class?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>  <span class="c1">// Get the symbol (the &#34;name&#34; of the type)
</span></span></span><span class="line"><span class="cl">  <span class="n">s</span><span class="o">.</span><span class="n">isClassDef</span> <span class="o">&amp;&amp;</span> <span class="n">s</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">Case</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>typeSymbol</code> - Every type has a symbol (like a unique ID)</li>
<li><code>isClassDef</code> - Is it a class? (not a trait, object, etc.)</li>
<li><code>flags.is(Flags.Case)</code> - Does it have the <code>case</code> modifier?</li>
</ul>
<p><strong>Try it mentally:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>  <span class="c1">// ✅ isClassDef=true, has Case flag
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">UserTrait</span>            <span class="c1">// ❌ isClassDef=false
</span></span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">UserObject</span>          <span class="o">//</span> <span class="o">❌</span> <span class="n">not</span> <span class="n">a</span> <span class="n">class</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-2-what-are-the-type-arguments">Inspector 2: What are the type arguments?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span> <span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">args</span>  <span class="c1">// Extract the arguments
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">_</span>                    <span class="k">=&gt;</span> <span class="nc">Nil</span>   <span class="o">//</span> <span class="nc">No</span> <span class="n">arguments</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>AppliedType</code> - Pattern for generic types like <code>List[String]</code> or <code>Map[String, Int]</code></li>
<li><code>args</code> - The type parameters: <code>[String]</code> or <code>[String, Int]</code></li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>        <span class="c1">// AppliedType(List, [String])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>    <span class="c1">// AppliedType(Map, [String, Int])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String, Int]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>              <span class="c1">// Not an AppliedType
</span></span></span><span class="line"><span class="cl">                    <span class="o">//</span> <span class="n">appliedArgs</span> <span class="n">returns</span> <span class="nc">Nil</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-3-is-this-an-option">Inspector 3: Is this an Option?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>  <span class="c1">// Is it a subtype of Option[Something]?
</span></span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span>           <span class="c1">// Yes! Extract the Something
</span></span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span>                             <span class="o">//</span> <span class="nc">Nope</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>&lt;:&lt;</code> - Subtype check: &ldquo;Is this type compatible with Option[?]?&rdquo;</li>
<li><code>Option[?]</code> - The <code>?</code> means &ldquo;Option of anything&rdquo;</li>
<li><code>appliedArgs(t).headOption</code> - Get the first (and only) type argument</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// ✅ Is subtype of Option[?]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// headOption returns Some(String)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Some</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>       <span class="c1">// ✅ Some is subtype of Option
</span></span></span><span class="line"><span class="cl">                <span class="c1">// Returns Some(Int)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>          <span class="c1">// ❌ Not an Option
</span></span></span><span class="line"><span class="cl">                <span class="o">//</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> Field-level optionality. When we see <code>age: Option[Int]</code>, we need to know:</p>
<ol>
<li>The field itself is optional</li>
<li>The underlying type is <code>Int</code></li>
</ol>
<h3 id="inspector-4-is-this-a-sequence">Inspector 4: Is this a sequence?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">isSeqLike</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Vector</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">isSeqLike</span> <span class="n">then</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span> <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s ANY kind of sequence type</li>
<li>If yes, extract what&rsquo;s inside (the element type)</li>
</ul>
<p><strong>Why check multiple types?</strong> Because Scala has many collection types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>    <span class="c1">// ✅ Is List[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Seq</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>        <span class="c1">// ✅ Is Seq[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Vector</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="c1">// ✅ Is Vector[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span>     <span class="c1">// ✅ Is Array[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>     <span class="o">//</span> <span class="o">✅</span> <span class="nc">Is</span> <span class="nc">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p>All these should become <code>SequenceShape(element)</code>.</p>
<h3 id="inspector-5-is-this-a-map">Inspector 5: Is this a Map?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[(</span><span class="kt">TypeRepr</span>, <span class="kt">TypeRepr</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Map</span><span class="o">[</span><span class="kt">?</span>, <span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">k</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">v</span> <span class="kt">::</span> <span class="kt">Nil</span> <span class="o">=&gt;</span> <span class="nc">Some</span><span class="o">((</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">))</span>  <span class="c1">// Got key and value
</span></span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Map requires two type args: </span><span class="si">${</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s a <code>Map[Something, Something]</code></li>
<li>Extract BOTH type arguments: key and value</li>
<li>If Map has wrong number of args, abort (compiler error)</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>  <span class="c1">// ✅ Returns Some((String, Int))
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Long</span>, <span class="kt">User</span><span class="o">]</span>   <span class="c1">// ✅ Returns Some((Long, User))
</span></span></span><span class="line"><span class="cl"><span class="nc">String</span>            <span class="o">//</span> <span class="o">❌</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-6-can-this-be-a-map-key">Inspector 6: Can this be a Map key?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isAtomicKey</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Short</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Boolean</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why this matters:</strong> Maps in Spark StructType only support primitive keys. You can&rsquo;t have <code>Map[User, String]</code> - User isn&rsquo;t a primitive.</p>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">User</span><span class="o">]</span>  <span class="c1">// ✅ String is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Data</span><span class="o">]</span>     <span class="c1">// ✅ Int is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">User</span>, <span class="kt">String</span><span class="o">]</span>  <span class="o">//</span> <span class="o">❌</span> <span class="nc">User</span> <span class="n">is</span> <span class="nc">NOT</span> <span class="n">atomic</span> <span class="o">-</span> <span class="n">compile</span> <span class="n">error</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="the-pattern-recap">The pattern recap:</h3>
<p>Each inspector has ONE job:</p>
<ol>
<li><strong>isCaseClass</strong> - &ldquo;Is this a case class?&rdquo;</li>
<li><strong>appliedArgs</strong> - &ldquo;What are the generic type arguments?&rdquo;</li>
<li><strong>optionArg</strong> - &ldquo;Is this an Option, and what&rsquo;s inside?&rdquo;</li>
<li><strong>seqArg</strong> - &ldquo;Is this a sequence, and what&rsquo;s the element type?&rdquo;</li>
<li><strong>mapArgs</strong> - &ldquo;Is this a Map, and what are key/value types?&rdquo;</li>
<li><strong>isAtomicKey</strong> - &ldquo;Can this be a Map key?&rdquo;</li>
</ol>
<p>Compose these to answer complex questions. That&rsquo;s the TypeInspector pattern.</p>
<hr>
<h2 id="part-6---building-shapes-recursively-the-interpreter-pattern">Part 6 - Building shapes recursively: The interpreter pattern</h2>
<p>Now we use those inspectors to build a <code>TypeShape</code> - our internal representation of a type&rsquo;s structure.</p>
<p>This is recursive. For <code>List[Option[User]]</code>, we build:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">SequenceShape(
</span></span><span class="line"><span class="cl">  OptionalShape(
</span></span><span class="line"><span class="cl">    StructShape([field1, field2, ...])
</span></span><span class="line"><span class="cl">  )
</span></span><span class="line"><span class="cl">)</span></span></code></pre></td></tr></table>
</div>
</div><p>Let me show you the algorithm step by step.</p>
<h3 id="the-shape-building-algorithm">The shape building algorithm:</h3>
<pre class="mermaid">flowchart TD
    T[&#34;typeShapeOf(t)&#34;] --&gt; O{&#34;Option?&#34;}
    O -- &#34;Yes&#34; --&gt; OI[&#34;inner = optionArg(t)&#34;] --&gt; R1[&#34;typeShapeOf(inner)&#34;]
    O -- &#34;No&#34; --&gt; S{&#34;Sequence?&#34;}
    S -- &#34;Yes&#34; --&gt; SE[&#34;elem = seqArg(t)&#34;] --&gt; R2[&#34;SequenceShape(typeShapeOf(elem))&#34;]
    S -- &#34;No&#34; --&gt; M{&#34;Map?&#34;}
    M -- &#34;Yes&#34; --&gt; MKV[&#34;(k,v) = mapArgs(t)&#34;]
    MKV --&gt; K{&#34;isAtomicKey(k)?&#34;}
    K -- &#34;No&#34; --&gt; ERR[&#34;abort: unsupported key&#34;]
    K -- &#34;Yes&#34; --&gt; MV[&#34;MapShape(PrimitiveShape(k), typeShapeOf(v))&#34;]
    M -- &#34;No&#34; --&gt; CC{&#34;Case class?&#34;}
    CC -- &#34;Yes&#34; --&gt; FS[&#34;StructShape(collect FieldShape...)&#34;]
    CC -- &#34;No&#34; --&gt; P[&#34;PrimitiveShape(t.show)&#34;]
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">TypeShape</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Step 1: Is it an Option?
</span></span></span><span class="line"><span class="cl">  <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Step 2: Is it a sequence?
</span></span></span><span class="line"><span class="cl">    <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">      <span class="c1">// Step 3: Is it a Map?
</span></span></span><span class="line"><span class="cl">      <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="o">!</span><span class="n">isAtomicKey</span><span class="o">(</span><span class="n">k</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Unsupported Map key: </span><span class="si">${</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="nc">MapShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="o">),</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">v</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1">// Step 4: Is it a case class?
</span></span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// ... handle case class
</span></span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// Step 5: Fallback - primitive
</span></span></span><span class="line"><span class="cl">          <span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>The order matters!</strong> We check Option first, then sequences, then maps, then case classes, finally primitives.</p>
<h3 id="walk-through-an-example-listoptionstring">Walk through an example: <code>List[Option[String]]</code></h3>
<pre class="mermaid">flowchart TD
    A[&#34;List[Option[String]]&#34;] --&gt;|typeShapeOf| B{&#34;Option?&#34;}
    B -- &#34;No&#34; --&gt; C{&#34;Sequence?&#34;}
    C -- &#34;Yes&#34; --&gt; D[&#34;elem = Option[String]&#34;]
    D --&gt; E[&#34;typeShapeOf(Option[String])&#34;]
    E --&gt; F{&#34;Option?&#34;}
    F -- &#34;Yes&#34; --&gt; G[&#34;inner = String&#34;]
    G --&gt; H[&#34;typeShapeOf(String)&#34;]
    H --&gt; I{&#34;Case class?&#34;}
    I -- &#34;No&#34; --&gt; J[&#34;PrimitiveShape(String)&#34;]
    J --&gt; K[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><ul>
<li>
<p><strong>Step 1:</strong> Is <code>List[Option[String]]</code> an Option?</p>
<ul>
<li>No. <code>optionArg</code> returns <code>None</code>.</li>
<li>Continue to Step 2.</li>
</ul>
</li>
<li>
<p><strong>Step 2:</strong> Is it a sequence?</p>
<ul>
<li>Yes! <code>seqArg</code> returns <code>Some(Option[String])</code>.</li>
<li>Build <code>SequenceShape(typeShapeOf(Option[String]))</code>.</li>
<li><strong>Recurse</strong> on <code>Option[String]</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 1:</strong> Is <code>Option[String]</code> an Option?</p>
<ul>
<li>Yes! <code>optionArg</code> returns <code>Some(String)</code>.</li>
<li><strong>Recurse</strong> on <code>String</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 2:</strong> Is <code>String</code> an Option?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 3:</strong> Is <code>String</code> a sequence?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 4:</strong> Is <code>String</code> a Map?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 5:</strong> Is <code>String</code> a case class?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 6:</strong> Fallback to primitive.</p>
<ul>
<li>Return <code>PrimitiveShape(&quot;String&quot;)</code>.</li>
</ul>
</li>
</ul>
<p><strong>Unwind the recursion:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="o">(</span><span class="n">from</span> <span class="nc">Option</span> <span class="n">recursion</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, where did the <code>Option</code> go? It got consumed! The <code>typeShapeOf</code> for <code>Option</code> just recurses on the inner type. This is the edge case I mentioned - element-level optionality gets lost.</p>
<h3 id="handling-case-classes">Handling case classes:</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">sym</span>    <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>                      <span class="c1">// Get the class symbol
</span></span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">params</span> <span class="k">=</span> <span class="n">sym</span><span class="o">.</span><span class="n">primaryConstructor</span><span class="o">.</span><span class="n">paramSymss</span><span class="o">.</span><span class="n">flatten</span>  <span class="c1">// Get constructor params
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">fields</span> <span class="k">=</span> <span class="n">params</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">p</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">name</span>       <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span>                      <span class="c1">// Field name
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">ptpe</span>       <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">memberType</span><span class="o">(</span><span class="n">p</span><span class="o">)</span>             <span class="c1">// Field type
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">HasDefault</span><span class="o">)</span> <span class="c1">// Has default value?
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nc">StructShape</span><span class="o">(</span><span class="n">fields</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Let&rsquo;s break this down line by line:</strong></p>
<ol>
<li><strong><code>t.typeSymbol</code></strong> - Get the symbol for the case class</li>
<li><strong><code>sym.primaryConstructor</code></strong> - Case classes have a primary constructor</li>
<li><strong><code>.paramSymss.flatten</code></strong> - Get all parameters (flatten handles multiple param lists)</li>
<li><strong>For each parameter <code>p</code>:</strong>
<ul>
<li><strong><code>p.name</code></strong> - The field name: <code>&quot;id&quot;</code>, <code>&quot;email&quot;</code>, etc.</li>
<li><strong><code>t.memberType(p)</code></strong> - The field&rsquo;s type as TypeRepr</li>
<li><strong><code>p.flags.is(Flags.HasDefault)</code></strong> - Check if it has a default value like <code>age: Int = 0</code></li>
<li><strong><code>optionArg(ptpe).fold(...)</code></strong> - Check if the field type is Option[T]:
<ul>
<li>If yes: extract T, set <code>isOpt = true</code></li>
<li>If no: use the type as-is, set <code>isOpt = false</code></li>
</ul>
</li>
<li><strong><code>typeShapeOf(uT)</code></strong> - <strong>Recurse!</strong> Build the shape for the field&rsquo;s underlying type</li>
</ul>
</li>
</ol>
<h3 id="example-case-class-userid-long-email-string-age-optionint--none">Example: <code>case class User(id: Long, email: String, age: Option[Int] = None)</code></h3>
<pre class="mermaid">flowchart TD
    U[&#34;User(id: Long, email: String, age: Option[Int] = None)&#34;]
    U --&gt; F1[&#34;Field &#39;id&#39; : Long&#34;]
    F1 --&gt; S1[&#34;PrimitiveShape(Long)&#34;]
    U --&gt; F2[&#34;Field &#39;email&#39; : String&#34;]
    F2 --&gt; S2[&#34;PrimitiveShape(String)&#34;]
    U --&gt; F3[&#34;Field &#39;age&#39; : Option[Int] (default None)&#34;]
    F3 --&gt; O1{&#34;Option?&#34;}
    O1 -- &#34;Yes&#34; --&gt; A1[&#34;inner = Int&#34;]
    A1 --&gt; S3[&#34;PrimitiveShape(Int)&#34;]
    S1 --&gt; COLLECT[&#34;Collect fields&#34;]
    S2 --&gt; COLLECT
    S3 --&gt; COLLECT
    COLLECT --&gt; FINAL[&#34;StructShape([ id:PrimitiveShape(Long), email:PrimitiveShape(String), age:PrimitiveShape(Int) (optional, default) ])&#34;]
</pre><p><strong>Processing:</strong></p>
<ul>
<li>
<p><strong>Field 1: id</strong></p>
<ul>
<li><code>name</code> = &ldquo;id&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Long</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(Long)</code> = None → <code>uT = Long</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(Long)</code> = <code>PrimitiveShape(&quot;Long&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;id&quot;, PrimitiveShape(&quot;Long&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 2: email</strong></p>
<ul>
<li><code>name</code> = &ldquo;email&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>String</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(String)</code> = None → <code>uT = String</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(String)</code> = <code>PrimitiveShape(&quot;String&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;email&quot;, PrimitiveShape(&quot;String&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 3: age</strong></p>
<ul>
<li><code>name</code> = &ldquo;age&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Option[Int]</code></li>
<li><code>hasDefault</code> = <strong>true</strong> (has <code>= None</code>)</li>
<li><code>optionArg(Option[Int])</code> = Some(<code>Int</code>) → <code>uT = Int</code>, <code>isOpt = true</code></li>
<li><code>typeShapeOf(Int)</code> = <code>PrimitiveShape(&quot;Int&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;age&quot;, PrimitiveShape(&quot;Int&quot;), hasDefault=true, isOptional=true)</code></li>
</ul>
</li>
<li>
<p><strong>Final shape:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">StructShape</span><span class="o">([</span>
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">id</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Long</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">email</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">String</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">age</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Int</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">true</span>, <span class="kt">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">])</span></span></span></code></pre></td></tr></table>
</div>
</div></li>
</ul>
<h3 id="why-this-matters-1">Why this matters:</h3>
<p>Now we have a normalized representation of the case class. We can compare two <code>StructShape</code>s to see if they match, regardless of the original Scala syntax.</p>
<p>The recursive structure handles arbitrarily nested types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Order</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>           <span class="c1">// Recurses into List, then into LineItem
</span></span></span><span class="line"><span class="cl">  <span class="n">metadata</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">],</span>   <span class="c1">// Recurses into Map, then String (twice)
</span></span></span><span class="line"><span class="cl">  <span class="n">address</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">]</span>         <span class="c1">// Recurses into Option, then Address
</span></span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Each recursion builds a piece of the tree. The tree is the schema.</p>
<hr>
<h2 id="part-7---field-vs-element-optionality-a-critical-distinction">Part 7 - Field vs Element optionality: A critical distinction</h2>
<p>Here&rsquo;s something subtle that most guides miss. The code in github handles two kinds of optionality:</p>
<h3 id="field-level-optionality-option-on-the-field-itself">Field-level optionality (Option on the field itself)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                                   <span class="o">^^^^^^^^^^^</span> <span class="n">field</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The macro detects this and sets <code>isOptional = true</code> on the FieldShape:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="element-level-optionality-option-inside-a-collection">Element-level optionality (Option inside a collection)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Data</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                          <span class="o">^^^^^^^^^^^^^^</span> <span class="n">element</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This creates a nested shape: <code>SequenceShape(OptionShape(PrimitiveShape(&quot;String&quot;)))</code></p>
<p>Wait, that&rsquo;s not in the code in github! Let me check&hellip; Actually, the code in github&rsquo;s <code>typeShapeOf</code> handles this by recursing:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>So <code>List[Option[String]]</code> becomes:</p>
<ol>
<li><code>seqArg</code> detects List → extract <code>Option[String]</code></li>
<li>Recurse on <code>Option[String]</code> → <code>typeShapeOf(Option[String])</code></li>
<li><code>optionArg</code> detects Option → extract <code>String</code></li>
<li>Base case: <code>PrimitiveShape(&quot;String&quot;)</code></li>
</ol>
<p>But the final shape is <code>SequenceShape(PrimitiveShape(&quot;String&quot;))</code> - the intermediate Option is consumed!</p>
<p><strong>Why this matters for contracts</strong>: When you write:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Producer</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span></span></span></code></pre></td></tr></table>
</div>
</div><p>These should probably NOT conform under Exact. The producer allows null elements, the contract doesn&rsquo;t. But the current code in github might allow it because the Option gets unwrapped.</p>
<p>This is the kind of edge case you discover when shipping to production. The fix would be to add an <code>OptionalShape</code> wrapper in the model.
<div class="admonition caution">
  <div class="admonition-title">Optionality edge case</div>
  <div class="admonition-content">
    The element-level optionality issue (List[Option[String]] getting flattened) is real and subtle. If your schemas use nullable collections, test that the compile-time check actually catches the difference between List[String] and List[Option[String]]. The current implementation may allow mismatches here. Add explicit test cases for your nullable collection patterns before relying on them in production.
  </div>
</div></p>
<p>Visual (field vs element optionality):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/optionality.svg"
            alt="Field vs Element Optionality"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    Field optionality vs element optionality - they&#39;re not the same.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<pre class="mermaid">flowchart LR
    A[&#34;List[Option[String]]&#34;] --&gt; B{&#34;Element optionality?&#34;}
    B -- &#34;intended shape&#34; --&gt; C[&#34;SequenceShape(OptionalShape(PrimitiveShape(String)))&#34;]
    B -- &#34;current code in github\n(consumes Option)&#34; --&gt; D[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><hr>
<h2 id="part-8---compiletime--evidence-the-full-picture">Part 8 - Compile‑time <strong>&ldquo;conforms&rdquo;</strong> evidence (the full picture)</h2>
<p>Now let&rsquo;s put it all together. Here&rsquo;s the complete macro from the code in github:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Conforms.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.quoted.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">class</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">object</span> <span class="nc">Conforms</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">materialize</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">$</span><span class="o">{</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">P:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span>, <span class="kt">Boolean</span>, <span class="kt">Boolean</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">fs</span> <span class="k">=</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span>
</span></span><span class="line"><span class="cl">        <span class="n">fs</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">hasDefault</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">isOptional</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Compare fields by name (unordered, case-sensitive for now)
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">outMap</span>      <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contractMap</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">missing</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">outMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s">:</span><span class="si">$t$</span><span class="s">{if opt then &#34;</span> <span class="o">(</span><span class="n">optional</span><span class="o">)</span><span class="s">&#34; else &#34;&#34;}&#34;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">extra</span>   <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span>      <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">contractMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mism</span>    <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span> <span class="o">!=</span> <span class="n">t</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> expected </span><span class="si">$t</span><span class="s">, found </span><span class="si">${</span><span class="n">outMap</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">_1</span><span class="si">}</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">!=</span> <span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> optionality mismatch&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">okMissing</span><span class="o">,</span> <span class="n">okExtra</span><span class="o">,</span> <span class="n">okMism</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Exact</span><span class="o">]</span>    <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Backward</span><span class="o">]</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">.</span><span class="n">filterNot</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="s">&#34;(optional)&#34;</span><span class="o">)),</span> <span class="nc">Nil</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Forward</span><span class="o">]</span>  <span class="k">=&gt;</span> <span class="o">(</span><span class="nc">Nil</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span>                  <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">okMissing</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okExtra</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okMism</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="s">s&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">        |Compile‑time contract drift (policy: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span><span class="si">}</span><span class="s">).
</span></span></span><span class="line"><span class="cl"><span class="s">        |Out: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span><span class="si">}</span><span class="s"> vs Contract: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Missing: </span><span class="si">${</span><span class="n">okMissing</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Extra: </span><span class="si">${</span><span class="n">okExtra</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Mismatched: </span><span class="si">${</span><span class="n">okMism</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |&#34;&#34;&#34;</span><span class="o">.</span><span class="n">stripMargin</span>
</span></span><span class="line"><span class="cl">      <span class="n">quotes</span><span class="o">.</span><span class="n">reflect</span><span class="o">.</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="n">msg</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="k">new</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Let me break down what&rsquo;s happening:</p>
<ol>
<li>
<p><strong>Extract shapes</strong>: We summon <code>Shape[Out]</code> and <code>Shape[Contract]</code> and get their field lists. Note the <code>.valueOrAbort</code> - this forces compile-time evaluation.</p>
</li>
<li>
<p><strong>Compare</strong>: We build maps of field names → (type, optionality) and find differences.</p>
</li>
<li>
<p><strong>Apply policy</strong>:</p>
<ul>
<li>Exact: all differences are errors</li>
<li>Backward: missing optional fields are OK, extras are OK</li>
<li>Forward: extras are OK (but missing required fields are not)</li>
</ul>
</li>
<li>
<p><strong>Abort or succeed</strong>: If there are violations, we call <code>report.errorAndAbort</code> with a detailed message showing exactly what&rsquo;s wrong. Otherwise, we return the evidence.</p>
</li>
</ol>
<p>The beauty here: <strong>this runs during compilation</strong>. If schemas don&rsquo;t match, your code won&rsquo;t compile. Period.
<div class="admonition danger">
  <div class="admonition-title">Build breakage alert</div>
  <div class="admonition-content">
    Compile-time contract checks will break your build when schemas drift. That&rsquo;s the point, but coordinate with your team. When upstream adds a field, every downstream pipeline build fails until they update contracts or transforms. This forces alignment, but can block deployments if not managed. Use CI branch builds to preview impact before merging schema changes.
  </div>
</div></p>
<p>Use it like this:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ConformsSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutOk</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutMissing</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkConformance</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutOk</span>,      <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>   <span class="c1">// ✅ compiles
</span></span></span><span class="line"><span class="cl">  <span class="c1">// summon[Conforms[OutMissing, Contract, Policy.Exact]] // ❌ compile‑time error (Missing: email:String)
</span></span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span> <span class="o">//</span> <span class="o">✅</span> <span class="n">compiles</span> <span class="o">(</span><span class="n">migration</span> <span class="n">mode</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>That&rsquo;s it. The entire core. In production, you&rsquo;d want to handle the element optionality issue I mentioned, but the structure stays the same.</p>
<hr>
<h2 id="part-9---developer-ergonomics-what-using-this-actually-feels-like">Part 9 - Developer ergonomics: What using this actually feels like</h2>
<p>Let&rsquo;s talk about the practical side - what happens when you actually use this in your daily work?</p>
<h3 id="compile-times">Compile times</h3>
<p><strong>Question: Does this slow down compilation?</strong></p>
<p>In practice, not noticeably. The macro runs once per <code>summon[Conforms[...]]</code> call during compilation. For a typical pipeline with 5-10 contract checks, you&rsquo;re adding maybe 100-200ms to your build. Compare that to the hours you&rsquo;d spend debugging a production schema mismatch.</p>
<h3 id="error-messages">Error messages</h3>
<p>When schemas drift, you get this:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[error] Compile-time contract drift (policy: Policy.Exact).
</span></span><span class="line"><span class="cl">[error] Out: CustomerProducer vs Contract: CustomerContract
</span></span><span class="line"><span class="cl">[error] Extra: segment
</span></span><span class="line"><span class="cl">[error] Missing:
</span></span><span class="line"><span class="cl">[error] Mismatched:</span></span></code></pre></td></tr></table>
</div>
</div><p>Clear, actionable, and it stops your build. No cryptic macro errors, no stack traces. Just: &ldquo;Hey, you have an extra field called <code>segment</code>.&rdquo;</p>
<h3 id="integration-patterns">Integration patterns</h3>
<p><strong>With Spark:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Your Spark job
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">df</span> <span class="k">=</span> <span class="n">spark</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">typed</span> <span class="k">=</span> <span class="n">df</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">]</span>  <span class="c1">// Spark encoder
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Compile-time check before transformation
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerProducer</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">output</span> <span class="k">=</span> <span class="n">typed</span><span class="o">.</span><span class="n">transform</span><span class="o">(</span><span class="n">dropSegment</span><span class="o">).</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>With schema registries:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Avro schema in registry → case class → contract check
</span></span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerAvro</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">amt</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerAvro</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span> <span class="nc">Build</span> <span class="n">fails</span> <span class="k">if</span> <span class="nc">Avro</span> <span class="n">schema</span> <span class="n">drift</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>CI/CD integration:</strong>
Just run <code>sbt compile</code>. If contracts drift, the build fails. No special tooling needed. Works with Jenkins, GitHub Actions, GitLab CI - anything that runs sbt.</p>
<h3 id="onboarding">Onboarding</h3>
<p><strong>New team members ask</strong>: &ldquo;Why won&rsquo;t this compile?&rdquo;</p>
<p><strong>Answer</strong>: &ldquo;Check the error. You&rsquo;re trying to use <code>CustomerProducer</code> but the contract expects <code>CustomerContract</code>. Either adjust your transform or update the contract.&rdquo;</p>
<p>That&rsquo;s it. The compiler guides them. After one or two cases, they get it.</p>
<h3 id="when-to-use-vs-skip">When to use vs skip</h3>
<p><strong>Use compile-time contracts when</strong>:</p>
<ul>
<li>Multiple teams work on the same pipeline codebase</li>
<li>Schema changes break production regularly</li>
<li>You want fast feedback (compile vs deploy-and-test)</li>
</ul>
<p><strong>Skip when</strong>:</p>
<ul>
<li>One-off scripts or exploratory work</li>
<li>External APIs you can&rsquo;t control (use runtime validation)</li>
<li>Schema is genuinely dynamic (JSON with unknown keys)</li>
</ul>
<hr>
<h2 id="part-10---nested-types-maps-lists-and-deep-structures">Part 10 - Nested types: Maps, Lists, and deep structures</h2>
<p>The code in github handles nested types beautifully. Look at this example from <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Deep / Nested structures
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">LineItem</span><span class="o">(</span><span class="n">sku</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">qty</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">attrs</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Address</span><span class="o">(</span><span class="n">street</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">zip</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderOut</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderContract</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>        <span class="c1">// List vs Seq - both are sequences
</span></span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>      <span class="c1">// Nested case class
</span></span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Nil</span>       <span class="c1">// Set vs Seq, has default
</span></span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">evDeepOk</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">OrderOut</span>, <span class="kt">OrderContract</span>, <span class="kt">SchemaPolicy.Backward.</span><span class="k">type</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening here:</strong></p>
<ol>
<li>
<p><strong>Nested case classes</strong> - <code>Address</code> inside <code>OrderOut</code>. The macro recurses: when it sees <code>Option[Address]</code>, it unwraps to <code>Address</code>, then checks if <code>Address</code> is a case class, and recurses again to get its fields.</p>
</li>
<li>
<p><strong>Collections</strong> - <code>List[LineItem]</code> vs <code>Seq[LineItem]</code>. Both match because <code>seqArg</code> treats them equivalently. The macro sees &ldquo;sequence of LineItem&rdquo; and recurses on <code>LineItem</code>.</p>
</li>
<li>
<p><strong>Maps</strong> - <code>Map[String, String]</code> in <code>attrs</code>. The macro checks key type (must be atomic), then recurses on value type.</p>
</li>
<li>
<p><strong>Default values</strong> - <code>tags: Seq[String] = Nil</code> has a default. Under Backward policy, if <code>OrderOut</code> is missing <code>tags</code>, it&rsquo;s OK because the contract has a default.</p>
</li>
</ol>
<p>This is why the TypeInspector pattern matters - it handles arbitrary nesting without special cases.</p>
<hr>
<h2 id="part-10---policy-modifiers-in-two-minutes-ordered-ci-byposition">Part 10 - Policy modifiers in two minutes (Ordered, CI, By‑Position)</h2>
<p>Sometimes you need stricter comparison. Or different rules. Here are three quick policy extensions:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/PolicyMods.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">PolicyMod</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">ExactOrdered</span>   <span class="kt">extends</span> <span class="kt">PolicyMod</span> <span class="c1">// names + order + types must match
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactCI</span>        <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// case‑insensitive field names
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// ignore names; types by index
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactOrdered</span>    <span class="k">extends</span> <span class="nc">ExactOrdered</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactCI</span>         <span class="k">extends</span> <span class="nc">ExactCI</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">ExactByPosition</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">ConformsMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">import</span> <span class="kt">scala.quoted.*</span>
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span> <span class="k">&lt;:</span> <span class="kt">PolicyMod</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">M:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">fields</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">))</span> <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mismatches</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">M</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">n</span> <span class="k">=</span> <span class="o">(</span><span class="n">s</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">toLowerCase</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">om</span> <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">cm</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">keySet</span> <span class="n">union</span> <span class="n">om</span><span class="o">.</span><span class="n">keySet</span><span class="o">).</span><span class="n">toList</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span> <span class="n">k</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">          <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">),</span> <span class="n">om</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">))</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="n">t1</span><span class="o">),</span> <span class="nc">Some</span><span class="o">(</span><span class="n">t2</span><span class="o">))</span> <span class="k">if</span> <span class="n">t1</span> <span class="o">!=</span> <span class="n">t2</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;</span><span class="si">$k</span><span class="s"> expected </span><span class="si">$t1</span><span class="s">, found </span><span class="si">$t2</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">),</span> <span class="nc">None</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;missing </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">None</span><span class="o">,</span> <span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">))</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;extra </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="k">_</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="k">_</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">!=</span> <span class="n">tC</span> <span class="k">=&gt;</span> <span class="s">s&#34;@</span><span class="si">$i</span><span class="s"> expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactOrdered</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="n">nO</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="n">nC</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">nameOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">nO</span> <span class="o">==</span> <span class="n">nC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(name) expected </span><span class="si">$nC</span><span class="s">, found </span><span class="si">$nO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">typeOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">==</span> <span class="n">tC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(type) expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="n">nameOk</span> <span class="o">++</span> <span class="n">typeOk</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;Unknown policy mod&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">mismatches</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Modifier drift: </span><span class="si">${</span><span class="n">mismatches</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="o">()</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Try these:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">com.example.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">A</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">B</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span>, <span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="c1">// ✅ same types by index
</span></span></span><span class="line"><span class="cl"><span class="c1">// ConformsMod[A, B, PolicyMod.ExactOrdered] // ❌ order matters → name mismatch
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIa</span><span class="o">(</span><span class="nc">ID</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIb</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">CIa</span>, <span class="kt">CIb</span>, <span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="o">//</span> <span class="o">✅</span> <span class="k">case</span><span class="err">‑</span><span class="n">insensitive</span> <span class="n">names</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Why these three? Because real systems need them:</p>
<ul>
<li><strong>ExactOrdered</strong>: For formats where position matters (some CSV parsers, protobuf with field numbers)</li>
<li><strong>ExactCI</strong>: Because someone always uses <code>UserId</code> when the contract says <code>userId</code></li>
<li><strong>ExactByPosition</strong>: For legacy systems that ignore field names entirely</li>
</ul>
<hr>
<h2 id="part-11---phantom-types-building-impossible-to-misuse-apis">Part 11 - Phantom types: Building impossible-to-misuse APIs</h2>
<p>Here&rsquo;s where things get interesting. The code, also uses phantom types, type-indexing / typestate builder patterns to build a pipeline builder that&rsquo;s <strong>impossible to misuse</strong>.
What are phantom types? They&rsquo;re type parameters that don&rsquo;t appear in the class body but control what you can do with an instance. Think of them as compile-time state machines.</p>
<pre class="mermaid">stateDiagram-v2
    [*] --&gt; Empty
    Empty --&gt; WithSource: addSource
    WithSource --&gt; WithTransform: transformAs[Next]
    WithTransform --&gt; Complete: addSink[Contract,P] + conforms
    Complete --&gt; [*]: build

    note right of WithTransform
      Evidence constraints:
      - S &lt;: WithSource to transform
      - S &lt;: WithTransform to sink
      - S = Complete to build
    end note
</pre><p>From the codebase <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Phantom type states
</span></span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Empty</span>         <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithSource</span>    <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithTransform</span> <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Complete</span>      <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">S</span> <span class="k">&lt;:</span> <span class="kt">BuilderState</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="nc">private</span> <span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">steps</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">PipelineStep</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add source when Empty
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSource</span><span class="o">[</span><span class="kt">C</span><span class="o">](</span><span class="n">src</span><span class="k">:</span> <span class="kt">TypedSource</span><span class="o">[</span><span class="kt">C</span><span class="o">])(</span><span class="n">using</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">])</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only transform after adding source
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">f</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span><span class="o">)(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span><span class="o">,</span>  <span class="c1">// This constraint enforces order!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">Next</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add sink after transform
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSink</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">](</span><span class="n">sink</span><span class="k">:</span> <span class="kt">TypedSink</span><span class="o">[</span><span class="kt">R</span><span class="o">])(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev0</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithTransform</span><span class="o">,</span>  <span class="c1">// Must have transform
</span></span></span><span class="line"><span class="cl">    <span class="n">ev1</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">CurContract</span>, <span class="kt">R</span>, <span class="kt">P</span><span class="o">],</span>  <span class="c1">// Compile-time contract check!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">R</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only build when Complete
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">build</span><span class="o">(</span><span class="n">using</span> <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="o">=</span><span class="k">:</span><span class="o">=</span> <span class="nc">Complete</span><span class="o">)</span><span class="k">:</span> <span class="kt">SparkSession</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="o">(</span><span class="n">spark</span><span class="k">:</span> <span class="kt">SparkSession</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PipelineBuilder</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">apply</span><span class="o">[</span><span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="nc">Nil</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="how-this-works">How this works:</h3>
<ol>
<li>
<p><strong>State transitions as types</strong> - Each method returns a new type parameter <code>S</code>. The compiler tracks which state you&rsquo;re in.</p>
</li>
<li>
<p><strong>Evidence constraints</strong> - <code>ev: S &lt;:&lt; WithSource</code> means &ldquo;S must be a subtype of WithSource&rdquo;. If it&rsquo;s not, this method doesn&rsquo;t exist.</p>
</li>
<li>
<p><strong>Compile-time state machine</strong> - You literally cannot call methods in the wrong order:</p>
</li>
</ol>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// ✅ This compiles
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;good&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">src</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// ❌ This doesn&#39;t compile - can&#39;t transform before adding source
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;bad&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">ERROR</span><span class="k">:</span> <span class="kt">No</span> <span class="kt">implicit</span> <span class="kt">evidence</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span></span></span></code></pre></td></tr></table>
</div>
</div><ol start="4">
<li><strong>Contract checking embedded</strong> - Notice <code>ev1: SchemaConforms[CurContract, R, P]</code> in <code>addSink</code>? That&rsquo;s where the compile-time contract check happens. The pipeline literally won&rsquo;t build if schemas drift.</li>
</ol>
<p>This pattern is called the <strong>Phantom Type Builder Pattern</strong>. For more details, see:</p>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala (Rhetorical Musings)</a>
</li>
</ul>
<p><strong>Why this matters</strong>: In production data pipelines, the order of operations matters. Read → Transform → Validate → Write. With phantom types, the compiler enforces this order. You can&rsquo;t accidentally write before transforming. You can&rsquo;t transform without a source.</p>
<p>This is the difference between &ldquo;here&rsquo;s how macros work&rdquo; and &ldquo;here&rsquo;s how to build production contract systems.&rdquo;</p>
<hr>
<h2 id="part-12---schema-evolution-and-versioning-handling-change-over-time">Part 12 - Schema evolution and versioning: Handling change over time</h2>
<p>Here&rsquo;s the reality: schemas evolve. Fields get added, deprecated, renamed. Teams roll out changes incrementally. The question isn&rsquo;t whether schemas will change - it&rsquo;s how you manage those changes without breaking production.</p>
<h3 id="the-schema-evolution-problem">The schema evolution problem</h3>
<p><strong>Scenario</strong>: You have <code>CustomerV1</code> running in production. Marketing wants to add a <code>segment</code> field for targeting. But you have 10 pipelines reading <code>CustomerV1</code>. How do you evolve without breaking everything?</p>
<p><strong>Wrong approach</strong>: Add <code>segment</code> to <code>CustomerV1</code>, deploy, hope for the best. Result: Some pipelines break because they don&rsquo;t handle the new field.</p>
<p><strong>Right approach</strong>: Version your contracts and use policies to manage transitions.</p>
<h3 id="versioning-strategy">Versioning strategy</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala - Current production
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala - New version with segment
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Notice <code>segment</code> is:</p>
<ol>
<li><strong>Optional</strong> (<code>Option[String]</code>)</li>
<li><strong>Has a default</strong> (<code>= None</code>)</li>
</ol>
<p>This makes it backward-compatible - old code can work with new data.</p>
<h3 id="migration-phases">Migration phases</h3>
<p><strong>Phase 1: Add the field (Backward policy)</strong></p>
<p>Deploy new producers writing <code>CustomerV2</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">producer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;write-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">rawData</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addSegment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">Policy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Backward allows optional fields
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Old consumers still read <code>CustomerV1</code>. They ignore the <code>segment</code> field. No breakage.</p>
<p><strong>Phase 2: Migrate consumers</strong></p>
<p>Update each consumer one by one:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">consumer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;read-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">source</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">EnrichedCustomer</span><span class="o">](</span><span class="n">useSegment</span><span class="o">)</span>  <span class="c1">// Now uses segment field
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">EnrichedCustomer</span>, <span class="kt">Policy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The compile-time contract ensures: &ldquo;Does this consumer handle the new schema?&rdquo; If you forget to update the case class, it won&rsquo;t compile.</p>
<p><strong>Phase 3: Deprecate V1</strong></p>
<p>Once all pipelines use <code>CustomerV2</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Mark V1 as deprecated
</span></span></span><span class="line"><span class="cl"><span class="nd">@deprecated</span><span class="o">(</span><span class="s">&#34;Use CustomerV2&#34;</span><span class="o">,</span> <span class="s">&#34;2025-10-01&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The compiler warns any remaining V1 usage. After a grace period, delete V1 entirely.</p>
<h3 id="handling-breaking-changes">Handling breaking changes</h3>
<p>What if you need to rename a field? Say, <code>email</code> → <code>emailAddress</code>?</p>
<p><strong>Non-breaking approach</strong>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add the new field, keep the old
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>              <span class="c1">// Old field
</span></span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New field
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, that&rsquo;s duplication. Better:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add alias constructor
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New canonical name
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">CustomerV3</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">fromV2</span><span class="o">(</span><span class="kt">v2:</span> <span class="kt">CustomerV2</span><span class="o">)</span><span class="kt">:</span> <span class="kt">CustomerV3</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">CustomerV3</span><span class="o">(</span><span class="n">v2</span><span class="o">.</span><span class="n">id</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">email</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">segment</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Step 2</strong>: Migrate producers to write <code>emailAddress</code>
<strong>Step 3</strong>: Migrate consumers to read <code>emailAddress</code>
<strong>Step 4</strong>: Remove the <code>fromV2</code> constructor</p>
<p>At each step, compile-time contracts verify: &ldquo;Does this transformation produce the expected schema?&rdquo;</p>
<h3 id="coexistence-during-migration">Coexistence during migration</h3>
<p>During migration, you have both V2 and V3 running. How to handle?</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V3</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Transformation handles both
</span></span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="n">normalize</span><span class="o">(</span><span class="n">schema</span><span class="k">:</span> <span class="kt">CustomerSchema</span><span class="o">)</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=</span> <span class="n">schema</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">V2</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">V3</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">v3</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=&gt;</span> <span class="n">v3</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The contract checks both paths:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V2</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span>  <span class="c1">// V2 → V3 allowed
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V3</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>     <span class="o">//</span> <span class="n">V3</span> <span class="o">→</span> <span class="n">V3</span> <span class="n">exact</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="real-world-migration-backward--exact">Real-world migration: Backward → Exact</h3>
<p>Let&rsquo;s look at a complete migration scenario from the code in github. This is exactly how you&rsquo;d roll out schema changes in production.</p>
<p>From <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="cm">/** Sink contract: the target schema we promise to write. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerContract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Producer: imagine an upstream source adds an extra field `segment`. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerProducer</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Declared &#34;Next&#34; schema after a transform (dropping `segment`). */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerNext</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Step 1: Migration phase - use Backward policy
</span></span></span><span class="line"><span class="cl"><span class="c1">// Producer has extra field, but contract allows it during migration
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">srcB</span> <span class="k">=</span> <span class="nc">TypedSource</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">](</span><span class="s">&#34;csv&#34;</span><span class="o">,</span> <span class="n">inPath</span><span class="o">,</span> <span class="nc">Map</span><span class="o">(</span><span class="s">&#34;header&#34;</span> <span class="o">-&gt;</span> <span class="s">&#34;true&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">sinkB</span> <span class="k">=</span> <span class="nc">TypedSink</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="n">tmpOutB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">dropExtras</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span> <span class="k">_</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="n">$</span><span class="s">&#34;id&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;email&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;age&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planB</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;CSV -&gt; Parquet B: transformAs[CustomerNext], Exact&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">srcB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerNext</span><span class="o">](</span><span class="s">&#34;drop segment&#34;</span><span class="o">)(</span><span class="n">dropExtras</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sinkB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="whats-happening-here">What&rsquo;s happening here:</h3>
<p><strong>Phase 1: Producer adds field</strong></p>
<ul>
<li>Upstream adds <code>segment</code> field to <code>CustomerProducer</code></li>
<li>Our contract is still <code>CustomerContract</code> (no <code>segment</code>)</li>
<li>We use <code>transformAs[CustomerNext]</code> to explicitly drop the extra field</li>
<li>Then check <code>CustomerNext</code> conforms to <code>CustomerContract</code> under <strong>Exact</strong></li>
</ul>
<p><strong>Why this works:</strong></p>
<ol>
<li>The transform explicitly declares the output schema (<code>CustomerNext</code>)</li>
<li>The compiler checks: Does <code>CustomerNext</code> match <code>CustomerContract</code>? Yes (both have id, email, age)</li>
<li>If we later change <code>CustomerNext</code> by accident, compilation fails</li>
</ol>
<p><strong>Phase 2: Stabilization</strong>
Once the migration is done:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Later: Everyone uses CustomerContract, no transform needed
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planStable</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;stable&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">contractSource</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">noTransform</span>  <span class="c1">// Direct pass-through
</span></span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This is real-world stuff. The code shows you exactly how to migrate schemas safely.</p>
<pre class="mermaid">flowchart TD
    P1[&#34;Producer adds field &#39;segment&#39;&#34;] --&gt; T1[&#34;transformAs[CustomerNext]\n(drop extras)&#34;]
    T1 --&gt; V1[&#34;summon Conforms[CustomerNext,CustomerContract,Policy.Exact]&#34;]
    V1 --&gt; W1[&#34;write sink&#34;]

    subgraph Phase 1 [&#34;Migration window&#34;]
      direction TB
      P1 --&gt; T1 --&gt; V1 --&gt; W1
    end

    W1 --&gt; P2[&#34;Stabilize: sources match contract&#34;]
    P2 --&gt; V2[&#34;Conforms[CustomerContract,CustomerContract,Policy.Exact]&#34;]
    V2 --&gt; W2[&#34;noTransform + write&#34;]
</pre><hr>
<h2 id="part-13---testing-compiletime-for-real-copypaste-tests">Part 13 - Testing compile‑time, for real (copy/paste tests)</h2>
<p>How do you test that code fails to compile? With <code>assertDoesNotCompile</code>:</p>
<pre class="mermaid">sequenceDiagram
    participant Dev
    participant CI as &#34;GitHub Actions&#34;
    participant SBT as &#34;sbt testOnly *CompileTimeSpec&#34;
    Dev-&gt;&gt;CI: open PR
    CI-&gt;&gt;SBT: run compile-fail suite
    SBT-&gt;&gt;SBT: assertCompiles / assertDoesNotCompile
    alt all pass
        SBT--&gt;&gt;CI: success
        CI--&gt;&gt;Dev: green check
    else drift detected
        SBT--&gt;&gt;CI: failure with errorAndAbort msg
        CI--&gt;&gt;Dev: red X + compiler diff
    end
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/CompileTimeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">org.scalatest.funsuite.AnyFunSuite</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CompileTimeSpec</span> <span class="k">extends</span> <span class="nc">AnyFunSuite</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">test</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Exact</span> <span class="kt">should</span> <span class="kt">fail</span> <span class="kt">when</span> <span class="kt">a</span> <span class="kt">required</span> <span class="kt">field</span> <span class="kt">is</span> <span class="kt">missing</span><span class="err">&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="kt">assertDoesNotCompile</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span><span class="err">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Backward should pass for the same mismatch&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OutMissing, Contract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;By‑Position should accept re‑ordered names with same types&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactByPosition]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Ordered should reject the same re‑ordering&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactOrdered]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Pipeline builder enforces correct order&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">      PipelineBuilder[Contract](&#34;bad&#34;)
</span></span></span><span class="line"><span class="cl"><span class="s">        .transformAs[Next](identity)  // Can&#39;t transform without source
</span></span></span><span class="line"><span class="cl"><span class="s">    &#34;&#34;&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Nested types conform correctly&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OrderOut, OrderContract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>You can wire this into CI:</p>
<div class="highlight highlight-yaml" data-lang="yaml"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/compile-fail.yml (concept)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">compile-fail</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">pull_request]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cf</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-java@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">distribution</span><span class="p">:</span><span class="w"> </span><span class="nt">temurin, java-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;17&#39;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Compile‑fail suite</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">sbt -v &#34;testOnly *CompileTimeSpec&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This is gold. Your CI literally proves that broken schemas can&rsquo;t be deployed.</p>
<hr>
<p><del>## Part 14 - Scala 2: how to do the same (and what&rsquo;s different)</del></p>
<p><del>Scala 2 uses def macros with a <code>Context</code> (blackbox/whitebox). The idea is identical, but the implementation uses <code>c.universe</code> trees instead of <code>Expr</code>/quotes.</del></p>
<p><del>Key differences:</del>
<del>- Scala 3: <code>inline</code> + <code>${ ... }</code> splices; Scala 2: <code>macro def</code> with a <code>Context</code></del>
<del>- Error reporting: <code>report.errorAndAbort</code> (Scala 3) vs <code>c.abort</code> (Scala 2)</del>
<del>- Trees: <code>Expr[T]</code> (Scala 3) vs raw <code>Tree</code> (Scala 2)</del>
<del>- Type inspection: <code>TypeRepr</code> (Scala 3) vs <code>Type</code> (Scala 2)</del></p>
<p><del>Scala 3 macros are safer. The quoted API prevents many common macro bugs. But Scala 2 macros work fine if you&rsquo;re careful.</del></p>
<p><del>For derivation, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works across both versions.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ Removed: Part 14 - Scala 2</div>
  <div class="admonition-content">
    Jobless/duplicate for detail-job objective.
  </div>
</div>
<hr>
<h2 id="part-15---integration-with-spark-runtime-defense-in-depth">Part 15 - Integration with Spark: Runtime defense-in-depth</h2>
<p>The code in github also shows how to mirror compile-time policies with Spark&rsquo;s built-in structural comparators for runtime validation.</p>
<p>From <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Derive Spark StructType from case class at compile time
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">struct</span><span class="k">:</span> <span class="kt">StructType</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">SparkSchema</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">derived</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="k">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Macro that converts case class → StructType
</span></span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]]</span> <span class="k">=</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Runtime policy mapping to Spark comparators
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyRuntime</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsIgnoreCaseAndNullability</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactByPosition.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurally</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="n">ignoreNullability</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactOrdered.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurallyByName</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="k">_</span> <span class="o">==</span> <span class="k">_</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Two-layer validation strategy</strong>:</p>
<ol>
<li><strong>Compile-time</strong> (macros) - Catches schema drift before deployment</li>
<li><strong>Runtime</strong> (Spark comparators) - Defensive check for external data sources</li>
</ol>
<pre class="mermaid">flowchart TD
    F[&#34;found: StructType&#34;] --- E[&#34;expected: StructType&#34;]
    F --&gt; EX[&#34;Policy.Exact&#34;] --&gt; C1[&#34;equalsIgnoreCaseAndNullability(found,expected)&#34;]
    F --&gt; EP[&#34;Policy.ExactByPosition&#34;] --&gt; C2[&#34;equalsStructurally(found,expected,true)&#34;]
    F --&gt; EO[&#34;Policy.ExactOrdered&#34;] --&gt; C3[&#34;equalsStructurallyByName(found,expected,resolver)&#34;]
</pre><p>For more on Spark&rsquo;s structural comparators, see the <a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">DataType API docs</a>
.</p>
<hr>
<h2 id="part-16---a-tiny-migration-playbook-you-can-adopt-tomorrow">Part 16 - A tiny migration playbook you can adopt tomorrow</h2>
<p>Here&rsquo;s how I use these policies in practice:</p>
<ul>
<li><strong>Critical paths → Exact</strong>. No surprises. If schemas don&rsquo;t match exactly, builds fail.</li>
<li><strong>Producer adding optional fields → Backward</strong> during rollout. Allows new optional fields on the producer side.</li>
<li><strong>Consumer that can ignore extras → Forward</strong> during rollout. Lets the consumer tolerate extra fields.</li>
<li><strong>After stabilization → tighten back to Exact</strong>.</li>
</ul>
<p>And remember:</p>
<ul>
<li>Keep transformations pure (no side effects inside map/filter)</li>
<li>Place I/O at edges (read once, write once)</li>
<li>Make side-effects idempotent (so retries don&rsquo;t duplicate writes)</li>
</ul>
<blockquote>
  <p>Compile-time prevents drift; runtime manages reality. Tests catch behavior, not shapes.</p>

</blockquote><hr>
<h2 id="part-17---production-patterns-what-i-learned-shipping-this">Part 17 - Production patterns: What I learned shipping this</h2>
<h3 id="pattern-1-organize-contracts-by-version">Pattern 1: Organize contracts by version</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// pipelines/CustomerMigration.scala
</span></span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">contracts._</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">migration</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;v1-to-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV1</span><span class="o">](</span><span class="n">sourceV1</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addNameField</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sinkV2</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-2-use-companion-objects-for-schema-caching">Pattern 2: Use companion objects for schema caching</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">User</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>  <span class="o">//</span> <span class="nc">Compute</span> <span class="n">once</span><span class="o">,</span> <span class="n">reuse</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-3-document-policy-choices-inline">Pattern 3: Document policy choices inline</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Good: Explicit reasoning
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Allow optional fields during Q2 migration
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Bad: No context
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Full</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">Why</span> <span class="nc">Full</span><span class="o">?</span> <span class="nc">When</span> <span class="n">will</span> <span class="n">we</span> <span class="n">tighten</span><span class="o">?</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-4-test-your-compile-fail-cases">Pattern 4: Test your compile-fail cases</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Keep these in version control as documentation
</span></span></span><span class="line"><span class="cl"><span class="n">test</span><span class="o">(</span><span class="s">&#34;CustomerV1 should not conform to CustomerV2 under Exact&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;summon[Conforms[CustomerV1, CustomerV2, Policy.Exact]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><hr>
<h2 id="appendix---full-demo-project-skeleton-copy--run">Appendix - Full demo project skeleton (copy &amp; run)</h2>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">├── build.sbt
</span></span><span class="line"><span class="cl">└── src
</span></span><span class="line"><span class="cl">    ├── main
</span></span><span class="line"><span class="cl">    │   └── scala
</span></span><span class="line"><span class="cl">    │       └── com/example/
</span></span><span class="line"><span class="cl">    │           ├── IntroMacro.scala
</span></span><span class="line"><span class="cl">    │           ├── Shape.scala
</span></span><span class="line"><span class="cl">    │           ├── Policy.scala
</span></span><span class="line"><span class="cl">    │           ├── Conforms.scala
</span></span><span class="line"><span class="cl">    │           └── SparkCore.scala
</span></span><span class="line"><span class="cl">    └── test
</span></span><span class="line"><span class="cl">        └── scala
</span></span><span class="line"><span class="cl">            └── com/example/
</span></span><span class="line"><span class="cl">                ├── ShapeSpec.scala
</span></span><span class="line"><span class="cl">                ├── ConformsSpec.scala
</span></span><span class="line"><span class="cl">                └── CompileTimeSpec.scala</span></span></code></pre></td></tr></table>
</div>
</div><p><strong>build.sbt (Scala 3)</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">scalaVersion</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;3.3.3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organization</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;com.example&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.apache.spark&#34;</span> <span class="o">%%</span> <span class="s">&#34;spark-sql&#34;</span> <span class="o">%</span> <span class="s">&#34;3.5.1&#34;</span> <span class="o">%</span> <span class="s">&#34;provided&#34;</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.scalatest&#34;</span> <span class="o">%%</span> <span class="s">&#34;scalatest&#34;</span> <span class="o">%</span> <span class="s">&#34;3.2.17&#34;</span> <span class="o">%</span> <span class="nc">Test</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Run it:</p>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><blockquote>
  <p>Tip: For Scala 2, add a sibling module with scala‑reflect and copy the S2 file from above.</p>

</blockquote><hr>
<h2 id="references">References</h2>
<h3 id="scala-3-macros--metaprogramming">Scala 3 Macros &amp; Metaprogramming</h3>
<ul>
<li><a href="https://rockthejvm.com/courses/scala-macros-and-metaprogramming" rel="noopener external" target="_blank">Rock the JVM - Scala Macros and Metaprogramming (course)</a>
</li>
<li><a href="https://rockthejvm.com/articles/scala-3-macros-comprehensive-guide" rel="noopener external" target="_blank">Rock the JVM - Scala 3 Macros comprehensive guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 Macros - Official overview</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/reflection.html" rel="noopener external" target="_blank">Scala 3 Reflection guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">Scala 3 Macro best practices</a>
</li>
</ul>
<h3 id="apache-spark-integration">Apache Spark Integration</h3>
<ul>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">Spark DataType structural comparators</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/DatasetHolder.html" rel="noopener external" target="_blank">Spark Dataset implicits (toDF/toDS)</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/sql-data-sources-csv.html" rel="noopener external" target="_blank">Spark CSV data source</a>
</li>
<li><a href="https://index.scala-lang.org/vincenzobaz/spark-scala3-encoders" rel="noopener external" target="_blank">Scala 3 encoders for Spark (community)</a>
</li>
</ul>
<h3 id="phantom-types--type-level-programming">Phantom Types &amp; Type-Level Programming</h3>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala - Builder Pattern (Rhetorical Musings)</a>
</li>
</ul>
<h3 id="other-resources">Other Resources</h3>
<ul>
<li><a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia - Generic derivation library</a>
</li>
</ul>
<hr>
<h2 id="what-compile-time-contracts-cant-catch">What compile-time contracts can&rsquo;t catch</h2>
<p>Let&rsquo;s be honest about the boundaries. Compile-time contracts handle a lot, but they&rsquo;re not magic. Here&rsquo;s what they <strong>can&rsquo;t</strong> do:</p>
<table>
	<thead>
			<tr>
					<th>Category</th>
					<th>What It Can&rsquo;t Catch</th>
					<th>Why</th>
					<th>Solution</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><strong>External data drift</strong></td>
					<td>• Third-party APIs that change without telling you; • Kafka topics from teams you don&rsquo;t coordinate with; • S3 buckets written by external vendors</td>
					<td>You don&rsquo;t control their build process</td>
					<td>Use runtime validation (schema registries, data contracts)</td>
			</tr>
			<tr>
					<td><strong>Late-arriving schema changes</strong></td>
					<td>• Schema registry updated after your deploy; • Database columns added while your job runs</td>
					<td>Compile-time happens before deploy</td>
					<td>Runtime checks with version monitoring</td>
			</tr>
			<tr>
					<td><strong>Data quality issues</strong></td>
					<td>• Nulls where you expect values (even if field is non-optional); • Out-of-range numbers (age = -5); • Malformed strings (email without @)</td>
					<td>Compile-time checks structure, not content</td>
					<td>Great Expectations, Deequ, or custom validators</td>
			</tr>
			<tr>
					<td><strong>Non-breaking additions</strong></td>
					<td>• Upstream adds optional field you don&rsquo;t use yet</td>
					<td>This is often fine! <code>Forward</code> policy handles it</td>
					<td>Policy-based awareness or stricter monitoring</td>
			</tr>
			<tr>
					<td><strong>Partial batch failures</strong></td>
					<td>• 1000 records match schema, 5 don&rsquo;t</td>
					<td>Compile-time is binary (compiles or doesn&rsquo;t)</td>
					<td>Runtime validation with error tables/quarantine</td>
			</tr>
	</tbody>
</table>
<p><strong>The takeaway</strong>: Compile-time contracts catch drift <strong>within your controlled codebase</strong> during development and deployment. For everything else - external sources, runtime changes, data quality - layer in runtime validation. Think defense-in-depth: compile-time as the first gate, runtime as the safety net.</p>
<hr>
<p><strong>TL;DR</strong></p>
<ul>
<li>Compile-time evidence + policy types make schema intent explicit and enforceable</li>
<li>TypeInspector pattern organizes type inspection into composable utilities</li>
<li>Phantom types with type-indexing/type-state builders create impossible-to-misuse APIs through compile-time state machines</li>
<li>Field vs element optionality matters for nested types</li>
<li>Nested structures (Maps, Lists, case classes) work through recursive shape building</li>
<li>Real migrations: use Backward during rollout, tighten to Exact after stabilization</li>
<li>Spark&rsquo;s comparators provide runtime defense-in-depth</li>
<li>Schema evolution needs versioning, coexistence strategies, and policy-based transitions</li>
<li>Developer ergonomics matter: clear errors, fast compile times, easy onboarding</li>
<li>Compile-time doesn&rsquo;t replace runtime validation - it complements it for controlled systems</li>
<li>This catches many drift classes early, but you still need runtime checks for external data</li>
<li>Compile-time contracts don&rsquo;t make pipelines unbreakable - they make breakage predictable</li>
</ul>
]]></content:encoded><enclosure url="https://vitthalmirji.com/images/og/2025-09-compile-time-data-contracts.png" type="image/png"/></item><item><title>DET week 1 homework 1: redlining compile-time data contracts for clarity wins</title><link>https://vitthalmirji.com/2025/11/det-week-1-homework-1-redlining-compile-time-data-contracts-for-clarity-wins/</link><guid isPermaLink="true">https://vitthalmirji.com/2025/11/det-week-1-homework-1-redlining-compile-time-data-contracts-for-clarity-wins/</guid><pubDate>Sun, 30 Nov 2025 00:00:00 +0000</pubDate><dc:creator>vitthal-mirji</dc:creator><category>technical-writing</category><category>det-workshop</category><category>writing-objectives</category><category>redlining</category><category>scala</category><category>scala-3</category><category>compile-time</category><category>data-contracts</category><category>metaprogramming</category><category>yaakov-bressler</category><category>homework</category><category>workshop-assignment</category><category>week-1</category><category>teach-to-resource</category><category>objectives-framework</category><description>Week 1 homework that transforms a Teaching to Resource objective through visual redlining. Shows every line removed with reasons, the logic behind the cut, and how a 70% reduction improves clarity without losing meaning for readers.</description><content:encoded><![CDATA[<h2 id="workshop-context">Workshop context</h2>
<p>This is week 1 homework for the <a href="https://dataengineerthings.org/posts/tech-writing-workshop/" rel="noopener external" target="_blank">DET (Data Engineer Things) Technical writing workshop</a>
 led by <a href="https://yaakovbressler.com/" rel="noopener external" target="_blank">Yaakov Bressler</a>
.</p>
<p><strong>Workshop site</strong>: <a href="https://www.dataengineerthings.org/index" rel="noopener external" target="_blank">dataengineerthings.org</a>
</p>
<h3 id="workshop-mental-model">Workshop mental model</h3>
<p>The DET workshop uses a <a href="https://dataengineerthings.org/posts/objectives-framework/" rel="noopener external" target="_blank">7 Writer Objectives × 6 Reader Objectives matrix</a>
 to align writer intent with reader needs.</p>
<pre class="mermaid">%%{init: {&#39;theme&#39;: &#39;base&#39;, &#39;themeVariables&#39;: { &#39;fontSize&#39;: &#39;14px&#39;}}}%%
flowchart TD
    W[&#34;Writer Objectives(7)&#34;] --&gt; M[&#34;Objectives Matrix 7×6 = 42 alignments&#34;]
    R[&#34;Reader Objectives(6)&#34;] --&gt; M
    M --&gt; A[&#34;Alignment Find best match&#34;]
    A --&gt; T[&#34;Transform Adjust content&#34;]

    W --&gt; EX[&#34;Examples:• Teach | Resource | Convince&#34;]
    R --&gt; EX2[&#34;Examples:• Learn | Reference | Decide&#34;]

    style M fill:#e1f5ff,stroke:#0066cc,color:#000
    style A fill:#d4edda,stroke:#28a745,color:#000
    style T fill:#fff3cd,stroke:#ffc107,color:#000
</pre><h3 id="assignment-context">Assignment context</h3>
<p><strong>Homework 1</strong>: Redline the compile-time data contracts article (<a href="/compile-time-data-contracts">original here</a>
) to show transformation from <strong>Teach</strong> → <strong>Resource</strong> objective.</p>
<h3 id="reviewer-instructions">Reviewer instructions</h3>
<p>This redlined document uses:</p>
<ul>
<li><del>Strike-through</del> for content to REMOVE</li>
<li>Admonition boxes explaining WHY each decision</li>
<li>Plain text for content to KEEP</li>
<li>Visual emphasis on the 70% reduction when objectives align</li>
</ul>
<p>The goal: demonstrate objectives-driven editing through transparent redlining.</p>
<hr>
<h2 id="original-vs-target-objectives">Original vs target objectives</h2>
<h3 id="original-article-objective-teach">Original article objective: Teach</h3>
<p><strong>Writer intent</strong>: Build understanding from scratch</p>
<p><strong>Reader need</strong>: Learn compile-time contracts conceptually</p>
<p><strong>Content structure</strong>:</p>
<ul>
<li>Personal stories - &ldquo;Once, an upstream team silently renamed..&rdquo;</li>
<li>Mental models - &ldquo;Here&rsquo;s how to think about macros..&rdquo;</li>
<li>Step-by-step walkthroughs - &ldquo;Let&rsquo;s break this down line by line..&rdquo;</li>
<li>Philosophy and motivation - &ldquo;Why this matters&rdquo;</li>
</ul>
<h3 id="target-objective-resource">Target objective: Resource</h3>
<p><strong>Writer intent</strong>: Provide quick reference for experts</p>
<p><strong>Reader need</strong>: Look up API, patterns, gotchas</p>
<p><strong>Content structure</strong>:</p>
<ul>
<li>Prerequisites (what you need to know first)</li>
<li>Quick start (runnable code immediately)</li>
<li>API reference (shapes, policies, macros)</li>
<li>Production patterns (what works in the real world)</li>
<li>Gotchas (common mistakes)</li>
</ul>
<h3 id="transformation-strategy">Transformation strategy</h3>
<p>This redlining systematically removes Teaching content (stories, walkthroughs, philosophy) while preserving Resource content (code, patterns, references, gotchas).</p>
<p>Expected reduction: ~70% (from 1,751 lines to ~500-600 lines of Resource content + homework annotations).</p>
<hr>
<h2 id="redlining-method">Redlining method</h2>
<p>Using Talmudic commentary style:</p>
<ol>
<li><strong>Circle</strong> (keep): Code examples, production patterns, gotchas, references</li>
<li><strong>Cross</strong> (remove): Stories, mental models, step-by-step explanations, &ldquo;What&rsquo;s happening&rdquo; sections</li>
<li><strong>Annotate</strong> (explain): WHY each decision using admonition boxes</li>
</ol>
<p>EVERY line from the original is preserved below - either struck-through with explanation or kept with explanation.</p>
<hr>
<h2 id="complete-original-article-redlined">Complete original article (redlined)</h2>
<p><del>What if broken pipelines never launched - because the compiler stopped them?</del></p>
<p><del>&gt; &ldquo;If it compiles, contracts align.&quot;</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE opening hook</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: The philosophical opening (&ldquo;What if&hellip;&rdquo;) is Teaching tone. Resource readers already know the problem (it&rsquo;s in their search query: &ldquo;Scala compile-time data contracts&rdquo;). They need the solution immediately, not motivation.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Built curiosity for learners</li>
<li>Positioned problem as worth solving</li>
<li>Set philosophical tone</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource objective = utility, not persuasion</li>
<li>Experts don&rsquo;t need to be sold on the approach</li>
<li>Opening should be Prerequisites + Quick Start instead</li>
</ul>

  </div>
</div>
<p><del>I&rsquo;ve been working on this problem for a while now. You know how it goes - you&rsquo;re running a data pipeline, everything&rsquo;s fine, then someone changes a field name upstream and boom, your job crashes at 2 AM. The on-call engineer (probably you) gets paged, and you spend the next hour figuring out what went wrong.</del></p>
<p><del>Once, an upstream team silently renamed a column <code>amount</code> → <code>amt</code>. Our pipeline didn&rsquo;t crash immediately - it happily wrote nulls for weeks. By the time we caught it, we had to backfill millions of records. That&rsquo;s the day I realized: runtime validation catches problems late; compile-time contracts catch them earlier.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE personal story (2 AM incident)</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Resource readers already had their 2 AM moment. That&rsquo;s why they&rsquo;re searching for this solution. Personal stories build empathy for learners, not experts.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Built empathy with reader</li>
<li>Explained problem in relatable terms</li>
<li>Motivated the solution</li>
<li>Established authority (&ldquo;I&rsquo;ve faced this&rdquo;)</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Experts don&rsquo;t need motivation, they need the solution</li>
<li>They already understand the problem (it&rsquo;s implicit in their search)</li>
<li>Resource objective = utility, not storytelling</li>
<li>Takes 150 words to say what experts already know</li>
</ul>
<p><strong>Word count removed</strong>: ~150 words</p>

  </div>
</div>
<p><del>This post shows you how to catch these issues at compile time. Not at runtime. Not even at test time. At compile time. Your code literally won&rsquo;t build if schemas drift.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE promise paragraph</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: The promise (&ldquo;this post shows you how&rdquo;) is implicit in a Resource article. Title already states what it is: &ldquo;Compile-time data contracts in Scala 3.&rdquo;</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Set expectations for tutorial journey</li>
<li>Built anticipation</li>
<li>Positioned value proposition</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers know what they want from the title</li>
<li>Don&rsquo;t tell them what you&rsquo;ll teach them - just give them the API</li>
<li>&ldquo;Shows you how&rdquo; implies tutorial, Resource provides reference</li>
</ul>
<p><strong>Word count removed</strong>: ~35 words</p>

  </div>
</div>
<p><del>We&rsquo;ll build this from scratch - first understand how Scala 3 macros work, then write a minimal contract system that proves schema compatibility before your code even runs. I&rsquo;ll also show the Scala 2 version because, let&rsquo;s be honest, many teams are still on Scala 2.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE build plan</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;We&rsquo;ll build this from scratch&rdquo; is the Teaching objective speaking. Resource doesn&rsquo;t build, it provides finished components.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Outlined tutorial structure</li>
<li>Set learning journey expectations</li>
<li>Promised comprehensive coverage</li>
<li>Built progressive understanding</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers want the answer, not the journey</li>
<li>&ldquo;Build from scratch&rdquo; implies tutorial</li>
<li>Resource = here&rsquo;s the thing, here&rsquo;s the code, done</li>
<li>Teaching progression (understand → write → prove) doesn&rsquo;t apply</li>
</ul>
<p><strong>Word count removed</strong>: ~45 words</p>

  </div>
</div>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP github link (but enhance)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the github reference but move it to Quick Start section.</p>
<p><strong>Why it fits Resource</strong>: Resource readers want to clone and run immediately. Working code repository is pure utility.</p>
<p><strong>Enhancement needed</strong>: Make it more prominent in a Quick Start section with actual commands instead of buried in opening paragraphs.</p>
<p><strong>Original text preserved below for reviewer</strong>:</p>

  </div>
</div>
<p>All code here runs in a vanilla sbt project. You can grab the working code in github from <code>https://github.com/vim89/compile-time-data-contracts</code> and run it yourself.</p>
<p><del>## Why this matters</del></p>
<p><del>Look, I get it. Writing tests is important. But tests are samples. They check what you think might break. The compiler, though? It&rsquo;s exhaustive for structure. It can prove your schemas match - every field, every type, every time.</del></p>
<p><del>Here&rsquo;s the thing - schemas change all the time in real projects. And when they do:</del>
<del>- Your build should fail early, not your production job</del>
<del>- Keep transformations pure and effects at the edges (so retries don&rsquo;t duplicate writes)</del>
<del>- Let the compiler do the heavy lifting</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE &#39;Why this matters&#39; section</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Philosophy section. Resource readers already know why it matters (it&rsquo;s in their search query).</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Motivated learners who might be skeptical</li>
<li>Explained value proposition (&ldquo;tests vs compiler&rdquo;)</li>
<li>Built case for approach</li>
<li>Positioned as best practice</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource objective = utility, not persuasion</li>
<li>Experts don&rsquo;t need to be sold</li>
<li>&ldquo;Why&rdquo; is for Teaching, &ldquo;How&rdquo; is for Resource</li>
<li>Takes 110 words to explain what&rsquo;s implicit in the title</li>
</ul>
<p><strong>Word count removed</strong>: ~110 words</p>

  </div>
</div>
<p><strong>Important upfront:</strong> This focuses on preventing schema drift within your controlled boundaries - the systems you build, deploy, and version. It doesn&rsquo;t replace runtime validation for external data sources or third-party APIs. Think of it as catching a large class of schema mismatches during development and deployment, not eliminating all drift everywhere. You&rsquo;ll still need runtime checks for data coming from systems you don&rsquo;t control.</p>
<div class="admonition warning">
  <div class="admonition-title">Scope boundary</div>
  <div class="admonition-content">
    Compile-time contracts catch drift in your controlled codebase - the transforms you write, the case classes you define, the schemas you version together. They won&rsquo;t catch upstream API changes, late-arriving schema registry updates, or external data sources that change independently. Use compile-time for what you control, runtime validation for everything else. Defense in depth, not either/or.
  </div>
</div>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP scope boundary warning</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep both the paragraph and the admonition.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Honest about limitations (what it CAN&rsquo;T do)</li>
<li>Sets realistic expectations</li>
<li>Resource readers appreciate knowing boundaries</li>
<li>Prevents misuse (trying to use it for external APIs)</li>
<li>Saves reader time (don&rsquo;t try to use it for X)</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>Not selling the approach, warning about boundaries</li>
<li>Practical, pragmatic</li>
<li>Gotcha/caveat content fits Resource objective</li>
</ul>
<p><strong>Word count kept</strong>: ~100 words</p>

  </div>
</div>
<p><del>## What we&rsquo;ll build</del></p>
<p><del>We&rsquo;re going to build a small compile-time validation system:</del>
<del>1. A way to describe case class structure at compile time (we&rsquo;ll call it &ldquo;shape&rdquo;)</del>
<del>2. A policy system - Exact, Backward, Forward compatibility modes</del>
<del>3. A macro that proves &ldquo;these two shapes conform under policy P&rdquo; - or refuses to compile</del>
<del>4. A phantom-typed pipeline builder that enforces correct construction order</del></p>
<p><del>First, Scala 3 (using inline + quotes), then Scala 2 (using blackbox macros). I&rsquo;ll point out the differences as we go.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE &#39;What we&#39;ll build&#39; section</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Tutorial roadmap. Resource readers don&rsquo;t need a learning journey, they need the API reference immediately.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Set expectations for progressive learning</li>
<li>Outlined tutorial structure</li>
<li>Built anticipation for upcoming sections</li>
<li>Promised comprehensive coverage</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>&ldquo;We&rsquo;re going to build&rdquo; is Teaching language</li>
<li>Resource doesn&rsquo;t build, it provides finished components</li>
<li>Table of contents serves this function better</li>
<li>Numbered list implies sequential learning path</li>
</ul>
<p><strong>Word count removed</strong>: ~80 words</p>

  </div>
</div>
<p><del>### Run the code in github</del></p>
<details class="details">
  <summary>❌ REMOVED: Github clone commands (click to see what was removed)</summary>
  <div class="details__body"><div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone https://github.com/vim89/compile-time-data-contracts.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> compile-time-data-contracts
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Scala 3</span>
</span></span><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div></div>
</details>

<details class="details">
  <summary>❌ REMOVED: Info admonition about compile-fail examples (click to see what was removed)</summary>
  <div class="details__body">The code in github includes working examples you can run immediately. All compile-fail cases are documented inline - uncomment them to see the compiler errors.</div>
</details>

<div class="admonition success">
  <div class="admonition-title">⭕ KEEP github quick start (but move)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep this content but move it earlier to a dedicated Quick Start section.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Immediate value (30 seconds to running code)</li>
<li>No preamble, just commands</li>
<li>Working examples = pure utility</li>
</ul>
<p><strong>Transformation needed</strong>:</p>
<ul>
<li>Move to top of article (after Prerequisites)</li>
<li>Add Prerequisites section first</li>
<li>Expand with basic usage example (success + failure case)</li>
</ul>
<p><strong>Original text preserved here for reviewer, will be reorganized in Resource version</strong>:</p>

  </div>
</div>
<hr>
<p><del>## Part 1 - Scala 3: the mental model (inline + quotes)</del></p>
<p><del>Scala 3 macros are different from Scala 2. Better, actually. They&rsquo;re safer and easier to reason about.</del></p>
<p><del>Two main ideas:</del>
<del>- <code>inline def</code>: tells the compiler &ldquo;expand this at compile time&rdquo;</del>
<del>- <code>quotes</code> + <code>Expr[T]</code>: gives you a typed AST that you can inspect</del></p>
<details class="details">
  <summary>❌ REMOVED: Macro execution sequence diagram (click to see what was removed)</summary>
  <div class="details__body"><pre class="mermaid">sequenceDiagram
    actor Dev
    participant Compiler
    participant Macro as &#34;Conforms.impl&#34;
    participant Reflect as &#34;TypeRepr/quotes&#34;
    Dev-&gt;&gt;Compiler: compile
    Compiler-&gt;&gt;Macro: splice ${ ... }
    Macro-&gt;&gt;Reflect: inspect types
    Reflect--&gt;&gt;Macro: shapes for Out and Contract
    alt conforming
        Macro--&gt;&gt;Compiler: emit Conforms[Out,Contract,P]
        Compiler--&gt;&gt;Dev: build succeeds
    else drift
        Macro--&gt;&gt;Compiler: report.errorAndAbort(msg)
        Compiler--&gt;&gt;Dev: compile error with drift details
    end
</pre></div>
</details>

<p><del>Here&rsquo;s a minimal macro to see how it works:</del></p>
<details class="details">
  <summary>❌ REMOVED: IntroMacro example code (click to see what was removed)</summary>
  <div class="details__body"><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// build.sbt: use Scala 3
</span></span></span><span class="line"><span class="cl"><span class="c1">// scalaVersion := &#34;3.3.3&#34;
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/IntroMacro.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.quoted.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">IntroMacro</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">def</span> <span class="kt">show</span><span class="o">(</span><span class="kt">inline</span> <span class="kt">x:</span> <span class="kt">Any</span><span class="o">)</span><span class="kt">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">showImpl</span><span class="o">(</span>&#39;x<span class="o">)</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">showImpl</span><span class="o">(</span><span class="n">x</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Any</span><span class="o">])(</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">Expr</span><span class="o">(</span><span class="s">s&#34;You passed: </span><span class="si">${</span><span class="n">x</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div></div>
</details>

<p><del>What&rsquo;s happening here?</del>
<del>- The <code>inline</code> method is what users call</del>
<del>- The <code>${ ... }</code> splice calls the implementation</del>
<del>- The implementation gets <code>'x</code> (quoted code) as <code>Expr[Any]</code></del>
<del>- We return <code>Expr[String]</code> - the AST for what the compiler will embed</del></p>
<p><del>Try it:</del></p>
<details class="details">
  <summary>❌ REMOVED: IntroMacro demo usage (click to see what was removed)</summary>
  <div class="details__body"><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Demo.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">run</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">println</span><span class="o">(</span><span class="nc">IntroMacro</span><span class="o">.</span><span class="n">show</span><span class="o">(</span><span class="mi">1</span> <span class="o">+</span> <span class="mi">2</span><span class="o">))</span> <span class="o">//</span> <span class="n">prints</span><span class="k">:</span> <span class="kt">You</span> <span class="kt">passed:</span> <span class="err">1</span><span class="kt">.+</span><span class="o">(</span><span class="err">2</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div></div>
</details>

<p><del>See how it printed the AST of <code>1 + 2</code>? That&rsquo;s the power here - we can inspect code structure at compile time.</del></p>
<p><del>Check out the <a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 macros overview</a>
 and <a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">best practices</a>
 for more details.</del></p>
<details class="details">
  <summary>❌ REMOVED: Tip admonition for macro debugging (click to see what was removed)</summary>
  <div class="details__body">If you&rsquo;re new to Scala 3 macros, start by reading macro-generated code in the REPL: sbt&gt; console, then scala&gt; import scala.quoted.*, then inspect what your macros actually expand to. Understanding the generated code is 80% of debugging macro issues. The compiler&rsquo;s doing the work - you just need to see what it&rsquo;s actually producing.</div>
</details>

<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE entire Part 1 mental model</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Complete Teaching section. &ldquo;Mental model&rdquo; is tutorial content for learners, not reference for experts.</p>
<p><strong>What this served in Teaching version</strong> (~800 words):</p>
<ul>
<li>Introduced <code>inline def</code> concept from scratch</li>
<li>Explained <code>quotes</code> and <code>Expr[T]</code> with examples</li>
<li>Built understanding with minimal <code>IntroMacro.show</code> example</li>
<li>&ldquo;What&rsquo;s happening here?&rdquo; walkthrough (Teaching language)</li>
<li>&ldquo;Try it&rdquo; exercise (Teaching technique)</li>
<li>&ldquo;See how it printed&hellip;?&rdquo; rhetorical question (Teaching engagement)</li>
<li>Sequence diagram showing macro execution flow</li>
<li>Tip admonition for debugging (valuable but assumes learning)</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers already know Scala 3 macros (stated in Prerequisites)</li>
<li>&ldquo;Mental model&rdquo; is for building understanding, Resource provides reference</li>
<li>Teaching progression (introduce → explain → demonstrate → exercise)</li>
<li>If readers don&rsquo;t know macros, Prerequisites already told them to leave</li>
<li>External links to Scala docs handle this better</li>
</ul>
<p><strong>Alternative for Resource version</strong>:</p>
<ul>
<li>Prerequisites section states: &ldquo;Understanding of <code>inline</code>, <code>quotes</code>, <code>Expr[T]</code>, <code>TypeRepr</code>&rdquo;</li>
<li>Links to official Scala 3 docs for those who need it</li>
<li>Jump straight to Shape API</li>
</ul>
<p><strong>Word count removed</strong>: ~800 words</p>

  </div>
</div>
<hr>
<h2 id="part-2---describe-shapes-at-compile-time-scala-3">Part 2 - Describe shapes at compile time (Scala 3)</h2>
<p><del>Now we need to describe case class structure. In real projects, you might use Shapeless, Magnolia, or Scala 3&rsquo;s Mirror. I&rsquo;ll show a compact Mirror-based version here because it&rsquo;s self-contained.</del></p>
<p><del>The idea: a typeclass <code>Shape[A]</code> that knows the fields of <code>A</code>.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE walkthrough introduction</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;Now we need to&hellip;&rdquo; and &ldquo;I&rsquo;ll show&hellip;&rdquo; is Teaching language. Resource version jumps straight to the API code.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Positioned as next step in learning journey</li>
<li>Explained why this section matters</li>
<li>Set context for code to follow</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers scan for Shape API, don&rsquo;t need narrative</li>
<li>&ldquo;We need to&rdquo; implies tutorial progression</li>
<li>Just show the API with minimal comment</li>
</ul>
<p><strong>Alternative for Resource</strong>: &ldquo;Shape API (Mirror-based)&rdquo; as section heading, code immediately follows.</p>
<p><strong>Word count removed</strong>: ~40 words</p>

  </div>
</div>
<pre class="mermaid">classDiagram
    class Field {
      +name: String
      +tpe: String
      +hasDefault: Boolean
      +isOptional: Boolean
    }
    class Shape {
      +fields(): List~Field~
    }
    Shape &#34;1&#34; --&gt; &#34;*&#34; Field : returns
</pre><div class="admonition tip">
  <div class="admonition-title">⭕ KEEP class diagram</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the class diagram.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Visual API reference showing Field and Shape structure</li>
<li>Quick lookup (&ldquo;what fields does Field have?&rdquo;) without reading code</li>
<li>Resource readers appreciate visual summaries they can scan</li>
<li>If not needed, easily skipped</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>Not explaining concepts, just showing structure</li>
<li>In playbook terms: &ldquo;visual reference&rdquo; not &ldquo;teaching aid&rdquo;</li>
<li>Comparable to API documentation diagrams</li>
</ul>
<p><strong>Word count kept</strong>: (diagram)</p>

  </div>
</div>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Shape.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.deriving.Mirror</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">tpe</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">hasDefault</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Shape</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>     <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">A</span><span class="o">]]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span> <span class="c1">// element optionality handled elsewhere
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="n">given</span> <span class="n">derived</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">using</span> <span class="n">m</span><span class="k">:</span> <span class="kt">Mirror.Of</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">m</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">p</span><span class="k">:</span> <span class="kt">Mirror.ProductOf</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">labels</span> <span class="k">=</span> <span class="n">constValueTuple</span><span class="o">[</span><span class="kt">p.MirroredElemLabels</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">types</span>  <span class="k">=</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">p.MirroredElemTypes</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">zipped</span> <span class="k">=</span> <span class="n">zip</span><span class="o">(</span><span class="n">labels</span><span class="o">,</span> <span class="n">types</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">          <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">            <span class="n">zipped</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">,</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span> <span class="k">=</span> <span class="n">tpe</span><span class="o">.</span><span class="n">startsWith</span><span class="o">(</span><span class="s">&#34;Option[&#34;</span><span class="o">))</span> <span class="o">}.</span><span class="n">toList</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">import</span> <span class="nn">scala.compiletime.</span><span class="o">{</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">constValueTuple</span><span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">T</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">inline</span> <span class="n">erasedValue</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">h</span><span class="o">]</span> <span class="k">:</span><span class="kt">:</span> <span class="k">type</span><span class="kt">Names</span><span class="o">[</span><span class="kt">t</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">constValue</span><span class="o">[</span><span class="err">&#34;&#34;</span> <span class="kt">+</span> <span class="kt">T</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">zip</span><span class="o">[</span><span class="kt">L</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span>, <span class="kt">R</span> <span class="k">&lt;:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">labels</span><span class="k">:</span> <span class="kt">L</span><span class="o">,</span> <span class="n">types</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">labels</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">constValue</span><span class="o">[</span><span class="kt">h</span><span class="o">].</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">-&gt;</span> <span class="n">types</span><span class="o">.</span><span class="n">head</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">zip</span><span class="o">[</span><span class="kt">t</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">types</span><span class="o">.</span><span class="n">tail</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition success">
  <div class="admonition-title">⭕ KEEP Shape implementation code</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the complete implementation.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>This IS the API - Field case class, Shape trait, derivation logic</li>
<li>Resource readers need the actual code to use or adapt</li>
<li>Implementation details matter for experts (Mirror.Of, constValueTuple, etc.)</li>
<li>Copy-paste ready for production use</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>No explanation of how it works (just the code)</li>
<li>Comments are minimal, API-level</li>
<li>Expert readers can read the implementation</li>
<li>Teaching version would explain Mirror, constValueTuple, inline match, etc.</li>
</ul>
<p><strong>Word count kept</strong>: ~150 words (code)</p>

  </div>
</div>
<p><del>Yeah, it&rsquo;s a bit dense. But here&rsquo;s what it does:</del>
<del>- Uses Scala 3&rsquo;s Mirror to reflect on case classes at compile time</del>
<del>- Extracts field names and types</del>
<del>- Marks optional fields (those wrapped in <code>Option</code>)</del>
<del>- Returns a <code>List[Field]</code> describing the structure</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE code walkthrough</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;Here&rsquo;s what it does&rdquo; is Teaching language. Resource readers read the code directly.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Explained dense code for learners</li>
<li>Bullet-point summary of functionality</li>
<li>Made code approachable</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Experts read implementation, don&rsquo;t need summary</li>
<li>&ldquo;Yeah, it&rsquo;s a bit dense&rdquo; apologizes to learners</li>
<li>Resource assumes you can read Scala 3 metaprogramming</li>
<li>Inline comments in code serve this better</li>
</ul>
<p><strong>Word count removed</strong>: ~60 words</p>

  </div>
</div>
<details class="details">
  <summary>❌ REMOVED: Shape derivation flow diagram (click to see what was removed)</summary>
  <div class="details__body"><pre class="mermaid">flowchart LR
    A[&#34;Case class A(...)&#34;] --&gt; M[&#34;Mirror.Of[A]&#34;]
    M --&gt; L[&#34;MirroredElemLabels&#34;]
    M --&gt; T[&#34;MirroredElemTypes&#34;]
    L --&gt; Z[&#34;zip(labels, types)&#34;]
    T --&gt; Z
    Z --&gt; S[&#34;Shape[A].fields: List[Field]&#34;]
</pre></div>
</details>

<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE flow diagram</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Teaching aid showing &ldquo;how it works&rdquo; step-by-step. Resource readers understand Mirror already (or they shouldn&rsquo;t be here per Prerequisites).</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Visual walkthrough of derivation process</li>
<li>Showed data flow through Mirror API</li>
<li>Helped learners understand execution</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Explains the process, Resource just gives the tool</li>
<li>If you need this diagram, you&rsquo;re missing prerequisites</li>
<li>Code itself shows the flow for experts</li>
</ul>
<p><strong>Word count removed</strong>: (diagram)</p>

  </div>
</div>
<p><del>Usage:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ShapeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">note</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkShape</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="n">println</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">fields</span><span class="o">)</span> <span class="o">//</span> <span class="nc">List</span><span class="o">(</span><span class="nc">Field</span><span class="o">(</span><span class="s">&#34;id&#34;</span><span class="o">,</span><span class="s">&#34;Long&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;email&#34;</span><span class="o">,</span><span class="s">&#34;String&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;note&#34;</span><span class="o">,</span><span class="s">&#34;Option[String]&#34;</span><span class="o">,</span><span class="kc">false</span><span class="o">,</span><span class="kc">true</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition success">
  <div class="admonition-title">⭕ KEEP usage example</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the usage code.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Shows working example of Shape API</li>
<li>Demonstrates summon syntax</li>
<li>Expected output in comment is helpful reference</li>
<li>Copy-paste ready</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>No &ldquo;Try it&rdquo; language</li>
<li>No explanation, just working code</li>
<li>Shows input → output clearly</li>
</ul>
<p><strong>Word count kept</strong>: ~30 words (code)</p>

  </div>
</div>
<p><del>Now we can ask the compiler &ldquo;what fields does this case class have?&rdquo; at compile time. That&rsquo;s the foundation.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE summary sentence</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;Now we can&hellip;&rdquo; and &ldquo;That&rsquo;s the foundation&rdquo; are Teaching narrative. Resource readers know what the API does from reading it.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Reinforced what was learned</li>
<li>Positioned as building block for next section</li>
<li>Motivational language</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource doesn&rsquo;t narrate what you learned</li>
<li>&ldquo;Foundation&rdquo; implies progressive building (Teaching)</li>
</ul>
<p><strong>Word count removed</strong>: ~25 words</p>

  </div>
</div>
<blockquote>
  <p>If you prefer libraries, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works great for both Scala 2 and 3.</p>

</blockquote><div class="admonition success">
  <div class="admonition-title">⭕ KEEP Magnolia reference</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the library alternative.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Practical alternative for production use</li>
<li>Resource readers appreciate knowing options</li>
<li>External link is pure utility</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>Not explaining Magnolia, just referencing it</li>
<li>Gives expert readers choice</li>
<li>Production-focused (mentions Scala 2 + 3 compat)</li>
</ul>
<p><strong>Word count kept</strong>: ~20 words</p>

  </div>
</div>
<hr>
<h2 id="part-3---policies-exact--backward--forward">Part 3 - Policies: Exact / Backward / Forward</h2>
<p><del>Here&rsquo;s where we define compatibility rules. Think of these as your migration strategies:</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE introductory narrative</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;Here&rsquo;s where we define&hellip;&rdquo; is Teaching setup. Resource readers scan for the API directly.</p>
<p><strong>What this served in Teaching version</strong>:</p>
<ul>
<li>Positioned section in tutorial flow</li>
<li>Set context for learners</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Section heading already describes content</li>
<li>Resource jumps straight to code</li>
</ul>
<p><strong>Word count removed</strong>: ~15 words</p>

  </div>
</div>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Policy.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Policy</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Policy</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">Exact</span>     <span class="kt">extends</span> <span class="kt">Policy</span>
</span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Backward</span>  <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// producer can add optional/defaulted fields
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Forward</span>   <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// consumer tolerates extra fields
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Exact</span>    <span class="k">extends</span> <span class="nc">Exact</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Backward</span> <span class="k">extends</span> <span class="nc">Backward</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Forward</span>  <span class="k">extends</span> <span class="nc">Forward</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition success">
  <div class="admonition-title">⭕ KEEP Policy trait code</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the complete Policy trait implementation.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>This IS the API - core type definitions</li>
<li>Three policy types (Exact, Backward, Forward) are the API surface</li>
<li>Inline comments explain each policy succinctly</li>
<li>Copy-paste ready for production use</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>No walkthrough, just code</li>
<li>Comments are API-level documentation</li>
<li>Expert readers understand sealed trait patterns</li>
</ul>
<p><strong>Word count kept</strong>: ~50 words (code)</p>

  </div>
</div>
<p>Why three policies?</p>
<ul>
<li><strong>Exact</strong>: Schemas must match exactly. No surprises. Use this for critical paths.</li>
<li><strong>Backward</strong>: Producer can add optional fields. Useful when rolling out new features without breaking consumers.</li>
<li><strong>Forward</strong>: Consumer ignores extra fields. Useful when the producer might send more than you need.</li>
</ul>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP policy explanations</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the three-bullet explanation.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Concise API documentation (what each policy does)</li>
<li>Use case guidance (when to use each)</li>
<li>Not tutorial explanation, just reference</li>
<li>Answers &ldquo;which policy should I use?&rdquo; quickly</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>Direct, practical descriptions</li>
<li>Focused on production usage (&ldquo;critical paths&rdquo;, &ldquo;rolling out&rdquo;)</li>
<li>No conceptual building, just what it does</li>
</ul>
<p><strong>Word count kept</strong>: ~60 words</p>

  </div>
</div>
<pre class="mermaid">flowchart LR
    EX[&#34;Policy.Exact&#34;]
    BW[&#34;Policy.Backward&#34;]
    FW[&#34;Policy.Forward&#34;]

    EX --- BW
    EX --- FW

    EX:::tight --&gt;|no diffs allowed| OK1[&#34;✔ match only&#34;]
    BW:::relax --&gt;|allow missing optional in producer\nallow extra fields in producer| OK2[&#34;✔ migration adds&#34;]
    FW:::relax --&gt;|allow extra fields in consumer side| OK3[&#34;✔ tolerant reader&#34;]

    classDef tight fill:#eef,stroke:#446;
    classDef relax fill:#efe,stroke:#484;
</pre><div class="admonition tip">
  <div class="admonition-title">⭕ KEEP policy relationships diagram</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the flowchart showing policy relationships.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Visual reference showing differences between policies</li>
<li>Quick lookup for which policy allows what</li>
<li>Not explaining concepts, showing constraints visually</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>Comparable to API documentation diagrams</li>
<li>Shows rules/constraints at a glance</li>
<li>Practical reference, not educational walkthrough</li>
</ul>
<p><strong>Word count kept</strong>: (diagram)</p>

  </div>
</div>
<p><del>Visual (save/share):</del>

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/policy-lattice.svg"
            alt="Policy lattice"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    A tiny policy lattice - Exact / Backward / Forward.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP policy lattice figure</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the SVG figure reference (but remove &ldquo;Visual (save/share)&rdquo; preamble).</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Alternative visual representation</li>
<li>Resource readers appreciate multiple formats</li>
<li>Figure is reference material, not teaching aid</li>
</ul>
<p><strong>Why remove preamble</strong>:</p>
<ul>
<li>&ldquo;Visual (save/share):&rdquo; is unnecessary narration</li>
<li>Figure speaks for itself</li>
</ul>
<p><strong>Word count kept</strong>: (figure), removed: ~3 words preamble</p>

  </div>
</div>
<p>In practice: start with Exact, relax to Backward/Forward during migrations, then tighten back to Exact once stable.</p>
<div class="admonition success">
  <div class="admonition-title">Migration strategy</div>
  <div class="admonition-content">
    The policy progression that works in production: Exact for stable pipelines, Backward during producer rollouts (allows new optional fields), Forward during consumer updates (tolerates extras), then back to Exact after migration completes. Document the expected timeline - &ldquo;Backward policy through Q2 migration, then tightening to Exact.&rdquo; Prevents policy drift from becoming permanent.
  </div>
</div>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP migration guidance</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep both the paragraph and the admonition.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Production best practices (how to actually use these policies)</li>
<li>Not teaching the concept, sharing real-world strategy</li>
<li>Practical timeline advice (&ldquo;document expected timeline&rdquo;)</li>
<li>Gotcha prevention (&ldquo;Prevents policy drift from becoming permanent&rdquo;)</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>&ldquo;In practice&rdquo; = lessons from production</li>
<li>Specific, actionable guidance</li>
<li>Resource readers need migration strategies</li>
</ul>
<p><strong>Word count kept</strong>: ~80 words</p>

  </div>
</div>
<hr>
<p><del>## Part 4 - How this fits with existing tools</del></p>
<p><del>Before we dive deeper, you might be thinking: &ldquo;We already have Avro, Protobuf, schema registries, and Great Expectations. Why add compile-time contracts?&quot;</del></p>
<p><del>Fair question. Here&rsquo;s how they compare:</del></p>
<p><del><strong>Schema Registry (Confluent, AWS Glue)</strong></del>
<del>- <strong>What it does</strong>: Centralized schema storage with version control</del>
<del>- <strong>When it helps</strong>: Runtime validation across services, schema evolution tracking</del>
<del>- <strong>The gap</strong>: You find mismatches when the job runs, not when you compile</del>
<del>- <strong>Use together</strong>: Schema registry for cross-service contracts; compile-time for intra-repo checks</del></p>
<p><del><strong>Avro / Protobuf</strong></del>
<del>- <strong>What they do</strong>: Binary serialization with embedded schemas</del>
<del>- <strong>When they help</strong>: Network efficiency, strong contracts between services</del>
<del>- <strong>The gap</strong>: Schema compatibility is checked at runtime or via external tooling</del>
<del>- <strong>Use together</strong>: Avro/Protobuf for wire format; compile-time checks to ensure your case classes match those schemas</del></p>
<p><del><strong>Great Expectations / Deequ</strong></del>
<del>- <strong>What they do</strong>: Runtime data quality validation (nulls, ranges, distributions)</del>
<del>- <strong>When they help</strong>: Catching bad data values, statistical anomalies</del>
<del>- <strong>The gap</strong>: They validate data content, not schema structure at compile time</del>
<del>- <strong>Use together</strong>: Compile-time for structure; Great Expectations for data quality</del></p>
<p><del><strong>The compile-time advantage</strong>: Catches drift in your controlled codebase before deployment. If your transform expects field <code>X</code> but the source produces field <code>Y</code>, your build fails - not your production job. Think of it as an additional safety layer for systems you build and version together.</del></p>
<p><del>Real scenario: You have Avro schemas in a registry, but your Scala pipeline reads them into case classes. Compile-time contracts verify those case classes match the expected contract before you deploy. If upstream changes the Avro schema, your build breaks - you don&rsquo;t wait for the job to crash.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE entire Part 4 tool comparison</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Teaching content explaining &ldquo;why this approach&rdquo; by comparing alternatives. Resource readers already decided to use this tool (it&rsquo;s in their search query).</p>
<p><strong>What this served in Teaching version</strong> (~350 words):</p>
<ul>
<li>Addressed skepticism (&ldquo;We already have Avro&hellip;&rdquo;)</li>
<li>Positioned against existing tools (Schema Registry, Avro, Protobuf, Great Expectations)</li>
<li>Explained gaps in each alternative</li>
<li>Built case for adopting this approach</li>
<li>&ldquo;Fair question&rdquo; acknowledges reader doubts (Teaching empathy)</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource objective = utility, not persuasion</li>
<li>Experts don&rsquo;t need to be sold on the approach</li>
<li>&ldquo;Before we dive deeper, you might be thinking&hellip;&rdquo; is Teaching narration</li>
<li>Comparison matrices are for deciding, not using</li>
<li>If reader needs this, they&rsquo;re at wrong article (Prerequisites failed)</li>
</ul>
<p><strong>Alternative for Resource</strong>:</p>
<ul>
<li>Brief Integration section showing &ldquo;Use with Avro&rdquo; code example</li>
<li>Not explaining why, just showing how to combine</li>
</ul>
<p><strong>Word count removed</strong>: ~350 words</p>

  </div>
</div>
<hr>
<p><del>## Part 5 - Understanding TypeRepr: The compiler&rsquo;s view of types</del></p>
<p><del>Before we dive into the macro, you need to understand <code>TypeRepr</code> - Scala 3&rsquo;s way of representing types during compilation.</del></p>
<p><del>Think of it like this: When you write <code>List[String]</code>, the compiler doesn&rsquo;t just see text. It sees a structured tree:</del></p>
<details class="details">
  <summary>❌ REMOVED: ASCII tree diagram of TypeRepr (click to see what was removed)</summary>
  <div class="details__body"><div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">AppliedType(List, [String])
</span></span><span class="line"><span class="cl">   │               │
</span></span><span class="line"><span class="cl">   │               └─ TypeRepr for String
</span></span><span class="line"><span class="cl">   └─ Type constructor</span></span></code></pre></td></tr></table>
</div>
</div></div>
</details>

<details class="details">
  <summary>❌ REMOVED: TypeRepr structure flowchart (click to see what was removed)</summary>
  <div class="details__body"><pre class="mermaid">flowchart TB
    A[&#34;List[String]&#34;] --&gt; B[&#34;AppliedType(List,[String])&#34;]
    B --&gt; C[&#34;tycon = List&#34;]
    B --&gt; D[&#34;args = [String]&#34;]
</pre></div>
</details>

<p><del><strong>Key TypeRepr operations:</strong></del></p>
<details class="details">
  <summary>❌ REMOVED: TypeRepr operations code examples (click to see what was removed)</summary>
  <div class="details__body"><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span> <span class="o">=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 1. Subtype checking: Is List[String] a subtype of Seq[?]?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>  <span class="c1">// true
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 2. Type equality: Is this exactly String?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// false
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 3. Pattern matching: Break it apart
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="n">tycon</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// tycon = List
</span></span></span><span class="line"><span class="cl">    <span class="o">//</span> <span class="n">args</span> <span class="k">=</span> <span class="o">[</span><span class="kt">String</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div></div>
</details>

<p><del><strong>Why we need this:</strong> To compare schemas, we need to ask questions like:</del>
<del>- &ldquo;Is this field an Option?&quot;</del>
<del>- &ldquo;What&rsquo;s inside this List?&quot;</del>
<del>- &ldquo;Does this case class have a primary constructor?&quot;</del></p>
<p><del><code>TypeRepr</code> lets us answer these at compile time.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE entire Part 5 TypeRepr tutorial</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Complete Teaching section. &ldquo;Understanding TypeRepr&rdquo; teaches macro fundamentals from scratch.</p>
<p><strong>What this served in Teaching version</strong> (~200 words):</p>
<ul>
<li>Explained TypeRepr concept for learners</li>
<li>&ldquo;Think of it like this&rdquo; (Teaching metaphor)</li>
<li>Visual tree diagram showing structure</li>
<li>Code examples demonstrating operations</li>
<li>&ldquo;Why we need this&rdquo; motivation paragraph</li>
<li>&ldquo;Before we dive into the macro, you need to understand&rdquo; (Teaching prerequisite setup)</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers already know TypeRepr (stated in Prerequisites: &ldquo;Understanding of <code>TypeRepr</code>&rdquo;)</li>
<li>Teaching progression (explain concept → show examples → motivate usage)</li>
<li>If readers don&rsquo;t understand TypeRepr, Prerequisites already told them to leave</li>
<li>External links to Scala 3 docs handle this better</li>
</ul>
<p><strong>Alternative for Resource version</strong>:</p>
<ul>
<li>Prerequisites states: &ldquo;Understanding of TypeRepr&rdquo;</li>
<li>Links to official Scala 3 macro docs</li>
<li>Jump straight to implementation code</li>
</ul>
<p><strong>Word count removed</strong>: ~200 words</p>

  </div>
</div>
<hr>
<p><del>## Part 5 - The TypeInspector pattern: One question at a time</del></p>
<p><del>Now let&rsquo;s build the utilities. Each function answers ONE question about a type. Start simple:</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE entire Part 5 TypeInspector tutorial walkthrough</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Complete Teaching section with step-by-step walkthrough. &ldquo;Each function answers ONE question&rdquo; is tutorial narrative.</p>
<p><strong>What this served in Teaching version</strong> (~650 words):</p>
<ul>
<li>&ldquo;Now let&rsquo;s build&hellip;&rdquo; (Teaching progression)</li>
<li>6 inspector functions with detailed walkthroughs</li>
<li>&ldquo;What&rsquo;s happening:&rdquo; explanations for each (Teaching technique)</li>
<li>&ldquo;Try it mentally:&rdquo; exercises (Teaching engagement)</li>
<li>&ldquo;Examples:&rdquo; with commented outputs (Teaching demonstration)</li>
<li>&ldquo;Why we need this:&rdquo; motivation paragraphs</li>
<li>&ldquo;The pattern recap:&rdquo; summary (Teaching reinforcement)</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers need the implementation code, not the walkthrough</li>
<li>&ldquo;What&rsquo;s happening&rdquo; explains code line-by-line (Teaching)</li>
<li>&ldquo;Try it mentally&rdquo; is a learning exercise</li>
<li>If readers can&rsquo;t understand TypeRepr code, Prerequisites failed</li>
<li>Inspector implementations are straightforward for experts</li>
</ul>
<p><strong>What to KEEP</strong>: None of the walkthrough text. If these functions are useful utilities, show them as a code block without explanation. But given they&rsquo;re internal helpers for the main macro, they may not need to be in Resource version at all.</p>
<p><strong>Word count removed</strong>: ~650 words</p>

  </div>
</div>
<details class="details">
  <summary>❌ REMOVED: All 6 TypeInspector functions with walkthroughs (~650 words - click to see what was removed)</summary>
  <div class="details__body"><h3 id="inspector-1-is-this-a-case-class">Inspector 1: Is this a case class?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>  <span class="c1">// Get the symbol (the &#34;name&#34; of the type)
</span></span></span><span class="line"><span class="cl">  <span class="n">s</span><span class="o">.</span><span class="n">isClassDef</span> <span class="o">&amp;&amp;</span> <span class="n">s</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">Case</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>typeSymbol</code> - Every type has a symbol (like a unique ID)</li>
<li><code>isClassDef</code> - Is it a class? (not a trait, object, etc.)</li>
<li><code>flags.is(Flags.Case)</code> - Does it have the <code>case</code> modifier?</li>
</ul>
<p><strong>Try it mentally:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>  <span class="c1">// ✅ isClassDef=true, has Case flag
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">UserTrait</span>            <span class="c1">// ❌ isClassDef=false
</span></span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">UserObject</span>          <span class="o">//</span> <span class="o">❌</span> <span class="n">not</span> <span class="n">a</span> <span class="n">class</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-2-what-are-the-type-arguments">Inspector 2: What are the type arguments?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span> <span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">args</span>  <span class="c1">// Extract the arguments
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">_</span>                    <span class="k">=&gt;</span> <span class="nc">Nil</span>   <span class="o">//</span> <span class="nc">No</span> <span class="n">arguments</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>AppliedType</code> - Pattern for generic types like <code>List[String]</code> or <code>Map[String, Int]</code></li>
<li><code>args</code> - The type parameters: <code>[String]</code> or <code>[String, Int]</code></li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>        <span class="c1">// AppliedType(List, [String])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>    <span class="c1">// AppliedType(Map, [String, Int])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String, Int]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>              <span class="c1">// Not an AppliedType
</span></span></span><span class="line"><span class="cl">                    <span class="o">//</span> <span class="n">appliedArgs</span> <span class="n">returns</span> <span class="nc">Nil</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-3-is-this-an-option">Inspector 3: Is this an Option?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>  <span class="c1">// Is it a subtype of Option[Something]?
</span></span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span>           <span class="c1">// Yes! Extract the Something
</span></span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span>                             <span class="o">//</span> <span class="nc">Nope</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>&lt;:&lt; - Subtype check: &ldquo;Is this type compatible with Option[?]?&rdquo;</li>
<li><code>Option[?]</code> - The <code>?</code> means &ldquo;Option of anything&rdquo;</li>
<li><code>appliedArgs(t).headOption</code> - Get the first (and only) type argument</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// ✅ Is subtype of Option[?]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// headOption returns Some(String)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Some</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>       <span class="c1">// ✅ Some is subtype of Option
</span></span></span><span class="line"><span class="cl">                <span class="c1">// Returns Some(Int)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>          <span class="c1">// ❌ Not an Option
</span></span></span><span class="line"><span class="cl">                <span class="o">//</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> Field-level optionality. When we see <code>age: Option[Int]</code>, we need to know:</p>
<ol>
<li>The field itself is optional</li>
<li>The underlying type is <code>Int</code></li>
</ol>
<h3 id="inspector-4-is-this-a-sequence">Inspector 4: Is this a sequence?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">isSeqLike</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Vector</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">isSeqLike</span> <span class="n">then</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span> <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s ANY kind of sequence type</li>
<li>If yes, extract what&rsquo;s inside (the element type)</li>
</ul>
<p><strong>Why check multiple types?</strong> Because Scala has many collection types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>    <span class="c1">// ✅ Is List[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Seq</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>        <span class="c1">// ✅ Is Seq[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Vector</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="c1">// ✅ Is Vector[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span>     <span class="c1">// ✅ Is Array[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>     <span class="o">//</span> <span class="o">✅</span> <span class="nc">Is</span> <span class="nc">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p>All these should become <code>SequenceShape(element)</code>.</p>
<h3 id="inspector-5-is-this-a-map">Inspector 5: Is this a Map?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[(</span><span class="kt">TypeRepr</span>, <span class="kt">TypeRepr</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Map</span><span class="o">[</span><span class="kt">?</span>, <span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">k</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">v</span> <span class="kt">::</span> <span class="kt">Nil</span> <span class="o">=&gt;</span> <span class="nc">Some</span><span class="o">((</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">))</span>  <span class="c1">// Got key and value
</span></span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Map requires two type args: </span><span class="si">${</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s a <code>Map[Something, Something]</code></li>
<li>Extract BOTH type arguments: key and value</li>
<li>If Map has wrong number of args, abort (compiler error)</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>  <span class="c1">// ✅ Returns Some((String, Int))
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Long</span>, <span class="kt">User</span><span class="o">]</span>   <span class="c1">// ✅ Returns Some((Long, User))
</span></span></span><span class="line"><span class="cl"><span class="nc">String</span>            <span class="o">//</span> <span class="o">❌</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-6-can-this-be-a-map-key">Inspector 6: Can this be a Map key?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isAtomicKey</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Short</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Boolean</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why this matters:</strong> Maps in Spark StructType only support primitive keys. You can&rsquo;t have <code>Map[User, String]</code> - User isn&rsquo;t a primitive.</p>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">User</span><span class="o">]</span>  <span class="c1">// ✅ String is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Data</span><span class="o">]</span>     <span class="c1">// ✅ Int is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">User</span>, <span class="kt">String</span><span class="o">]</span>  <span class="o">//</span> <span class="o">❌</span> <span class="nc">User</span> <span class="n">is</span> <span class="nc">NOT</span> <span class="n">atomic</span> <span class="o">-</span> <span class="n">compile</span> <span class="n">error</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="the-pattern-recap">The pattern recap:</h3>
<p>Each inspector has ONE job:</p>
<ol>
<li><strong>isCaseClass</strong> - &ldquo;Is this a case class?&rdquo;</li>
<li><strong>appliedArgs</strong> - &ldquo;What are the generic type arguments?&rdquo;</li>
<li><strong>optionArg</strong> - &ldquo;Is this an Option, and what&rsquo;s inside?&rdquo;</li>
<li><strong>seqArg</strong> - &ldquo;Is this a sequence, and what&rsquo;s the element type?&rdquo;</li>
<li><strong>mapArgs</strong> - &ldquo;Is this a Map, and what are key/value types?&rdquo;</li>
<li><strong>isAtomicKey</strong> - &ldquo;Can this be a Map key?&rdquo;</li>
</ol>
<p>Compose these to answer complex questions. That&rsquo;s the TypeInspector pattern.</p>
</div>
</details>

<hr>
<p><del>## Part 6 - Building shapes recursively: The interpreter pattern</del></p>
<p><del>Now we use those inspectors to build a <code>TypeShape</code> - our internal representation of a type&rsquo;s structure.</del></p>
<p><del>This is recursive. For <code>List[Option[User]]</code>, we build:</del>
<details class="details">
  <summary>❌ REMOVED: Recursive shape example (click to see what was removed)</summary>
  <div class="details__body"><div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">SequenceShape(
</span></span><span class="line"><span class="cl">  OptionalShape(
</span></span><span class="line"><span class="cl">    StructShape([field1, field2, ...])
</span></span><span class="line"><span class="cl">  )
</span></span><span class="line"><span class="cl">)</span></span></code></pre></td></tr></table>
</div>
</div></div>
</details>
</p>
<p><del>Let me show you the algorithm step by step.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE entire Part 6 recursive algorithm walkthrough</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Complete Teaching section with step-by-step algorithm walkthrough. &ldquo;Let me show you the algorithm step by step&rdquo; is classic Teaching narration.</p>
<p><strong>What this served in Teaching version</strong> (~550 words):</p>
<ul>
<li>&ldquo;Now we use those inspectors&hellip;&rdquo; (Teaching progression)</li>
<li>Recursive tree example showing structure</li>
<li>&ldquo;The shape building algorithm:&rdquo; flowchart</li>
<li>Algorithm implementation with inline comments</li>
<li>&ldquo;Walk through an example: <code>List[Option[String]]</code>&rdquo; (Teaching demonstration)</li>
<li>Line-by-line recursion walkthrough with bullet points</li>
<li>&ldquo;Let&rsquo;s break this down line by line:&rdquo; (Teaching technique)</li>
<li>Case class handling with numbered explanations</li>
<li>Multiple mermaid diagrams showing execution flow</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Resource readers need the implementation, not the walkthrough</li>
<li>&ldquo;Let me show you&rdquo; is Teaching language</li>
<li>Step-by-step recursion trace is a learning exercise</li>
<li>&ldquo;Let&rsquo;s break this down&rdquo; explains code line-by-line</li>
<li>If readers can&rsquo;t understand recursion, Prerequisites failed</li>
<li>Multiple pedagogical diagrams (how it works vs what it does)</li>
</ul>
<p><strong>What to KEEP (if anything)</strong>: The core <code>typeShapeOf</code> implementation code as a reference, but without the walkthrough. However, this might be internal implementation detail not needed in Resource version.</p>
<p><strong>Word count removed</strong>: ~550 words</p>

  </div>
</div>
<details class="details">
  <summary>❌ REMOVED: Part 6 complete content - algorithm flowchart, code walkthrough, recursion example (~550 words - click to see what was removed)</summary>
  <div class="details__body"><h3 id="the-shape-building-algorithm">The shape building algorithm:</h3>
<pre class="mermaid">flowchart TD
    T[&#34;typeShapeOf(t)&#34;] --&gt; O{&#34;Option?&#34;}
    O -- &#34;Yes&#34; --&gt; OI[&#34;inner = optionArg(t)&#34;] --&gt; R1[&#34;typeShapeOf(inner)&#34;]
    O -- &#34;No&#34; --&gt; S{&#34;Sequence?&#34;}
    S -- &#34;Yes&#34; --&gt; SE[&#34;elem = seqArg(t)&#34;] --&gt; R2[&#34;SequenceShape(typeShapeOf(elem))&#34;]
    S -- &#34;No&#34; --&gt; M{&#34;Map?&#34;}
    M -- &#34;Yes&#34; --&gt; MKV[&#34;(k,v) = mapArgs(t)&#34;]
    MKV --&gt; K{&#34;isAtomicKey(k)?&#34;}
    K -- &#34;No&#34; --&gt; ERR[&#34;abort: unsupported key&#34;]
    K -- &#34;Yes&#34; --&gt; MV[&#34;MapShape(PrimitiveShape(k), typeShapeOf(v))&#34;]
    M -- &#34;No&#34; --&gt; CC{&#34;Case class?&#34;}
    CC -- &#34;Yes&#34; --&gt; FS[&#34;StructShape(collect FieldShape...)&#34;]
    CC -- &#34;No&#34; --&gt; P[&#34;PrimitiveShape(t.show)&#34;]
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">TypeShape</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Step 1: Is it an Option?
</span></span></span><span class="line"><span class="cl">  <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Step 2: Is it a sequence?
</span></span></span><span class="line"><span class="cl">    <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">      <span class="c1">// Step 3: Is it a Map?
</span></span></span><span class="line"><span class="cl">      <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="o">!</span><span class="n">isAtomicKey</span><span class="o">(</span><span class="n">k</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Unsupported Map key: </span><span class="si">${</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="nc">MapShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="o">),</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">v</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1">// Step 4: Is it a case class?
</span></span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// ... handle case class
</span></span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// Step 5: Fallback - primitive
</span></span></span><span class="line"><span class="cl">          <span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>The order matters!</strong> We check Option first, then sequences, then maps, then case classes, finally primitives.</p>
<h3 id="walk-through-an-example-listoptionstring">Walk through an example: <code>List[Option[String]]</code></h3>
<pre class="mermaid">flowchart TD
    A[&#34;List[Option[String]]&#34;] --&gt;|typeShapeOf| B{&#34;Option?&#34;}
    B -- &#34;No&#34; --&gt; C{&#34;Sequence?&#34;}
    C -- &#34;Yes&#34; --&gt; D[&#34;elem = Option[String]&#34;]
    D --&gt; E[&#34;typeShapeOf(Option[String])&#34;]
    E --&gt; F{&#34;Option?&#34;}
    F -- &#34;Yes&#34; --&gt; G[&#34;inner = String&#34;]
    G --&gt; H[&#34;typeShapeOf(String)&#34;]
    H --&gt; I{&#34;Case class?&#34;}
    I -- &#34;No&#34; --&gt; J[&#34;PrimitiveShape(String)&#34;]
    J --&gt; K[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><ul>
<li>
<p><strong>Step 1:</strong> Is <code>List[Option[String]]</code> an Option?</p>
<ul>
<li>No. <code>optionArg</code> returns <code>None</code>.</li>
<li>Continue to Step 2.</li>
</ul>
</li>
<li>
<p><strong>Step 2:</strong> Is it a sequence?</p>
<ul>
<li>Yes! <code>seqArg</code> returns <code>Some(Option[String])</code>.</li>
<li>Build <code>SequenceShape(typeShapeOf(Option[String]))</code>.</li>
<li><strong>Recurse</strong> on <code>Option[String]</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 1:</strong> Is <code>Option[String]</code> an Option?</p>
<ul>
<li>Yes! <code>optionArg</code> returns <code>Some(String)</code>.</li>
<li><strong>Recurse</strong> on <code>String</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 2:</strong> Is <code>String</code> an Option?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 3:</strong> Is <code>String</code> a sequence?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 4:</strong> Is <code>String</code> a Map?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 5:</strong> Is <code>String</code> a case class?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 6:</strong> Fallback to primitive.</p>
<ul>
<li>Return <code>PrimitiveShape(&quot;String&quot;)</code>.</li>
</ul>
</li>
</ul>
<p><strong>Unwind the recursion:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="o">(</span><span class="n">from</span> <span class="nc">Option</span> <span class="n">recursion</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, where did the <code>Option</code> go? It got consumed! The <code>typeShapeOf</code> for <code>Option</code> just recurses on the inner type. This is the edge case I mentioned - element-level optionality gets lost.</p>
<h3 id="handling-case-classes">Handling case classes:</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">sym</span>    <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>                      <span class="c1">// Get the class symbol
</span></span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">params</span> <span class="k">=</span> <span class="n">sym</span><span class="o">.</span><span class="n">primaryConstructor</span><span class="o">.</span><span class="n">paramSymss</span><span class="o">.</span><span class="n">flatten</span>  <span class="c1">// Get constructor params
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">fields</span> <span class="k">=</span> <span class="n">params</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">p</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">name</span>       <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span>                      <span class="c1">// Field name
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">ptpe</span>       <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">memberType</span><span class="o">(</span><span class="n">p</span><span class="o">)</span>             <span class="c1">// Field type
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">HasDefault</span><span class="o">)</span> <span class="c1">// Has default value?
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nc">StructShape</span><span class="o">(</span><span class="n">fields</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Let&rsquo;s break this down line by line:</strong></p>
<ol>
<li><strong><code>t.typeSymbol</code></strong> - Get the symbol for the case class</li>
<li><strong><code>sym.primaryConstructor</code></strong> - Case classes have a primary constructor</li>
<li><strong><code>.paramSymss.flatten</code></strong> - Get all parameters (flatten handles multiple param lists)</li>
<li><strong>For each parameter <code>p</code>:</strong>
<ul>
<li><strong><code>p.name</code></strong> - The field name: <code>&quot;id&quot;</code>, <code>&quot;email&quot;</code>, etc.</li>
<li><strong><code>t.memberType(p)</code></strong> - The field&rsquo;s type as TypeRepr</li>
<li><strong><code>p.flags.is(Flags.HasDefault)</code></strong> - Check if it has a default value like <code>age: Int = 0</code></li>
<li><strong><code>optionArg(ptpe).fold(...)</code></strong> - Check if the field type is Option[T]:
<ul>
<li>If yes: extract T, set <code>isOpt = true</code></li>
<li>If no: use the type as-is, set <code>isOpt = false</code></li>
</ul>
</li>
<li><strong><code>typeShapeOf(uT)</code></strong> - <strong>Recurse!</strong> Build the shape for the field&rsquo;s underlying type</li>
</ul>
</li>
</ol>
<h3 id="example-case-class-userid-long-email-string-age-optionint--none">Example: <code>case class User(id: Long, email: String, age: Option[Int] = None)</code></h3>
<pre class="mermaid">flowchart TD
    U[&#34;User(id: Long, email: String, age: Option[Int] = None)&#34;]
    U --&gt; F1[&#34;Field &#39;id&#39; : Long&#34;]
    F1 --&gt; S1[&#34;PrimitiveShape(Long)&#34;]
    U --&gt; F2[&#34;Field &#39;email&#39; : String&#34;]
    F2 --&gt; S2[&#34;PrimitiveShape(String)&#34;]
    U --&gt; F3[&#34;Field &#39;age&#39; : Option[Int] (default None)&#34;]
    F3 --&gt; O1{&#34;Option?&#34;}
    O1 -- &#34;Yes&#34; --&gt; A1[&#34;inner = Int&#34;]
    A1 --&gt; S3[&#34;PrimitiveShape(Int)&#34;]
    S1 --&gt; COLLECT[&#34;Collect fields&#34;]
    S2 --&gt; COLLECT
    S3 --&gt; COLLECT
    COLLECT --&gt; FINAL[&#34;StructShape([ id:PrimitiveShape(Long), email:PrimitiveShape(String), age:PrimitiveShape(Int) (optional, default) ])&#34;]
</pre><p><strong>Processing:</strong></p>
<ul>
<li>
<p><strong>Field 1: id</strong></p>
<ul>
<li><code>name</code> = &ldquo;id&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Long</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(Long)</code> = None → <code>uT = Long</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(Long)</code> = <code>PrimitiveShape(&quot;Long&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;id&quot;, PrimitiveShape(&quot;Long&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 2: email</strong></p>
<ul>
<li><code>name</code> = &ldquo;email&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>String</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(String)</code> = None → <code>uT = String</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(String)</code> = <code>PrimitiveShape(&quot;String&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;email&quot;, PrimitiveShape(&quot;String&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 3: age</strong></p>
<ul>
<li><code>name</code> = &ldquo;age&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Option[Int]</code></li>
<li><code>hasDefault</code> = <strong>true</strong> (has <code>= None</code>)</li>
<li><code>optionArg(Option[Int])</code> = Some(<code>Int</code>) → <code>uT = Int</code>, <code>isOpt = true</code></li>
<li><code>typeShapeOf(Int)</code> = <code>PrimitiveShape(&quot;Int&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;age&quot;, PrimitiveShape(&quot;Int&quot;), hasDefault=true, isOptional=true)</code></li>
</ul>
</li>
<li>
<p><strong>Final shape:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">StructShape</span><span class="o">([</span>
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">id</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Long</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">email</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">String</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">age</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Int</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">true</span>, <span class="kt">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">])</span></span></span></code></pre></td></tr></table>
</div>
</div></li>
</ul>
<h3 id="why-this-matters">Why this matters:</h3>
<p>Now we have a normalized representation of the case class. We can compare two <code>StructShape</code>s to see if they match, regardless of the original Scala syntax.</p>
<p>The recursive structure handles arbitrarily nested types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Order</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>           <span class="c1">// Recurses into List, then into LineItem
</span></span></span><span class="line"><span class="cl">  <span class="n">metadata</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">],</span>   <span class="c1">// Recurses into Map, then String (twice)
</span></span></span><span class="line"><span class="cl">  <span class="n">address</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">]</span>         <span class="c1">// Recurses into Option, then Address
</span></span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Each recursion builds a piece of the tree. The tree is the schema.</p>
</div>
</details>

<hr>
<h2 id="part-7---field-vs-element-optionality-a-critical-distinction">Part 7 - Field vs Element optionality: A critical distinction</h2>
<p><del>Here&rsquo;s something subtle that most guides miss. The code in github handles two kinds of optionality:</del></p>
<div class="admonition tip">
  <div class="admonition-title">⭕ KEEP Part 7 optionality gotcha</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 7 with minor edits - remove Teaching preamble, keep technical content.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Gotcha/edge case documentation (critical for production)</li>
<li>&ldquo;Most guides miss&rdquo; → experts appreciate knowing pitfalls</li>
<li>Specific bug: <code>List[Option[String]]</code> getting flattened</li>
<li>Includes workaround and warning admonition</li>
<li>Visual diagram showing the issue</li>
</ul>
<p><strong>Why this is Resource, not Teaching</strong>:</p>
<ul>
<li>Not teaching concepts, warning about bugs</li>
<li>Production safety content</li>
<li>Experts need to know limitations before deploying</li>
</ul>
<p><strong>Minor edit needed</strong>: Remove &ldquo;Here&rsquo;s something subtle&hellip;&rdquo; preamble (Teaching tone).</p>
<p><strong>Word count kept</strong>: ~250 words</p>

  </div>
</div>
<p>The code in github handles two kinds of optionality:</p>
<h3 id="field-level-optionality-option-on-the-field-itself">Field-level optionality (Option on the field itself)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                                   <span class="o">^^^^^^^^^^^</span> <span class="n">field</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The macro detects this and sets <code>isOptional = true</code> on the FieldShape:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="element-level-optionality-option-inside-a-collection">Element-level optionality (Option inside a collection)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Data</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                          <span class="o">^^^^^^^^^^^^^^</span> <span class="n">element</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This creates a nested shape: <code>SequenceShape(OptionShape(PrimitiveShape(&quot;String&quot;)))</code></p>
<p>Wait, that&rsquo;s not in the code in github! Let me check&hellip; Actually, the code in github&rsquo;s <code>typeShapeOf</code> handles this by recursing:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>So <code>List[Option[String]]</code> becomes:</p>
<ol>
<li><code>seqArg</code> detects List → extract <code>Option[String]</code></li>
<li>Recurse on <code>Option[String]</code> → <code>typeShapeOf(Option[String])</code></li>
<li><code>optionArg</code> detects Option → extract <code>String</code></li>
<li>Base case: <code>PrimitiveShape(&quot;String&quot;)</code></li>
</ol>
<p>But the final shape is <code>SequenceShape(PrimitiveShape(&quot;String&quot;))</code> - the intermediate Option is consumed!</p>
<p><strong>Why this matters for contracts</strong>: When you write:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Producer</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span></span></span></code></pre></td></tr></table>
</div>
</div><p>These should probably NOT conform under Exact. The producer allows null elements, the contract doesn&rsquo;t. But the current code in github might allow it because the Option gets unwrapped.</p>
<p>This is the kind of edge case you discover when shipping to production. The fix would be to add an <code>OptionalShape</code> wrapper in the model.</p>
<div class="admonition caution">
  <div class="admonition-title">Optionality edge case</div>
  <div class="admonition-content">
    The element-level optionality issue (List[Option[String]] getting flattened) is real and subtle. If your schemas use nullable collections, test that the compile-time check actually catches the difference between List[String] and List[Option[String]]. The current implementation may allow mismatches here. Add explicit test cases for your nullable collection patterns before relying on them in production.
  </div>
</div>
<p>Visual (field vs element optionality):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/optionality.svg"
            alt="Field vs Element Optionality"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    Field optionality vs element optionality - they&#39;re not the same.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<pre class="mermaid">flowchart LR
    A[&#34;List[Option[String]]&#34;] --&gt; B{&#34;Element optionality?&#34;}
    B -- &#34;intended shape&#34; --&gt; C[&#34;SequenceShape(OptionalShape(PrimitiveShape(String)))&#34;]
    B -- &#34;current code in github\n(consumes Option)&#34; --&gt; D[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><hr>
<h2 id="part-8---compiletime--evidence-the-full-picture">Part 8 - Compile‑time <strong>&ldquo;conforms&rdquo;</strong> evidence (the full picture)</h2>
<p><del>Now let&rsquo;s put it all together. Here&rsquo;s the complete macro from the code in github:</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 8 Conforms macro (remove walkthrough)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the macro implementation code, remove &ldquo;Let me break down&rdquo; walkthrough.</p>
<p><strong>Why implementation fits Resource</strong>:</p>
<ul>
<li>This IS the core API - the Conforms macro</li>
<li>Complete, working implementation</li>
<li>Production-ready code</li>
<li>Copy-paste ready</li>
</ul>
<p><strong>What to REMOVE</strong>:</p>
<ul>
<li>&ldquo;Now let&rsquo;s put it all together&rdquo; (Teaching progression)</li>
<li>&ldquo;Let me break down what&rsquo;s happening:&rdquo; section (Teaching walkthrough)</li>
<li>Numbered explanations of each step</li>
</ul>
<p><strong>Why walkthrough doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>Experts read macro code directly</li>
<li>&ldquo;Let me break down&rdquo; is Teaching language</li>
<li>If readers can&rsquo;t understand this, Prerequisites failed</li>
</ul>
<p><strong>Word count</strong>: Keep ~120 words (code), Remove ~150 words (walkthrough)</p>

  </div>
</div>
<p>Here&rsquo;s the complete macro from the code in github:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Conforms.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.quoted.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">class</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">object</span> <span class="nc">Conforms</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">materialize</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">$</span><span class="o">{</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">P:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span>, <span class="kt">Boolean</span>, <span class="kt">Boolean</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">fs</span> <span class="k">=</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span>
</span></span><span class="line"><span class="cl">        <span class="n">fs</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">hasDefault</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">isOptional</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Compare fields by name (unordered, case-sensitive for now)
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">outMap</span>      <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contractMap</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">missing</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">outMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s">:</span><span class="si">$t$</span><span class="s">{if opt then &#34;</span> <span class="o">(</span><span class="n">optional</span><span class="o">)</span><span class="s">&#34; else &#34;&#34;}&#34;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">extra</span>   <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span>      <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">contractMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mism</span>    <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span> <span class="o">!=</span> <span class="n">t</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> expected </span><span class="si">$t</span><span class="s">, found </span><span class="si">${</span><span class="n">outMap</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">_1</span><span class="si">}</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">!=</span> <span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> optionality mismatch&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">okMissing</span><span class="o">,</span> <span class="n">okExtra</span><span class="o">,</span> <span class="n">okMism</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Exact</span><span class="o">]</span>    <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Backward</span><span class="o">]</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">.</span><span class="n">filterNot</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="s">&#34;(optional)&#34;</span><span class="o">)),</span> <span class="nc">Nil</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Forward</span><span class="o">]</span>  <span class="k">=&gt;</span> <span class="o">(</span><span class="nc">Nil</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span>                  <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">okMissing</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okExtra</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okMism</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="s">s&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">        |Compile‑time contract drift (policy: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span><span class="si">}</span><span class="s">).
</span></span></span><span class="line"><span class="cl"><span class="s">        |Out: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span><span class="si">}</span><span class="s"> vs Contract: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Missing: </span><span class="si">${</span><span class="n">okMissing</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Extra: </span><span class="si">${</span><span class="n">okExtra</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Mismatched: </span><span class="si">${</span><span class="n">okMism</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |&#34;&#34;&#34;</span><span class="o">.</span><span class="n">stripMargin</span>
</span></span><span class="line"><span class="cl">      <span class="n">quotes</span><span class="o">.</span><span class="n">reflect</span><span class="o">.</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="n">msg</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="k">new</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>Let me break down what&rsquo;s happening:</del></p>
<p><del>1. <strong>Extract shapes</strong>: We summon <code>Shape[Out]</code> and <code>Shape[Contract]</code> and get their field lists. Note the <code>.valueOrAbort</code> - this forces compile-time evaluation.</del></p>
<p><del>2. <strong>Compare</strong>: We build maps of field names → (type, optionality) and find differences.</del></p>
<p><del>3. <strong>Apply policy</strong>:</del>
<del>- Exact: all differences are errors</del>
<del>- Backward: missing optional fields are OK, extras are OK</del>
<del>- Forward: extras are OK (but missing required fields are not)</del></p>
<p><del>4. <strong>Abort or succeed</strong>: If there are violations, we call <code>report.errorAndAbort</code> with a detailed message showing exactly what&rsquo;s wrong. Otherwise, we return the evidence.</del></p>
<p><del>The beauty here: <strong>this runs during compilation</strong>. If schemas don&rsquo;t match, your code won&rsquo;t compile. Period.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE numbered walkthrough</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;Let me break down what&rsquo;s happening:&rdquo; is classic Teaching technique. Resource readers read the macro code directly.</p>
<p><strong>What this served</strong>: Step-by-step explanation of macro logic for learners.</p>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>: Experts understand macro implementations without numbered guides.</p>
<p><strong>Word count removed</strong>: ~150 words</p>

  </div>
</div>
<div class="admonition danger">
  <div class="admonition-title">Build breakage alert</div>
  <div class="admonition-content">
    Compile-time contract checks will break your build when schemas drift. That&rsquo;s the point, but coordinate with your team. When upstream adds a field, every downstream pipeline build fails until they update contracts or transforms. This forces alignment, but can block deployments if not managed. Use CI branch builds to preview impact before merging schema changes.
  </div>
</div>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP warning admonition</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the &ldquo;Build breakage alert&rdquo; admonition.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Production gotcha/warning</li>
<li>Practical deployment advice</li>
<li>Team coordination guidance</li>
</ul>
<p><strong>Word count kept</strong>: ~50 words</p>

  </div>
</div>
<p>Use it like this:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ConformsSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutOk</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutMissing</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkConformance</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutOk</span>,      <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>   <span class="c1">// ✅ compiles
</span></span></span><span class="line"><span class="cl">  <span class="c1">// summon[Conforms[OutMissing, Contract, Policy.Exact]] // ❌ compile‑time error (Missing: email:String)
</span></span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span> <span class="o">//</span> <span class="o">✅</span> <span class="n">compiles</span> <span class="o">(</span><span class="n">migration</span> <span class="n">mode</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition success">
  <div class="admonition-title">⭕ KEEP usage example</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the usage code example.</p>
<p><strong>Why it fits Resource</strong>:</p>
<ul>
<li>Shows API usage clearly</li>
<li>Success and failure cases</li>
<li>Copy-paste ready</li>
<li>No Teaching narration, just code</li>
</ul>
<p><strong>Word count kept</strong>: ~40 words (code)</p>

  </div>
</div>
<p><del>That&rsquo;s it. The entire core. In production, you&rsquo;d want to handle the element optionality issue I mentioned, but the structure stays the same.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE conclusion paragraph</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: &ldquo;That&rsquo;s it. The entire core.&rdquo; is Teaching summary. Resource readers don&rsquo;t need narration.</p>
<p><strong>Word count removed</strong>: ~25 words</p>

  </div>
</div>
<hr>
<p><del>## Part 9 - Developer ergonomics: What using this actually feels like</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE entire Part 9 developer ergonomics</div>
  <div class="admonition-content">
    <p><strong>Why remove</strong>: Complete Teaching section about &ldquo;what it feels like&rdquo; to use the tool. Resource readers want the API, not subjective experiences.</p>
<p><strong>What this served in Teaching version</strong> (~350 words):</p>
<ul>
<li>&ldquo;Let&rsquo;s talk about the practical side&hellip;&rdquo; (Teaching narrative)</li>
<li>Compile times discussion (answering learner concerns)</li>
<li>Error messages walkthrough (showing what it looks like)</li>
<li>Integration patterns (Spark, schema registries, CI/CD examples)</li>
<li>Onboarding guidance (&ldquo;New team members ask&hellip;&rdquo;)</li>
<li>&ldquo;When to use vs skip&rdquo; decision matrix</li>
</ul>
<p><strong>Why it doesn&rsquo;t fit Resource</strong>:</p>
<ul>
<li>&ldquo;What using this actually feels like&rdquo; is Teaching empathy</li>
<li>Resource doesn&rsquo;t discuss feelings, just facts</li>
<li>Integration examples belong in dedicated Integration section (kept elsewhere)</li>
<li>&ldquo;When to use&rdquo; is for deciding, Resource assumes you decided</li>
<li>Onboarding guidance is for managers, not engineers using the API</li>
</ul>
<p><strong>Word count removed</strong>: ~350 words</p>

  </div>
</div>
<details class="details">
  <summary>❌ REMOVED: Part 9 complete content - ergonomics, compile times, integration patterns (~350 words - click to see what was removed)</summary>
  <div class="details__body"><p>Let&rsquo;s talk about the practical side - what happens when you actually use this in your daily work?</p>
<h3 id="compile-times">Compile times</h3>
<p><strong>Question: Does this slow down compilation?</strong></p>
<p>In practice, not noticeably. The macro runs once per <code>summon[Conforms[...]]</code> call during compilation. For a typical pipeline with 5-10 contract checks, you&rsquo;re adding maybe 100-200ms to your build. Compare that to the hours you&rsquo;d spend debugging a production schema mismatch.</p>
<h3 id="error-messages">Error messages</h3>
<p>When schemas drift, you get this:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[error] Compile-time contract drift (policy: Policy.Exact).
</span></span><span class="line"><span class="cl">[error] Out: CustomerProducer vs Contract: CustomerContract
</span></span><span class="line"><span class="cl">[error] Extra: segment
</span></span><span class="line"><span class="cl">[error] Missing:
</span></span><span class="line"><span class="cl">[error] Mismatched:</span></span></code></pre></td></tr></table>
</div>
</div><p>Clear, actionable, and it stops your build. No cryptic macro errors, no stack traces. Just: &ldquo;Hey, you have an extra field called <code>segment</code>.&rdquo;</p>
<h3 id="integration-patterns">Integration patterns</h3>
<p><strong>With Spark:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Your Spark job
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">df</span> <span class="k">=</span> <span class="n">spark</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">typed</span> <span class="k">=</span> <span class="n">df</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">]</span>  <span class="c1">// Spark encoder
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Compile-time check before transformation
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerProducer</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">output</span> <span class="k">=</span> <span class="n">typed</span><span class="o">.</span><span class="n">transform</span><span class="o">(</span><span class="n">dropSegment</span><span class="o">).</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>With schema registries:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Avro schema in registry → case class → contract check
</span></span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerAvro</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">amt</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerAvro</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span> <span class="nc">Build</span> <span class="n">fails</span> <span class="k">if</span> <span class="nc">Avro</span> <span class="n">schema</span> <span class="n">drift</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>CI/CD integration:</strong>
Just run <code>sbt compile</code>. If contracts drift, the build fails. No special tooling needed. Works with Jenkins, GitHub Actions, GitLab CI - anything that runs sbt.</p>
<h3 id="onboarding">Onboarding</h3>
<p><strong>New team members ask</strong>: &ldquo;Why won&rsquo;t this compile?&rdquo;</p>
<p><strong>Answer</strong>: &ldquo;Check the error. You&rsquo;re trying to use <code>CustomerProducer</code> but the contract expects <code>CustomerContract</code>. Either adjust your transform or update the contract.&rdquo;</p>
<p>That&rsquo;s it. The compiler guides them. After one or two cases, they get it.</p>
<h3 id="when-to-use-vs-skip">When to use vs skip</h3>
<p><strong>Use compile-time contracts when</strong>:</p>
<ul>
<li>Multiple teams work on the same pipeline codebase</li>
<li>Schema changes break production regularly</li>
<li>You want fast feedback (compile vs deploy-and-test)</li>
</ul>
<p><strong>Skip when</strong>:</p>
<ul>
<li>One-off scripts or exploratory work</li>
<li>External APIs you can&rsquo;t control (use runtime validation)</li>
<li>Schema is genuinely dynamic (JSON with unknown keys)</li>
</ul>
</div>
</details>

<hr>
<h2 id="part-10---nested-types-maps-lists-and-deep-structures">Part 10 - Nested types: Maps, Lists, and deep structures</h2>
<p><del>The code in github handles nested types beautifully. Look at this example from <code>CtdcPoc.scala</code>:</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 10 nested types example (remove walkthrough)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the code example, remove &ldquo;What&rsquo;s happening here:&rdquo; walkthrough.</p>
<p><strong>Why example fits Resource</strong>:</p>
<ul>
<li>Working code showing nested structures</li>
<li>Real-world patterns (Maps, Lists, nested case classes)</li>
<li>Copy-paste ready</li>
</ul>
<p><strong>What to REMOVE</strong>:</p>
<ul>
<li>&ldquo;The code in github handles nested types beautifully&rdquo; (unnecessary praise)</li>
<li>&ldquo;What&rsquo;s happening here:&rdquo; numbered walkthrough</li>
</ul>
<p><strong>Word count</strong>: Keep ~60 words (code), Remove ~100 words (walkthrough)</p>

  </div>
</div>
<p>Look at this example from <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Deep / Nested structures
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">LineItem</span><span class="o">(</span><span class="n">sku</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">qty</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">attrs</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Address</span><span class="o">(</span><span class="n">street</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">zip</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderOut</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderContract</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>        <span class="c1">// List vs Seq - both are sequences
</span></span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>      <span class="c1">// Nested case class
</span></span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Nil</span>       <span class="c1">// Set vs Seq, has default
</span></span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">evDeepOk</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">OrderOut</span>, <span class="kt">OrderContract</span>, <span class="kt">SchemaPolicy.Backward.</span><span class="k">type</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del><strong>What&rsquo;s happening here:</strong></del></p>
<p><del>1. <strong>Nested case classes</strong> - <code>Address</code> inside <code>OrderOut</code>. The macro recurses: when it sees <code>Option[Address]</code>, it unwraps to <code>Address</code>, then checks if <code>Address</code> is a case class, and recurses again to get its fields.</del></p>
<p><del>2. <strong>Collections</strong> - <code>List[LineItem]</code> vs <code>Seq[LineItem]</code>. Both match because <code>seqArg</code> treats them equivalently. The macro sees &ldquo;sequence of LineItem&rdquo; and recurses on <code>LineItem</code>.</del></p>
<p><del>3. <strong>Maps</strong> - <code>Map[String, String]</code> in <code>attrs</code>. The macro checks key type (must be atomic), then recurses on value type.</del></p>
<p><del>4. <strong>Default values</strong> - <code>tags: Seq[String] = Nil</code> has a default. Under Backward policy, if <code>OrderOut</code> is missing <code>tags</code>, it&rsquo;s OK because the contract has a default.</del></p>
<p><del>This is why the TypeInspector pattern matters - it handles arbitrary nesting without special cases.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE &#39;What&#39;s happening here&#39; walkthrough</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through the numbered explanation walkthrough.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>Teaching technique: &ldquo;What&rsquo;s happening here:&rdquo; followed by step-by-step explanation</li>
<li>Explains how the macro works internally (recurses, unwraps, treats equivalently)</li>
<li>Resource readers can read the code example and understand the nested types capability</li>
<li>The explanation of TypeInspector pattern is Teaching content</li>
</ul>
<p><strong>Word count removed</strong>: ~115 words</p>

  </div>
</div>
<hr>
<h2 id="part-10---policy-modifiers-in-two-minutes-ordered-ci-byposition">Part 10 - Policy modifiers in two minutes (Ordered, CI, By‑Position)</h2>
<p>Sometimes you need stricter comparison. Or different rules. Here are three quick policy extensions:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/PolicyMods.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">PolicyMod</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">ExactOrdered</span>   <span class="kt">extends</span> <span class="kt">PolicyMod</span> <span class="c1">// names + order + types must match
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactCI</span>        <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// case‑insensitive field names
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// ignore names; types by index
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactOrdered</span>    <span class="k">extends</span> <span class="nc">ExactOrdered</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactCI</span>         <span class="k">extends</span> <span class="nc">ExactCI</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">ExactByPosition</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">ConformsMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">import</span> <span class="kt">scala.quoted.*</span>
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span> <span class="k">&lt;:</span> <span class="kt">PolicyMod</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">M:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">fields</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">))</span> <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mismatches</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">M</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">n</span> <span class="k">=</span> <span class="o">(</span><span class="n">s</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">toLowerCase</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">om</span> <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">cm</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">keySet</span> <span class="n">union</span> <span class="n">om</span><span class="o">.</span><span class="n">keySet</span><span class="o">).</span><span class="n">toList</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span> <span class="n">k</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">          <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">),</span> <span class="n">om</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">))</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="n">t1</span><span class="o">),</span> <span class="nc">Some</span><span class="o">(</span><span class="n">t2</span><span class="o">))</span> <span class="k">if</span> <span class="n">t1</span> <span class="o">!=</span> <span class="n">t2</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;</span><span class="si">$k</span><span class="s"> expected </span><span class="si">$t1</span><span class="s">, found </span><span class="si">$t2</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">),</span> <span class="nc">None</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;missing </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">None</span><span class="o">,</span> <span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">))</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;extra </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="k">_</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="k">_</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">!=</span> <span class="n">tC</span> <span class="k">=&gt;</span> <span class="s">s&#34;@</span><span class="si">$i</span><span class="s"> expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactOrdered</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="n">nO</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="n">nC</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">nameOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">nO</span> <span class="o">==</span> <span class="n">nC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(name) expected </span><span class="si">$nC</span><span class="s">, found </span><span class="si">$nO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">typeOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">==</span> <span class="n">tC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(type) expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="n">nameOk</span> <span class="o">++</span> <span class="n">typeOk</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;Unknown policy mod&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">mismatches</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Modifier drift: </span><span class="si">${</span><span class="n">mismatches</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="o">()</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Try these:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">com.example.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">A</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">B</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span>, <span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="c1">// ✅ same types by index
</span></span></span><span class="line"><span class="cl"><span class="c1">// ConformsMod[A, B, PolicyMod.ExactOrdered] // ❌ order matters → name mismatch
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIa</span><span class="o">(</span><span class="nc">ID</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIb</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">CIa</span>, <span class="kt">CIb</span>, <span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="o">//</span> <span class="o">✅</span> <span class="k">case</span><span class="err">‑</span><span class="n">insensitive</span> <span class="n">names</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>Why these three? Because real systems need them:</del>
<del>- <strong>ExactOrdered</strong>: For formats where position matters (some CSV parsers, protobuf with field numbers)</del>
<del>- <strong>ExactCI</strong>: Because someone always uses <code>UserId</code> when the contract says <code>userId</code></del>
<del>- <strong>ExactByPosition</strong>: For legacy systems that ignore field names entirely</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 10 Policy modifiers (remove motivation explanation)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep the policy modifier code and examples, remove &ldquo;Why these three?&rdquo; explanation.</p>
<p><strong>Why keep the code</strong>:</p>
<ul>
<li>API extension patterns showing how to add custom policies</li>
<li>Three concrete policy variants with implementation</li>
<li>Usage examples showing when each compiles vs fails</li>
<li>This is reference documentation for advanced usage</li>
</ul>
<p><strong>Why remove &ldquo;Why these three?&rdquo;</strong>:</p>
<ul>
<li>Teaching motivation explaining &ldquo;Because real systems need them&rdquo;</li>
<li>Justification for design decisions (Teaching content)</li>
<li>Resource readers can understand from code and usage examples</li>
</ul>
<p><strong>Word count removed</strong>: ~40 words</p>

  </div>
</div>
<hr>
<h2 id="part-11---phantom-types-building-impossible-to-misuse-apis">Part 11 - Phantom types: Building impossible-to-misuse APIs</h2>
<p><del>Here&rsquo;s where things get interesting.</del> The code, also uses phantom types, type-indexing / typestate builder patterns to build a pipeline builder that&rsquo;s <strong>impossible to misuse</strong>.
<del>What are phantom types? They&rsquo;re type parameters that don&rsquo;t appear in the class body but control what you can do with an instance. Think of them as compile-time state machines.</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 11 Phantom types (remove teaching explanations)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 11 - it demonstrates advanced production pattern, but remove Teaching explanations.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Advanced production pattern (Phantom Type Builder Pattern)</li>
<li>Shows compile-time state machine enforcement</li>
<li>Demonstrates integration with SchemaConforms contract checking</li>
<li>References to authoritative sources (Xebia, Rhetorical Musings)</li>
<li>Production rationale about enforcing operation order</li>
</ul>
<p><strong>Why remove some content</strong>:</p>
<ul>
<li>&ldquo;Here&rsquo;s where things get interesting&rdquo; - Teaching enthusiasm</li>
<li>&ldquo;What are phantom types?&rdquo; definition - Teaching 101</li>
<li>&ldquo;Think of them as&hellip;&rdquo; - Teaching mental model</li>
</ul>
<p><strong>Strategy</strong>: Strike through introductory teaching, keep code, diagram, and production discussion.</p>

  </div>
</div>
<pre class="mermaid">stateDiagram-v2
    [*] --&gt; Empty
    Empty --&gt; WithSource: addSource
    WithSource --&gt; WithTransform: transformAs[Next]
    WithTransform --&gt; Complete: addSink[Contract,P] + conforms
    Complete --&gt; [*]: build

    note right of WithTransform
      Evidence constraints:
      - S &lt;: WithSource to transform
      - S &lt;: WithTransform to sink
      - S = Complete to build
    end note
</pre><p>From the codebase <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Phantom type states
</span></span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Empty</span>         <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithSource</span>    <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithTransform</span> <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Complete</span>      <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">S</span> <span class="k">&lt;:</span> <span class="kt">BuilderState</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="nc">private</span> <span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">steps</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">PipelineStep</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add source when Empty
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSource</span><span class="o">[</span><span class="kt">C</span><span class="o">](</span><span class="n">src</span><span class="k">:</span> <span class="kt">TypedSource</span><span class="o">[</span><span class="kt">C</span><span class="o">])(</span><span class="n">using</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">])</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only transform after adding source
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">f</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span><span class="o">)(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span><span class="o">,</span>  <span class="c1">// This constraint enforces order!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">Next</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add sink after transform
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSink</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">](</span><span class="n">sink</span><span class="k">:</span> <span class="kt">TypedSink</span><span class="o">[</span><span class="kt">R</span><span class="o">])(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev0</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithTransform</span><span class="o">,</span>  <span class="c1">// Must have transform
</span></span></span><span class="line"><span class="cl">    <span class="n">ev1</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">CurContract</span>, <span class="kt">R</span>, <span class="kt">P</span><span class="o">],</span>  <span class="c1">// Compile-time contract check!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">R</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only build when Complete
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">build</span><span class="o">(</span><span class="n">using</span> <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="o">=</span><span class="k">:</span><span class="o">=</span> <span class="nc">Complete</span><span class="o">)</span><span class="k">:</span> <span class="kt">SparkSession</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="o">(</span><span class="n">spark</span><span class="k">:</span> <span class="kt">SparkSession</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PipelineBuilder</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">apply</span><span class="o">[</span><span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="nc">Nil</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>### How this works:</del></p>
<p><del>1. <strong>State transitions as types</strong> - Each method returns a new type parameter <code>S</code>. The compiler tracks which state you&rsquo;re in.</del></p>
<p><del>2. <strong>Evidence constraints</strong> - ev: S &lt;:&lt; WithSource means &ldquo;S must be a subtype of WithSource&rdquo;. If it&rsquo;s not, this method doesn&rsquo;t exist.</del></p>
<p><del>3. <strong>Compile-time state machine</strong> - You literally cannot call methods in the wrong order:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// ✅ This compiles
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;good&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">src</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// ❌ This doesn&#39;t compile - can&#39;t transform before adding source
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;bad&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">ERROR</span><span class="k">:</span> <span class="kt">No</span> <span class="kt">implicit</span> <span class="kt">evidence</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>4. <strong>Contract checking embedded</strong> - Notice <code>ev1: SchemaConforms[CurContract, R, P]</code> in <code>addSink</code>? That&rsquo;s where the compile-time contract check happens. The pipeline literally won&rsquo;t build if schemas drift.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE &#39;How this works&#39; walkthrough</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through the numbered walkthrough, keep the usage examples.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;How this works:&rdquo; - Teaching walkthrough structure</li>
<li>Numbered explanations of state transitions, evidence constraints</li>
<li>&ldquo;Notice&hellip;?&rdquo; - Teaching call-out technique</li>
<li>&ldquo;That&rsquo;s where&hellip;&rdquo; - Teaching explanation of mechanism</li>
</ul>
<p><strong>Why keep usage examples</strong>:</p>
<ul>
<li>Shows what compiles vs what doesn&rsquo;t</li>
<li>Concrete code demonstrating the API</li>
<li>Error messages document expected behavior</li>
</ul>
<p><strong>Word count removed</strong>: ~95 words</p>

  </div>
</div>
<p>This pattern is called the <strong>Phantom Type Builder Pattern</strong>. For more details, see:</p>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala (Rhetorical Musings)</a>
</li>
</ul>
<p><del><strong>Why this matters</strong>: In production data pipelines, the order of operations matters. Read → Transform → Validate → Write. With phantom types, the compiler enforces this order. You can&rsquo;t accidentally write before transforming. You can&rsquo;t transform without a source.</del></p>
<p><del>This is the difference between &ldquo;here&rsquo;s how macros work&rdquo; and &ldquo;here&rsquo;s how to build production contract systems.&quot;</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE closing teaching explanation</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through &ldquo;Why this matters&rdquo; and philosophical closing.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;Why this matters:&rdquo; - Teaching motivation pattern</li>
<li>Explains the importance and benefits (Teaching technique)</li>
<li>&ldquo;This is the difference between&hellip;&rdquo; - Meta-commentary about teaching itself</li>
<li>Resource readers can understand the value from the code and references</li>
</ul>
<p><strong>Word count removed</strong>: ~60 words</p>
<p><strong>Note</strong>: References to Xebia and Rhetorical Musings blog posts are KEPT as they&rsquo;re Resource links.</p>

  </div>
</div>
<hr>
<h2 id="part-12---schema-evolution-and-versioning-handling-change-over-time">Part 12 - Schema evolution and versioning: Handling change over time</h2>
<p><del>Here&rsquo;s the reality:</del> Schemas evolve. Fields get added, deprecated, renamed. Teams roll out changes incrementally. <del>The question isn&rsquo;t whether schemas will change - it&rsquo;s how you manage those changes without breaking production.</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 12 Schema evolution (remove teaching scenario)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 12 - it documents production migration strategies and versioning patterns.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Production migration strategies (Phase 1/2/3 deployment)</li>
<li>Versioning patterns with code examples</li>
<li>Handling breaking changes (field renames)</li>
<li>Coexistence patterns during migration</li>
<li>Complete real-world migration example from codebase</li>
<li>Mermaid diagram showing migration phases</li>
</ul>
<p><strong>Why remove some content</strong>:</p>
<ul>
<li>&ldquo;Here&rsquo;s the reality:&rdquo; - Teaching preamble</li>
<li>&ldquo;The question isn&rsquo;t whether&hellip;&rdquo; - Teaching philosophy</li>
<li>Wrong/Right approach contrast - Teaching technique showing anti-patterns first</li>
<li>&ldquo;What&rsquo;s happening here:&rdquo; - Teaching walkthrough</li>
<li>&ldquo;This is real-world stuff&rdquo; - Teaching enthusiasm</li>
</ul>
<p><strong>Strategy</strong>: Strike through Teaching framing, keep all technical content, versioning strategies, and code.</p>

  </div>
</div>
<h3 id="the-schema-evolution-problem">The schema evolution problem</h3>
<p><del><strong>Scenario</strong>: You have <code>CustomerV1</code> running in production. Marketing wants to add a <code>segment</code> field for targeting. But you have 10 pipelines reading <code>CustomerV1</code>. How do you evolve without breaking everything?</del></p>
<p><del><strong>Wrong approach</strong>: Add <code>segment</code> to <code>CustomerV1</code>, deploy, hope for the best. Result: Some pipelines break because they don&rsquo;t handle the new field.</del></p>
<p><del><strong>Right approach</strong>: Version your contracts and use policies to manage transitions.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE teaching scenario setup</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through the scenario, wrong/right approach contrast.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>Teaching technique: presents problem scenario first</li>
<li>Shows anti-pattern (&ldquo;Wrong approach&rdquo;) before solution</li>
<li>Conversational &ldquo;How do you&hellip;&rdquo; question to reader</li>
<li>Resource readers want the solution directly, not the motivation</li>
</ul>
<p><strong>Word count removed</strong>: ~60 words</p>

  </div>
</div>
<h3 id="versioning-strategy">Versioning strategy</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala - Current production
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala - New version with segment
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>Notice <code>segment</code> is:</del>
<del>1. <strong>Optional</strong> (<code>Option[String]</code>)</del>
<del>2. <strong>Has a default</strong> (<code>= None</code>)</del></p>
<p><del>This makes it backward-compatible - old code can work with new data.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE &#39;Notice...&#39; teaching callout</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through the &ldquo;Notice&hellip;&rdquo; explanation.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;Notice&hellip;&rdquo; - Teaching call-out technique</li>
<li>Numbered list explaining design choices</li>
<li>Explains &ldquo;why&rdquo; (backward compatibility) - Teaching motivation</li>
<li>Resource readers can see <code>Option[String] = None</code> in the code</li>
</ul>
<p><strong>Word count removed</strong>: ~25 words</p>

  </div>
</div>
<h3 id="migration-phases">Migration phases</h3>
<p><strong>Phase 1: Add the field (Backward policy)</strong></p>
<p><del>Deploy new producers writing <code>CustomerV2</code>:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">producer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;write-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">rawData</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addSegment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">Policy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Backward allows optional fields
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>Old consumers still read <code>CustomerV1</code>. They ignore the <code>segment</code> field. No breakage.</del></p>
<p><strong>Phase 2: Migrate consumers</strong></p>
<p><del>Update each consumer one by one:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">consumer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;read-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">source</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">EnrichedCustomer</span><span class="o">](</span><span class="n">useSegment</span><span class="o">)</span>  <span class="c1">// Now uses segment field
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">EnrichedCustomer</span>, <span class="kt">Policy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>The compile-time contract ensures: &ldquo;Does this consumer handle the new schema?&rdquo; If you forget to update the case class, it won&rsquo;t compile.</del></p>
<p><strong>Phase 3: Deprecate V1</strong></p>
<p><del>Once all pipelines use <code>CustomerV2</code>:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Mark V1 as deprecated
</span></span></span><span class="line"><span class="cl"><span class="nd">@deprecated</span><span class="o">(</span><span class="s">&#34;Use CustomerV2&#34;</span><span class="o">,</span> <span class="s">&#34;2025-10-01&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>The compiler warns any remaining V1 usage. After a grace period, delete V1 entirely.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE teaching narrative from migration phases</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through teaching narrative around the phase code examples.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;Deploy new producers&hellip;&rdquo; - Teaching instructions</li>
<li>&ldquo;Old consumers still read&hellip;&rdquo; - Teaching explanation of what happens</li>
<li>&ldquo;Update each consumer one by one:&rdquo; - Teaching step-by-step</li>
<li>&ldquo;The compile-time contract ensures:&hellip;&rdquo; - Teaching explanation with rhetorical question</li>
<li>&ldquo;Once all pipelines&hellip;&rdquo; - Teaching narrative</li>
<li>&ldquo;The compiler warns&hellip; After a grace period&hellip;&rdquo; - Teaching explanation</li>
</ul>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Phase structure (Phase 1/2/3) - Documentation organization</li>
<li>Code examples showing producer, consumer, deprecation patterns</li>
<li>Inline comments in code (they&rsquo;re part of the code)</li>
</ul>
<p><strong>Word count removed</strong>: ~70 words</p>

  </div>
</div>
<h3 id="handling-breaking-changes">Handling breaking changes</h3>
<p><del>What if you need to rename a field? Say, <code>email</code> → <code>emailAddress</code>?</del></p>
<p><del><strong>Non-breaking approach</strong>:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add the new field, keep the old
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>              <span class="c1">// Old field
</span></span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New field
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>Wait, that&rsquo;s duplication. Better:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add alias constructor
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New canonical name
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">CustomerV3</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">fromV2</span><span class="o">(</span><span class="kt">v2:</span> <span class="kt">CustomerV2</span><span class="o">)</span><span class="kt">:</span> <span class="kt">CustomerV3</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">CustomerV3</span><span class="o">(</span><span class="n">v2</span><span class="o">.</span><span class="n">id</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">email</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">segment</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del><strong>Step 2</strong>: Migrate producers to write <code>emailAddress</code></del>
<del><strong>Step 3</strong>: Migrate consumers to read <code>emailAddress</code></del>
<del><strong>Step 4</strong>: Remove the <code>fromV2</code> constructor</del></p>
<p><del>At each step, compile-time contracts verify: &ldquo;Does this transformation produce the expected schema?&quot;</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE teaching questions and narrative</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through teaching questions and step labels, keep code examples.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;What if you need&hellip;?&rdquo; - Teaching rhetorical question</li>
<li>&ldquo;Non-breaking approach:&rdquo; - Teaching label</li>
<li>&ldquo;Wait, that&rsquo;s duplication. Better:&rdquo; - Teaching self-correction dialogue</li>
<li>&ldquo;Step 2/3/4&rdquo; labels - Teaching step-by-step instructions</li>
<li>&ldquo;At each step&hellip;&rdquo; - Teaching explanation</li>
</ul>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Code examples showing both approaches (duplication vs alias constructor)</li>
<li>Inline comments explaining field roles</li>
</ul>
<p><strong>Word count removed</strong>: ~45 words</p>

  </div>
</div>
<h3 id="coexistence-during-migration">Coexistence during migration</h3>
<p><del>During migration, you have both V2 and V3 running. How to handle?</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V3</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Transformation handles both
</span></span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="n">normalize</span><span class="o">(</span><span class="n">schema</span><span class="k">:</span> <span class="kt">CustomerSchema</span><span class="o">)</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=</span> <span class="n">schema</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">V2</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">V3</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">v3</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=&gt;</span> <span class="n">v3</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>The contract checks both paths:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V2</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span>  <span class="c1">// V2 → V3 allowed
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V3</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>     <span class="o">//</span> <span class="n">V3</span> <span class="o">→</span> <span class="n">V3</span> <span class="n">exact</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition warning">
  <div class="admonition-title">❌ REMOVE teaching question and explanation</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through teaching question and explanation, keep code.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;How to handle?&rdquo; - Teaching rhetorical question</li>
<li>&ldquo;The contract checks both paths:&rdquo; - Teaching explanation of what the code does</li>
</ul>
<p><strong>Word count removed</strong>: ~15 words</p>

  </div>
</div>
<h3 id="real-world-migration-backward--exact">Real-world migration: Backward → Exact</h3>
<p><del>Let&rsquo;s look at a complete migration scenario from the code in github. This is exactly how you&rsquo;d roll out schema changes in production.</del></p>
<p>From <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="cm">/** Sink contract: the target schema we promise to write. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerContract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Producer: imagine an upstream source adds an extra field `segment`. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerProducer</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Declared &#34;Next&#34; schema after a transform (dropping `segment`). */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerNext</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Step 1: Migration phase - use Backward policy
</span></span></span><span class="line"><span class="cl"><span class="c1">// Producer has extra field, but contract allows it during migration
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">srcB</span> <span class="k">=</span> <span class="nc">TypedSource</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">](</span><span class="s">&#34;csv&#34;</span><span class="o">,</span> <span class="n">inPath</span><span class="o">,</span> <span class="nc">Map</span><span class="o">(</span><span class="s">&#34;header&#34;</span> <span class="o">-&gt;</span> <span class="s">&#34;true&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">sinkB</span> <span class="k">=</span> <span class="nc">TypedSink</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="n">tmpOutB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">dropExtras</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span> <span class="k">_</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="n">$</span><span class="s">&#34;id&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;email&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;age&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planB</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;CSV -&gt; Parquet B: transformAs[CustomerNext], Exact&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">srcB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerNext</span><span class="o">](</span><span class="s">&#34;drop segment&#34;</span><span class="o">)(</span><span class="n">dropExtras</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sinkB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>### What&rsquo;s happening here:</del></p>
<p><del><strong>Phase 1: Producer adds field</strong></del>
<del>- Upstream adds <code>segment</code> field to <code>CustomerProducer</code></del>
<del>- Our contract is still <code>CustomerContract</code> (no <code>segment</code>)</del>
<del>- We use <code>transformAs[CustomerNext]</code> to explicitly drop the extra field</del>
<del>- Then check <code>CustomerNext</code> conforms to <code>CustomerContract</code> under <strong>Exact</strong></del></p>
<p><del><strong>Why this works:</strong></del>
<del>1. The transform explicitly declares the output schema (<code>CustomerNext</code>)</del>
<del>2. The compiler checks: Does <code>CustomerNext</code> match <code>CustomerContract</code>? Yes (both have id, email, age)</del>
<del>3. If we later change <code>CustomerNext</code> by accident, compilation fails</del></p>
<p><del><strong>Phase 2: Stabilization</strong></del>
<del>Once the migration is done:</del></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Later: Everyone uses CustomerContract, no transform needed
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planStable</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;stable&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">contractSource</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">noTransform</span>  <span class="c1">// Direct pass-through
</span></span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>This is real-world stuff. The code shows you exactly how to migrate schemas safely.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE &#39;What&#39;s happening here&#39; walkthrough</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through the entire &ldquo;What&rsquo;s happening here:&rdquo; section, keep code examples.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;What&rsquo;s happening here:&rdquo; - Teaching walkthrough structure</li>
<li>Bulleted explanations of each step</li>
<li>&ldquo;Why this works:&rdquo; - Teaching explanation with numbered breakdown</li>
<li>&ldquo;Once the migration is done:&rdquo; - Teaching narrative</li>
<li>&ldquo;This is real-world stuff. The code shows you&hellip;&rdquo; - Teaching enthusiasm</li>
</ul>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Code example showing <code>CustomerProducer</code> → <code>CustomerNext</code> transform</li>
<li>Code example showing stabilized pipeline with <code>noTransform</code></li>
<li>Mermaid diagram showing migration flow</li>
<li>Inline code comments</li>
</ul>
<p><strong>Word count removed</strong>: ~110 words</p>

  </div>
</div>
<pre class="mermaid">flowchart TD
    P1[&#34;Producer adds field &#39;segment&#39;&#34;] --&gt; T1[&#34;transformAs[CustomerNext]\n(drop extras)&#34;]
    T1 --&gt; V1[&#34;summon Conforms[CustomerNext,CustomerContract,Policy.Exact]&#34;]
    V1 --&gt; W1[&#34;write sink&#34;]

    subgraph Phase 1 [&#34;Migration window&#34;]
      direction TB
      P1 --&gt; T1 --&gt; V1 --&gt; W1
    end

    W1 --&gt; P2[&#34;Stabilize: sources match contract&#34;]
    P2 --&gt; V2[&#34;Conforms[CustomerContract,CustomerContract,Policy.Exact]&#34;]
    V2 --&gt; W2[&#34;noTransform + write&#34;]
</pre><hr>
<h2 id="part-13---testing-compiletime-for-real-copypaste-tests">Part 13 - Testing compile‑time, for real (copy/paste tests)</h2>
<p><del>How do you test that code fails to compile? With <code>assertDoesNotCompile</code>:</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 13 Testing patterns (remove teaching question)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 13 - it shows testing patterns for compile-time contracts.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Testing patterns with <code>assertDoesNotCompile</code> / <code>assertCompiles</code></li>
<li>Complete test suite example</li>
<li>CI/CD integration (GitHub Actions YAML)</li>
<li>Mermaid diagram showing test flow</li>
<li>Production testing best practices</li>
</ul>
<p><strong>Why remove some content</strong>:</p>
<ul>
<li>&ldquo;How do you&hellip;?&rdquo; - Teaching rhetorical question</li>
<li>&ldquo;This is gold&rdquo; - Teaching enthusiasm</li>
</ul>
<p><strong>Strategy</strong>: Strike through teaching questions and enthusiasm, keep all test code and CI config.</p>

  </div>
</div>
<pre class="mermaid">sequenceDiagram
    participant Dev
    participant CI as &#34;GitHub Actions&#34;
    participant SBT as &#34;sbt testOnly *CompileTimeSpec&#34;
    Dev-&gt;&gt;CI: open PR
    CI-&gt;&gt;SBT: run compile-fail suite
    SBT-&gt;&gt;SBT: assertCompiles / assertDoesNotCompile
    alt all pass
        SBT--&gt;&gt;CI: success
        CI--&gt;&gt;Dev: green check
    else drift detected
        SBT--&gt;&gt;CI: failure with errorAndAbort msg
        CI--&gt;&gt;Dev: red X + compiler diff
    end
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/CompileTimeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">org.scalatest.funsuite.AnyFunSuite</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CompileTimeSpec</span> <span class="k">extends</span> <span class="nc">AnyFunSuite</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">test</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Exact</span> <span class="kt">should</span> <span class="kt">fail</span> <span class="kt">when</span> <span class="kt">a</span> <span class="kt">required</span> <span class="kt">field</span> <span class="kt">is</span> <span class="kt">missing</span><span class="err">&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="kt">assertDoesNotCompile</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span><span class="err">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Backward should pass for the same mismatch&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OutMissing, Contract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;By‑Position should accept re‑ordered names with same types&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactByPosition]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Ordered should reject the same re‑ordering&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactOrdered]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Pipeline builder enforces correct order&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">      PipelineBuilder[Contract](&#34;bad&#34;)
</span></span></span><span class="line"><span class="cl"><span class="s">        .transformAs[Next](identity)  // Can&#39;t transform without source
</span></span></span><span class="line"><span class="cl"><span class="s">    &#34;&#34;&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Nested types conform correctly&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OrderOut, OrderContract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>You can wire this into CI:</p>
<div class="highlight highlight-yaml" data-lang="yaml"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/compile-fail.yml (concept)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">compile-fail</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">pull_request]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cf</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-java@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">distribution</span><span class="p">:</span><span class="w"> </span><span class="nt">temurin, java-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;17&#39;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Compile‑fail suite</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">sbt -v &#34;testOnly *CompileTimeSpec&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><p><del>This is gold. Your CI literally proves that broken schemas can&rsquo;t be deployed.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE teaching enthusiasm</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through enthusiastic closing statement.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;This is gold&rdquo; - Teaching enthusiasm</li>
<li>&ldquo;Your CI literally proves&hellip;&rdquo; - Teaching explanation of benefit</li>
<li>Resource readers can understand CI value from the test code and YAML config</li>
</ul>
<p><strong>Word count removed</strong>: ~12 words</p>

  </div>
</div>
<hr>
<h2 id="part-14---scala-2-how-to-do-the-same-and-whats-different">Part 14 - Scala 2: how to do the same (and what&rsquo;s different)</h2>
<p><del>Scala 2 uses def macros with a <code>Context</code> (blackbox/whitebox). The idea is identical, but the implementation uses <code>c.universe</code> trees instead of <code>Expr</code>/quotes.</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 14 Scala 2 implementation (remove teaching preamble)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 14 - it shows Scala 2 variant of the API.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Scala 2 macro implementation (API variant for different Scala version)</li>
<li>Complete code showing blackbox Context approach</li>
<li>Key differences table (Scala 3 vs Scala 2)</li>
<li>Reference to Magnolia for cross-version derivation</li>
</ul>
<p><strong>Why remove some content</strong>:</p>
<ul>
<li>Opening sentence explaining &ldquo;The idea is identical&hellip;&rdquo; - Teaching comparison</li>
<li>&ldquo;Scala 3 macros are safer&hellip;&rdquo; - Teaching judgment about trade-offs</li>
</ul>
<p><strong>Strategy</strong>: Strike through teaching comparisons, keep code and differences table.</p>

  </div>
</div>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// build.sbt (S2 module)
</span></span></span><span class="line"><span class="cl"><span class="c1">// scalaVersion := &#34;2.13.14&#34;
</span></span></span><span class="line"><span class="cl"><span class="c1">// libraryDependencies += &#34;org.scala-lang&#34; % &#34;scala-reflect&#34; % scalaVersion.value
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// src/main/scala-2/com/example/ConformsS2.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.reflect.macros.blackbox</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">class</span> <span class="nc">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">object</span> <span class="nc">ConformsS2</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">implicit</span> <span class="k">def</span> <span class="n">materialize</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">](</span><span class="k">implicit</span> <span class="n">sOut</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="n">sCon</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">macro</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">c.WeakTypeTag</span>, <span class="kt">Contract:</span> <span class="kt">c.WeakTypeTag</span>, <span class="kt">P:</span> <span class="kt">c.WeakTypeTag</span><span class="o">](</span><span class="n">c</span><span class="k">:</span> <span class="kt">blackbox.Context</span><span class="o">)(</span><span class="n">sOut</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">]],</span> <span class="n">sCon</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]])</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">c.universe._</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">fieldsOf</span><span class="o">[</span><span class="kt">T:</span> <span class="kt">c.WeakTypeTag</span><span class="o">](</span><span class="n">sh</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">T</span><span class="o">]])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span>,<span class="kt">Boolean</span>,<span class="kt">Boolean</span><span class="o">)]</span> <span class="k">=</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="n">c</span><span class="o">.</span><span class="n">eval</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="nc">Expr</span><span class="o">[</span><span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span>,<span class="kt">Boolean</span>,<span class="kt">Boolean</span><span class="o">)]](</span><span class="n">q</span><span class="s">&#34;$sh.fields.map(f =&gt; (f.name, f.tpe, f.hasDefault, f.isOptional))&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">fieldsOf</span><span class="o">[</span><span class="kt">Out</span><span class="o">](</span><span class="n">sOut</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">fieldsOf</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="n">sCon</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">outMap</span>      <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contractMap</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">missing</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">outMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s">:</span><span class="si">$t$</span><span class="s">{if (opt) &#34;</span> <span class="o">(</span><span class="n">optional</span><span class="o">)</span><span class="s">&#34; else &#34;&#34;}&#34;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">extra</span>   <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span>      <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">contractMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mism</span>    <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span> <span class="o">!=</span> <span class="n">t</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> expected </span><span class="si">$t</span><span class="s">, found </span><span class="si">${</span><span class="n">outMap</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">_1</span><span class="si">}</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">o</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">!=</span> <span class="n">o</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> optionality mismatch&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="o">(</span><span class="n">missing</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">extra</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">mism</span><span class="o">.</span><span class="n">nonEmpty</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="n">c</span><span class="o">.</span><span class="n">abort</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="n">enclosingPosition</span><span class="o">,</span> <span class="s">s&#34;Drift: missing=</span><span class="si">${</span><span class="n">missing</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;,&#34;</span><span class="o">)</span><span class="si">}</span><span class="s"> extra=</span><span class="si">${</span><span class="n">extra</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;,&#34;</span><span class="o">)</span><span class="si">}</span><span class="s"> mism=</span><span class="si">${</span><span class="n">mism</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;;&#34;</span><span class="o">)</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span>
</span></span><span class="line"><span class="cl">      <span class="n">c</span><span class="o">.</span><span class="nc">Expr</span><span class="o">[</span><span class="kt">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]](</span><span class="n">q</span><span class="s">&#34;new _root_.com.example.ConformsS2[${weakTypeOf[Out]}, ${weakTypeOf[Contract]}, ${weakTypeOf[P]}]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Key differences:</strong></p>
<ul>
<li>Scala 3: <code>inline</code> + <code>${ ... }</code> splices; Scala 2: <code>macro def</code> with a <code>Context</code></li>
<li>Error reporting: <code>report.errorAndAbort</code> (Scala 3) vs <code>c.abort</code> (Scala 2)</li>
<li>Trees: <code>Expr[T]</code> (Scala 3) vs raw <code>Tree</code> (Scala 2)</li>
<li>Type inspection: <code>TypeRepr</code> (Scala 3) vs <code>Type</code> (Scala 2)</li>
</ul>
<p><del>Scala 3 macros are safer. The quoted API prevents many common macro bugs. But Scala 2 macros work fine if you&rsquo;re careful.</del></p>
<div class="admonition warning">
  <div class="admonition-title">❌ REMOVE teaching judgment</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Strike through teaching comparison about safety.</p>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;Scala 3 macros are safer&rdquo; - Teaching judgment/opinion</li>
<li>&ldquo;But&hellip; if you&rsquo;re careful&rdquo; - Teaching advice</li>
<li>Resource readers can see differences in the table above</li>
</ul>
<p><strong>Word count removed</strong>: ~20 words</p>

  </div>
</div>
<p>For derivation, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works across both versions.</p>
<hr>
<h2 id="part-15---integration-with-spark-runtime-defense-in-depth">Part 15 - Integration with Spark: Runtime defense-in-depth</h2>
<p><del>The code in github also shows how to mirror compile-time policies with Spark&rsquo;s built-in structural comparators for runtime validation.</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 15 Spark integration (remove teaching preamble)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 15 - it shows integration pattern with Spark.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Integration pattern with Spark StructType</li>
<li>Runtime policy mapping code</li>
<li>Two-layer validation strategy (compile-time + runtime)</li>
<li>Mermaid diagram showing policy mapping</li>
<li>Reference to Spark DataType API docs</li>
</ul>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;The code in github also shows how&hellip;&rdquo; - Teaching narrative introduction</li>
</ul>
<p><strong>Strategy</strong>: Strike through teaching introduction, keep all integration code and strategy.</p>

  </div>
</div>
<p>From <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Derive Spark StructType from case class at compile time
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">struct</span><span class="k">:</span> <span class="kt">StructType</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">SparkSchema</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">derived</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="k">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Macro that converts case class → StructType
</span></span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]]</span> <span class="k">=</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Runtime policy mapping to Spark comparators
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyRuntime</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsIgnoreCaseAndNullability</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactByPosition.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurally</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="n">ignoreNullability</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactOrdered.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurallyByName</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="k">_</span> <span class="o">==</span> <span class="k">_</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Two-layer validation strategy</strong>:</p>
<ol>
<li><strong>Compile-time</strong> (macros) - Catches schema drift before deployment</li>
<li><strong>Runtime</strong> (Spark comparators) - Defensive check for external data sources</li>
</ol>
<pre class="mermaid">flowchart TD
    F[&#34;found: StructType&#34;] --- E[&#34;expected: StructType&#34;]
    F --&gt; EX[&#34;Policy.Exact&#34;] --&gt; C1[&#34;equalsIgnoreCaseAndNullability(found,expected)&#34;]
    F --&gt; EP[&#34;Policy.ExactByPosition&#34;] --&gt; C2[&#34;equalsStructurally(found,expected,true)&#34;]
    F --&gt; EO[&#34;Policy.ExactOrdered&#34;] --&gt; C3[&#34;equalsStructurallyByName(found,expected,resolver)&#34;]
</pre><p>For more on Spark&rsquo;s structural comparators, see the <a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">DataType API docs</a>
.</p>
<hr>
<h2 id="part-16---a-tiny-migration-playbook-you-can-adopt-tomorrow">Part 16 - A tiny migration playbook you can adopt tomorrow</h2>
<p><del>Here&rsquo;s how I use these policies in practice:</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 16 Migration playbook (remove personal framing)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 16 - it&rsquo;s production best practices and migration playbook.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Concrete policy usage patterns (Critical paths → Exact, etc.)</li>
<li>Production best practices (pure transformations, I/O at edges, idempotency)</li>
<li>Pithy summary quote about compile-time vs runtime</li>
</ul>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;Here&rsquo;s how I use&hellip;&rdquo; - Personal/teaching framing</li>
<li>&ldquo;And remember:&rdquo; - Teaching instruction tone</li>
<li>&ldquo;you can adopt tomorrow&rdquo; in heading - Teaching promise</li>
</ul>
<p><strong>Strategy</strong>: Strike through personal framing, keep all technical guidance.</p>

  </div>
</div>
<ul>
<li><strong>Critical paths → Exact</strong>. No surprises. If schemas don&rsquo;t match exactly, builds fail.</li>
<li><strong>Producer adding optional fields → Backward</strong> during rollout. Allows new optional fields on the producer side.</li>
<li><strong>Consumer that can ignore extras → Forward</strong> during rollout. Lets the consumer tolerate extra fields.</li>
<li><strong>After stabilization → tighten back to Exact</strong>.</li>
</ul>
<p><del>And remember:</del></p>
<ul>
<li>Keep transformations pure (no side effects inside map/filter)</li>
<li>Place I/O at edges (read once, write once)</li>
<li>Make side-effects idempotent (so retries don&rsquo;t duplicate writes)</li>
</ul>
<blockquote>
  <p>Compile-time prevents drift; runtime manages reality. Tests catch behavior, not shapes.</p>

</blockquote><hr>
<h2 id="part-17---production-patterns-what-i-learned-shipping-this">Part 17 - Production patterns: <del>What I learned shipping this</del></h2>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Part 17 Production patterns (remove personal framing)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Part 17 100% - this is RESOURCE GOLD per FINAL-EXECUTION-PLAN.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Pattern 1: Contract organization by version</li>
<li>Pattern 2: Schema caching with companion objects</li>
<li>Pattern 3: Inline policy documentation</li>
<li>Pattern 4: Version-controlled compile-fail tests</li>
<li>All patterns are concrete, production-ready best practices</li>
</ul>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;What I learned shipping this&rdquo; - Personal teaching narrative in heading</li>
</ul>
<p><strong>Strategy</strong>: Strike through personal framing in heading only, keep all 4 patterns intact.</p>

  </div>
</div>
<h3 id="pattern-1-organize-contracts-by-version">Pattern 1: Organize contracts by version</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// pipelines/CustomerMigration.scala
</span></span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">contracts._</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">migration</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;v1-to-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV1</span><span class="o">](</span><span class="n">sourceV1</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addNameField</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sinkV2</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-2-use-companion-objects-for-schema-caching">Pattern 2: Use companion objects for schema caching</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">User</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>  <span class="o">//</span> <span class="nc">Compute</span> <span class="n">once</span><span class="o">,</span> <span class="n">reuse</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-3-document-policy-choices-inline">Pattern 3: Document policy choices inline</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Good: Explicit reasoning
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Allow optional fields during Q2 migration
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Bad: No context
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Full</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">Why</span> <span class="nc">Full</span><span class="o">?</span> <span class="nc">When</span> <span class="n">will</span> <span class="n">we</span> <span class="n">tighten</span><span class="o">?</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-4-test-your-compile-fail-cases">Pattern 4: Test your compile-fail cases</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Keep these in version control as documentation
</span></span></span><span class="line"><span class="cl"><span class="n">test</span><span class="o">(</span><span class="s">&#34;CustomerV1 should not conform to CustomerV2 under Exact&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;summon[Conforms[CustomerV1, CustomerV2, Policy.Exact]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><hr>
<h2 id="appendix---full-demo-project-skeleton-copy--run">Appendix - Full demo project skeleton (copy &amp; run)</h2>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Appendix - Runnable project skeleton</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep Appendix - runnable code skeleton is perfect Resource content.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Complete project directory structure</li>
<li>build.sbt configuration for Scala 3</li>
<li>Run command</li>
<li>Tip for Scala 2 variant</li>
</ul>
<p><strong>No removals needed</strong>: This section is pure Resource content (copy-paste-ready setup).</p>

  </div>
</div>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">├── build.sbt
</span></span><span class="line"><span class="cl">└── src
</span></span><span class="line"><span class="cl">    ├── main
</span></span><span class="line"><span class="cl">    │   └── scala
</span></span><span class="line"><span class="cl">    │       └── com/example/
</span></span><span class="line"><span class="cl">    │           ├── IntroMacro.scala
</span></span><span class="line"><span class="cl">    │           ├── Shape.scala
</span></span><span class="line"><span class="cl">    │           ├── Policy.scala
</span></span><span class="line"><span class="cl">    │           ├── Conforms.scala
</span></span><span class="line"><span class="cl">    │           └── SparkCore.scala
</span></span><span class="line"><span class="cl">    └── test
</span></span><span class="line"><span class="cl">        └── scala
</span></span><span class="line"><span class="cl">            └── com/example/
</span></span><span class="line"><span class="cl">                ├── ShapeSpec.scala
</span></span><span class="line"><span class="cl">                ├── ConformsSpec.scala
</span></span><span class="line"><span class="cl">                └── CompileTimeSpec.scala</span></span></code></pre></td></tr></table>
</div>
</div><p><strong>build.sbt (Scala 3)</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">scalaVersion</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;3.3.3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organization</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;com.example&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.apache.spark&#34;</span> <span class="o">%%</span> <span class="s">&#34;spark-sql&#34;</span> <span class="o">%</span> <span class="s">&#34;3.5.1&#34;</span> <span class="o">%</span> <span class="s">&#34;provided&#34;</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.scalatest&#34;</span> <span class="o">%%</span> <span class="s">&#34;scalatest&#34;</span> <span class="o">%</span> <span class="s">&#34;3.2.17&#34;</span> <span class="o">%</span> <span class="nc">Test</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Run it:</p>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><blockquote>
  <p>Tip: For Scala 2, add a sibling module with scala‑reflect and copy the S2 file from above.</p>

</blockquote><hr>
<h2 id="references">References</h2>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP References - Essential links</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep References section - essential Resource content.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Curated links to Scala 3 macros resources (Rock the JVM, official docs)</li>
<li>Apache Spark integration documentation</li>
<li>Phantom types articles</li>
<li>Magnolia library reference</li>
</ul>
<p><strong>No removals needed</strong>: Reference sections are pure Resource content.</p>

  </div>
</div>
<h3 id="scala-3-macros--metaprogramming">Scala 3 Macros &amp; Metaprogramming</h3>
<ul>
<li><a href="https://rockthejvm.com/courses/scala-macros-and-metaprogramming" rel="noopener external" target="_blank">Rock the JVM - Scala Macros and Metaprogramming (course)</a>
</li>
<li><a href="https://rockthejvm.com/articles/scala-3-macros-comprehensive-guide" rel="noopener external" target="_blank">Rock the JVM - Scala 3 Macros comprehensive guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 Macros - Official overview</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/reflection.html" rel="noopener external" target="_blank">Scala 3 Reflection guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">Scala 3 Macro best practices</a>
</li>
</ul>
<h3 id="apache-spark-integration">Apache Spark Integration</h3>
<ul>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">Spark DataType structural comparators</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/DatasetHolder.html" rel="noopener external" target="_blank">Spark Dataset implicits (toDF/toDS)</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/sql-data-sources-csv.html" rel="noopener external" target="_blank">Spark CSV data source</a>
</li>
<li><a href="https://index.scala-lang.org/vincenzobaz/spark-scala3-encoders" rel="noopener external" target="_blank">Scala 3 encoders for Spark (community)</a>
</li>
</ul>
<h3 id="phantom-types--type-level-programming">Phantom Types &amp; Type-Level Programming</h3>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala - Builder Pattern (Rhetorical Musings)</a>
</li>
</ul>
<h3 id="other-resources">Other Resources</h3>
<ul>
<li><a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia - Generic derivation library</a>
</li>
</ul>
<hr>
<h2 id="what-compile-time-contracts-cant-catch">What compile-time contracts can&rsquo;t catch</h2>
<p><del>Let&rsquo;s be honest about the boundaries. Compile-time contracts handle a lot, but they&rsquo;re not magic. Here&rsquo;s what they <strong>can&rsquo;t</strong> do:</del></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP Limitations table (remove teaching preamble)</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep limitations/gotchas table - critical Resource content.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Comprehensive table of what compile-time contracts can&rsquo;t catch</li>
<li>Categories: External data drift, late-arriving changes, data quality, partial failures</li>
<li>Solutions column provides runtime validation alternatives</li>
<li>&ldquo;The takeaway&rdquo; summary about defense-in-depth</li>
</ul>
<p><strong>Why remove</strong>:</p>
<ul>
<li>&ldquo;Let&rsquo;s be honest about the boundaries&rdquo; - Teaching conversational tone</li>
<li>&ldquo;they&rsquo;re not magic&rdquo; - Teaching demystification</li>
</ul>
<p><strong>Strategy</strong>: Strike through teaching preamble, keep table and takeaway.</p>

  </div>
</div>
<table>
	<thead>
			<tr>
					<th>Category</th>
					<th>What it can&rsquo;t catch</th>
					<th>Why</th>
					<th>Solution</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><strong>External data drift</strong></td>
					<td>• Third-party APIs that change without telling you; • Kafka topics from teams you don&rsquo;t coordinate with; • S3 buckets written by external vendors</td>
					<td>You don&rsquo;t control their build process</td>
					<td>Use runtime validation (schema registries, data contracts)</td>
			</tr>
			<tr>
					<td><strong>Late-arriving schema changes</strong></td>
					<td>• Schema registry updated after your deploy; • Database columns added while your job runs</td>
					<td>Compile-time happens before deploy</td>
					<td>Runtime checks with version monitoring</td>
			</tr>
			<tr>
					<td><strong>Data quality issues</strong></td>
					<td>• Nulls where you expect values (even if field is non-optional); • Out-of-range numbers (age = -5); • Malformed strings (email without @)</td>
					<td>Compile-time checks structure, not content</td>
					<td>Great Expectations, Deequ, or custom validators</td>
			</tr>
			<tr>
					<td><strong>Non-breaking additions</strong></td>
					<td>• Upstream adds optional field you don&rsquo;t use yet</td>
					<td>This is often fine! <code>Forward</code> policy handles it</td>
					<td>Policy-based awareness or stricter monitoring</td>
			</tr>
			<tr>
					<td><strong>Partial batch failures</strong></td>
					<td>• 1000 records match schema, 5 don&rsquo;t</td>
					<td>Compile-time is binary (compiles or doesn&rsquo;t)</td>
					<td>Runtime validation with error tables/quarantine</td>
			</tr>
	</tbody>
</table>
<p><strong>The takeaway</strong>: Compile-time contracts catch drift <strong>within your controlled codebase</strong> during development and deployment. For everything else - external sources, runtime changes, data quality - layer in runtime validation. Think defense-in-depth: compile-time as the first gate, runtime as the safety net.</p>
<hr>
<p><strong>TL;DR</strong></p>
<div class="admonition success">
  <div class="admonition-title">⭕ KEEP TL;DR Summary</div>
  <div class="admonition-content">
    <p><strong>Decision</strong>: Keep TL;DR - concise summary is perfect Resource content.</p>
<p><strong>Why keep</strong>:</p>
<ul>
<li>Bulleted summary of key concepts and patterns</li>
<li>No teaching narrative, just technical takeaways</li>
<li>Covers: compile-time evidence, TypeInspector, phantom types, optionality, migrations, Spark integration, limitations</li>
<li>Each bullet is a concrete technical point</li>
</ul>
<p><strong>No removals needed</strong>: This is already concise Resource-style summary.</p>

  </div>
</div>
<ul>
<li>Compile-time evidence + policy types make schema intent explicit and enforceable</li>
<li>TypeInspector pattern organizes type inspection into composable utilities</li>
<li>Phantom types with type-indexing/type-state builders create impossible-to-misuse APIs through compile-time state machines</li>
<li>Field vs element optionality matters for nested types</li>
<li>Nested structures (Maps, Lists, case classes) work through recursive shape building</li>
<li>Real migrations: use Backward during rollout, tighten to Exact after stabilization</li>
<li>Spark&rsquo;s comparators provide runtime defense-in-depth</li>
<li>Schema evolution needs versioning, coexistence strategies, and policy-based transitions</li>
<li>Developer ergonomics matter: clear errors, fast compile times, easy onboarding</li>
<li>Compile-time doesn&rsquo;t replace runtime validation - it complements it for controlled systems</li>
<li>This catches many drift classes early, but you still need runtime checks for external data</li>
<li>Compile-time contracts don&rsquo;t make pipelines unbreakable - they make breakage predictable</li>
</ul>
]]></content:encoded><enclosure url="https://vitthalmirji.com/images/og/2025-09-compile-time-data-contracts.png" type="image/png"/></item><item><title>Compile-time data contracts: Catch schema mismatches at compile time in Scala 3</title><link>https://vitthalmirji.com/2025/09/compile-time-data-contracts-catch-schema-mismatches-at-compile-time-in-scala-3/</link><guid isPermaLink="true">https://vitthalmirji.com/2025/09/compile-time-data-contracts-catch-schema-mismatches-at-compile-time-in-scala-3/</guid><pubDate>Tue, 30 Sep 2025 09:00:00 +0000</pubDate><dc:creator>vitthal-mirji</dc:creator><category>scala</category><category>scala-3</category><category>macros</category><category>metaprogramming</category><category>compile-time</category><category>type-safety</category><category>data-contracts</category><category>schema-validation</category><category>data-engineering</category><category>apache-spark</category><description>Build Scala compile-time data contracts with macros and TypeInspector patterns so schema drift fails at compile time, not midnight in prod. Includes patterns for required vs optional fields and compile errors that block bad data early.</description><content:encoded><![CDATA[<p>What if broken pipelines never launched - because the compiler stopped them?</p>
<blockquote>
  <p>&ldquo;If it compiles, contracts align.&rdquo;</p>

</blockquote><p>I&rsquo;ve been working on this problem for a while now. You know how it goes - you&rsquo;re running a data pipeline, everything&rsquo;s fine, then someone changes a field name upstream and boom, your job crashes at 2 AM. The on-call engineer (probably you) gets paged, and you spend the next hour figuring out what went wrong.</p>
<p>Once, an upstream team silently renamed a column <code>amount</code> → <code>amt</code>. Our pipeline didn&rsquo;t crash immediately - it happily wrote nulls for weeks. By the time we caught it, we had to backfill millions of records. That&rsquo;s the day I realized: runtime validation catches problems late; compile-time contracts catch them earlier.</p>
<p>This post shows you how to catch these issues at compile time. Not at runtime. Not even at test time. At compile time. Your code literally won&rsquo;t build if schemas drift.</p>
<p>We&rsquo;ll build this from scratch - first understand how Scala 3 macros work, then write a minimal contract system that proves schema compatibility before your code even runs. I&rsquo;ll also show the Scala 2 version because, let&rsquo;s be honest, many teams are still on Scala 2.</p>
<p>All code here runs in a vanilla sbt project. You can grab the working code in github from <code>https://github.com/vim89/compile-time-data-contracts</code> and run it yourself.</p>
<h2 id="why-this-matters">Why this matters</h2>
<p>Look, I get it. Writing tests is important. But tests are samples. They check what you think might break. The compiler, though? It&rsquo;s exhaustive for structure. It can prove your schemas match - every field, every type, every time.</p>
<p>Here&rsquo;s the thing - schemas change all the time in real projects. And when they do:</p>
<ul>
<li>Your build should fail early, not your production job</li>
<li>Keep transformations pure and effects at the edges (so retries don&rsquo;t duplicate writes)</li>
<li>Let the compiler do the heavy lifting</li>
</ul>
<p><strong>Important upfront:</strong> This focuses on preventing schema drift within your controlled boundaries - the systems you build, deploy, and version. It doesn&rsquo;t replace runtime validation for external data sources or third-party APIs. Think of it as catching a large class of schema mismatches during development and deployment, not eliminating all drift everywhere. You&rsquo;ll still need runtime checks for data coming from systems you don&rsquo;t control.</p>
<div class="admonition warning">
  <div class="admonition-title">Scope boundary</div>
  <div class="admonition-content">
    Compile-time contracts catch drift in your controlled codebase - the transforms you write, the case classes you define, the schemas you version together. They won&rsquo;t catch upstream API changes, late-arriving schema registry updates, or external data sources that change independently. Use compile-time for what you control, runtime validation for everything else. Defense in depth, not either/or.
  </div>
</div>
<h2 id="what-well-build">What we&rsquo;ll build</h2>
<p>We&rsquo;re going to build a small compile-time validation system:</p>
<ol>
<li>A way to describe case class structure at compile time (we&rsquo;ll call it &ldquo;shape&rdquo;)</li>
<li>A policy system - Exact, Backward, Forward compatibility modes</li>
<li>A macro that proves &ldquo;these two shapes conform under policy P&rdquo; - or refuses to compile</li>
<li>A phantom-typed pipeline builder that enforces correct construction order</li>
</ol>
<p>First, Scala 3 (using inline + quotes), then Scala 2 (using blackbox macros). I&rsquo;ll point out the differences as we go.</p>
<h3 id="run-the-code-in-github">Run the code in github</h3>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone https://github.com/vim89/compile-time-data-contracts.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> compile-time-data-contracts
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Scala 3</span>
</span></span><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><div class="admonition info">
  <div class="admonition-title">Information</div>
  <div class="admonition-content">
    The code in github includes working examples you can run immediately. All compile-fail cases are documented inline - uncomment them to see the compiler errors.
  </div>
</div>
<hr>
<h2 id="part-1---scala-3-the-mental-model-inline--quotes">Part 1 - Scala 3: the mental model (inline + quotes)</h2>
<p>Scala 3 macros are different from Scala 2. Better, actually. They&rsquo;re safer and easier to reason about.</p>
<p>Two main ideas:</p>
<ul>
<li><code>inline def</code>: tells the compiler &ldquo;expand this at compile time&rdquo;</li>
<li><code>quotes</code> + <code>Expr[T]</code>: gives you a typed AST that you can inspect</li>
</ul>
<pre class="mermaid">sequenceDiagram
    actor Dev
    participant Compiler
    participant Macro as &#34;Conforms.impl&#34;
    participant Reflect as &#34;TypeRepr/quotes&#34;
    Dev-&gt;&gt;Compiler: compile
    Compiler-&gt;&gt;Macro: splice ${ ... }
    Macro-&gt;&gt;Reflect: inspect types
    Reflect--&gt;&gt;Macro: shapes for Out and Contract
    alt conforming
        Macro--&gt;&gt;Compiler: emit Conforms[Out,Contract,P]
        Compiler--&gt;&gt;Dev: build succeeds
    else drift
        Macro--&gt;&gt;Compiler: report.errorAndAbort(msg)
        Compiler--&gt;&gt;Dev: compile error with drift details
    end
</pre><p>Here&rsquo;s a minimal macro to see how it works:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// build.sbt: use Scala 3
</span></span></span><span class="line"><span class="cl"><span class="c1">// scalaVersion := &#34;3.3.3&#34;
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/IntroMacro.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.quoted.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">IntroMacro</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">def</span> <span class="kt">show</span><span class="o">(</span><span class="kt">inline</span> <span class="kt">x:</span> <span class="kt">Any</span><span class="o">)</span><span class="kt">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">showImpl</span><span class="o">(</span>&#39;x<span class="o">)</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">showImpl</span><span class="o">(</span><span class="n">x</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Any</span><span class="o">])(</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">Expr</span><span class="o">(</span><span class="s">s&#34;You passed: </span><span class="si">${</span><span class="n">x</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>What&rsquo;s happening here?</p>
<ul>
<li>The <code>inline</code> method is what users call</li>
<li>The <code>${ ... }</code> splice calls the implementation</li>
<li>The implementation gets <code>'x</code> (quoted code) as <code>Expr[Any]</code></li>
<li>We return <code>Expr[String]</code> - the AST for what the compiler will embed</li>
</ul>
<p>Try it:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Demo.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">run</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">println</span><span class="o">(</span><span class="nc">IntroMacro</span><span class="o">.</span><span class="n">show</span><span class="o">(</span><span class="mi">1</span> <span class="o">+</span> <span class="mi">2</span><span class="o">))</span> <span class="o">//</span> <span class="n">prints</span><span class="k">:</span> <span class="kt">You</span> <span class="kt">passed:</span> <span class="err">1</span><span class="kt">.+</span><span class="o">(</span><span class="err">2</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>See how it printed the AST of <code>1 + 2</code>? That&rsquo;s the power here - we can inspect code structure at compile time.</p>
<p>Check out the <a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 macros overview</a>
 and <a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">best practices</a>
 for more details.</p>
<div class="admonition tip">
  <div class="admonition-title">Quick win</div>
  <div class="admonition-content">
    If you&rsquo;re new to Scala 3 macros, start by reading macro-generated code in the REPL: sbt&gt; console, then scala&gt; import scala.quoted.*, then inspect what your macros actually expand to. Understanding the generated code is 80% of debugging macro issues. The compiler&rsquo;s doing the work - you just need to see what it&rsquo;s actually producing.
  </div>
</div>
<hr>
<h2 id="part-2---describe-shapes-at-compile-time-scala-3">Part 2 - Describe shapes at compile time (Scala 3)</h2>
<p>Now we need to describe case class structure. In real projects, you might use Shapeless, Magnolia, or Scala 3&rsquo;s Mirror. I&rsquo;ll show a compact Mirror-based version here because it&rsquo;s self-contained.</p>
<p>The idea: a typeclass <code>Shape[A]</code> that knows the fields of <code>A</code>.</p>
<pre class="mermaid">classDiagram
    class Field {
      +name: String
      +tpe: String
      +hasDefault: Boolean
      +isOptional: Boolean
    }
    class Shape {
      +fields(): List~Field~
    }
    Shape &#34;1&#34; --&gt; &#34;*&#34; Field : returns
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Shape.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.deriving.Mirror</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">tpe</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">hasDefault</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Shape</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>     <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="o">[</span><span class="kt">A</span><span class="o">]</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">A</span><span class="o">]]</span> <span class="k">with</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span> <span class="c1">// element optionality handled elsewhere
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="n">given</span> <span class="n">derived</span><span class="o">[</span><span class="kt">A</span><span class="o">](</span><span class="n">using</span> <span class="n">m</span><span class="k">:</span> <span class="kt">Mirror.Of</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">m</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">p</span><span class="k">:</span> <span class="kt">Mirror.ProductOf</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">labels</span> <span class="k">=</span> <span class="n">constValueTuple</span><span class="o">[</span><span class="kt">p.MirroredElemLabels</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">types</span>  <span class="k">=</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">p.MirroredElemTypes</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">zipped</span> <span class="k">=</span> <span class="n">zip</span><span class="o">(</span><span class="n">labels</span><span class="o">,</span> <span class="n">types</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">          <span class="k">def</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Field</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">            <span class="n">zipped</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">Field</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">tpe</span><span class="o">,</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="kc">false</span><span class="o">,</span> <span class="n">isOptional</span> <span class="k">=</span> <span class="n">tpe</span><span class="o">.</span><span class="n">startsWith</span><span class="o">(</span><span class="s">&#34;Option[&#34;</span><span class="o">))</span> <span class="o">}.</span><span class="n">toList</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="k">new</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]</span> <span class="o">{</span> <span class="k">def</span> <span class="n">fields</span> <span class="k">=</span> <span class="nc">Nil</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">import</span> <span class="nn">scala.compiletime.</span><span class="o">{</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">constValueTuple</span><span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">typeNames</span><span class="o">[</span><span class="kt">T</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">inline</span> <span class="n">erasedValue</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">    <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">h</span><span class="o">]</span> <span class="k">:</span><span class="kt">:</span> <span class="k">type</span><span class="kt">Names</span><span class="o">[</span><span class="kt">t</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">summonTypeName</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">constValue</span><span class="o">[</span><span class="err">&#34;&#34;</span> <span class="kt">+</span> <span class="kt">T</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="n">inline</span> <span class="k">def</span> <span class="n">zip</span><span class="o">[</span><span class="kt">L</span> <span class="k">&lt;:</span> <span class="kt">Tuple</span>, <span class="kt">R</span> <span class="k">&lt;:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">labels</span><span class="k">:</span> <span class="kt">L</span><span class="o">,</span> <span class="n">types</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">inline</span> <span class="n">labels</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="kt">EmptyTuple</span> <span class="o">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_:</span> <span class="o">(</span><span class="kt">h</span> <span class="kt">*:</span> <span class="kt">t</span><span class="o">)</span>   <span class="o">=&gt;</span> <span class="n">constValue</span><span class="o">[</span><span class="kt">h</span><span class="o">].</span><span class="n">asInstanceOf</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">-&gt;</span> <span class="n">types</span><span class="o">.</span><span class="n">head</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">zip</span><span class="o">[</span><span class="kt">t</span>, <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]](</span><span class="n">erasedValue</span><span class="o">,</span> <span class="n">types</span><span class="o">.</span><span class="n">tail</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Yeah, it&rsquo;s a bit dense. But here&rsquo;s what it does:</p>
<ul>
<li>Uses Scala 3&rsquo;s Mirror to reflect on case classes at compile time</li>
<li>Extracts field names and types</li>
<li>Marks optional fields (those wrapped in <code>Option</code>)</li>
<li>Returns a <code>List[Field]</code> describing the structure</li>
</ul>
<pre class="mermaid">flowchart LR
    A[&#34;Case class A(...)&#34;] --&gt; M[&#34;Mirror.Of[A]&#34;]
    M --&gt; L[&#34;MirroredElemLabels&#34;]
    M --&gt; T[&#34;MirroredElemTypes&#34;]
    L --&gt; Z[&#34;zip(labels, types)&#34;]
    T --&gt; Z
    Z --&gt; S[&#34;Shape[A].fields: List[Field]&#34;]
</pre><p>Usage:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ShapeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">note</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkShape</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="n">println</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">fields</span><span class="o">)</span> <span class="o">//</span> <span class="nc">List</span><span class="o">(</span><span class="nc">Field</span><span class="o">(</span><span class="s">&#34;id&#34;</span><span class="o">,</span><span class="s">&#34;Long&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;email&#34;</span><span class="o">,</span><span class="s">&#34;String&#34;</span><span class="o">),</span> <span class="nc">Field</span><span class="o">(</span><span class="s">&#34;note&#34;</span><span class="o">,</span><span class="s">&#34;Option[String]&#34;</span><span class="o">,</span><span class="kc">false</span><span class="o">,</span><span class="kc">true</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Now we can ask the compiler &ldquo;what fields does this case class have?&rdquo; at compile time. That&rsquo;s the foundation.</p>
<blockquote>
  <p>If you prefer libraries, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works great for both Scala 2 and 3.</p>

</blockquote><hr>
<h2 id="part-3---policies-exact--backward--forward">Part 3 - Policies: Exact / Backward / Forward</h2>
<p>Here&rsquo;s where we define compatibility rules. Think of these as your migration strategies:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Policy.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Policy</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">Policy</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">Exact</span>     <span class="kt">extends</span> <span class="kt">Policy</span>
</span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Backward</span>  <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// producer can add optional/defaulted fields
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Forward</span>   <span class="k">extends</span> <span class="nc">Policy</span> <span class="c1">// consumer tolerates extra fields
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Exact</span>    <span class="k">extends</span> <span class="nc">Exact</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Backward</span> <span class="k">extends</span> <span class="nc">Backward</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">Forward</span>  <span class="k">extends</span> <span class="nc">Forward</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Why three policies?</p>
<ul>
<li><strong>Exact</strong>: Schemas must match exactly. No surprises. Use this for critical paths.</li>
<li><strong>Backward</strong>: Producer can add optional fields. Useful when rolling out new features without breaking consumers.</li>
<li><strong>Forward</strong>: Consumer ignores extra fields. Useful when the producer might send more than you need.</li>
</ul>
<pre class="mermaid">flowchart LR
    EX[&#34;Policy.Exact&#34;]
    BW[&#34;Policy.Backward&#34;]
    FW[&#34;Policy.Forward&#34;]

    EX --- BW
    EX --- FW

    EX:::tight --&gt;|no diffs allowed| OK1[&#34;✔ match only&#34;]
    BW:::relax --&gt;|allow missing optional in producer\nallow extra fields in producer| OK2[&#34;✔ migration adds&#34;]
    FW:::relax --&gt;|allow extra fields in consumer side| OK3[&#34;✔ tolerant reader&#34;]

    classDef tight fill:#eef,stroke:#446;
    classDef relax fill:#efe,stroke:#484;
</pre><p>Visual (save/share):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/policy-lattice.svg"
            alt="Policy lattice"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    A tiny policy lattice - Exact / Backward / Forward.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<p>In practice: start with Exact, relax to Backward/Forward during migrations, then tighten back to Exact once stable.</p>
<div class="admonition success">
  <div class="admonition-title">Migration strategy</div>
  <div class="admonition-content">
    The policy progression that works in production: Exact for stable pipelines, Backward during producer rollouts (allows new optional fields), Forward during consumer updates (tolerates extras), then back to Exact after migration completes. Document the expected timeline - &ldquo;Backward policy through Q2 migration, then tightening to Exact.&rdquo; Prevents policy drift from becoming permanent.
  </div>
</div>
<hr>
<h2 id="part-4---how-this-fits-with-existing-tools">Part 4 - How this fits with existing tools</h2>
<p>Before we dive deeper, you might be thinking: &ldquo;We already have Avro, Protobuf, schema registries, and Great Expectations. Why add compile-time contracts?&rdquo;</p>
<p>Fair question. Here&rsquo;s how they compare:</p>
<p><strong>Schema Registry (Confluent, AWS Glue)</strong></p>
<ul>
<li><strong>What it does</strong>: Centralized schema storage with version control</li>
<li><strong>When it helps</strong>: Runtime validation across services, schema evolution tracking</li>
<li><strong>The gap</strong>: You find mismatches when the job runs, not when you compile</li>
<li><strong>Use together</strong>: Schema registry for cross-service contracts; compile-time for intra-repo checks</li>
</ul>
<p><strong>Avro / Protobuf</strong></p>
<ul>
<li><strong>What they do</strong>: Binary serialization with embedded schemas</li>
<li><strong>When they help</strong>: Network efficiency, strong contracts between services</li>
<li><strong>The gap</strong>: Schema compatibility is checked at runtime or via external tooling</li>
<li><strong>Use together</strong>: Avro/Protobuf for wire format; compile-time checks to ensure your case classes match those schemas</li>
</ul>
<p><strong>Great Expectations / Deequ</strong></p>
<ul>
<li><strong>What they do</strong>: Runtime data quality validation (nulls, ranges, distributions)</li>
<li><strong>When they help</strong>: Catching bad data values, statistical anomalies</li>
<li><strong>The gap</strong>: They validate data content, not schema structure at compile time</li>
<li><strong>Use together</strong>: Compile-time for structure; Great Expectations for data quality</li>
</ul>
<p><strong>The compile-time advantage</strong>: Catches drift in your controlled codebase before deployment. If your transform expects field <code>X</code> but the source produces field <code>Y</code>, your build fails - not your production job. Think of it as an additional safety layer for systems you build and version together.</p>
<p>Real scenario: You have Avro schemas in a registry, but your Scala pipeline reads them into case classes. Compile-time contracts verify those case classes match the expected contract before you deploy. If upstream changes the Avro schema, your build breaks - you don&rsquo;t wait for the job to crash.</p>
<hr>
<h2 id="part-5---understanding-typerepr-the-compilers-view-of-types">Part 5 - Understanding TypeRepr: The compiler&rsquo;s view of types</h2>
<p>Before we dive into the macro, you need to understand <code>TypeRepr</code> - Scala 3&rsquo;s way of representing types during compilation.</p>
<p>Think of it like this: When you write <code>List[String]</code>, the compiler doesn&rsquo;t just see text. It sees a structured tree:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">AppliedType(List, [String])
</span></span><span class="line"><span class="cl">   │               │
</span></span><span class="line"><span class="cl">   │               └─ TypeRepr for String
</span></span><span class="line"><span class="cl">   └─ Type constructor</span></span></code></pre></td></tr></table>
</div>
</div><pre class="mermaid">flowchart TB
    A[&#34;List[String]&#34;] --&gt; B[&#34;AppliedType(List,[String])&#34;]
    B --&gt; C[&#34;tycon = List&#34;]
    B --&gt; D[&#34;args = [String]&#34;]
</pre><p><strong>Key TypeRepr operations:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span> <span class="o">=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 1. Subtype checking: Is List[String] a subtype of Seq[?]?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>  <span class="c1">// true
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 2. Type equality: Is this exactly String?
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// false
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// 3. Pattern matching: Break it apart
</span></span></span><span class="line"><span class="cl"><span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="n">tycon</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// tycon = List
</span></span></span><span class="line"><span class="cl">    <span class="o">//</span> <span class="n">args</span> <span class="k">=</span> <span class="o">[</span><span class="kt">String</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> To compare schemas, we need to ask questions like:</p>
<ul>
<li>&ldquo;Is this field an Option?&rdquo;</li>
<li>&ldquo;What&rsquo;s inside this List?&rdquo;</li>
<li>&ldquo;Does this case class have a primary constructor?&rdquo;</li>
</ul>
<p><code>TypeRepr</code> lets us answer these at compile time.</p>
<hr>
<h2 id="part-5---the-typeinspector-pattern-one-question-at-a-time">Part 5 - The TypeInspector pattern: One question at a time</h2>
<p>Now let&rsquo;s build the utilities. Each function answers ONE question about a type. Start simple:</p>
<h3 id="inspector-1-is-this-a-case-class">Inspector 1: Is this a case class?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>  <span class="c1">// Get the symbol (the &#34;name&#34; of the type)
</span></span></span><span class="line"><span class="cl">  <span class="n">s</span><span class="o">.</span><span class="n">isClassDef</span> <span class="o">&amp;&amp;</span> <span class="n">s</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">Case</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>typeSymbol</code> - Every type has a symbol (like a unique ID)</li>
<li><code>isClassDef</code> - Is it a class? (not a trait, object, etc.)</li>
<li><code>flags.is(Flags.Case)</code> - Does it have the <code>case</code> modifier?</li>
</ul>
<p><strong>Try it mentally:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>  <span class="c1">// ✅ isClassDef=true, has Case flag
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">UserTrait</span>            <span class="c1">// ❌ isClassDef=false
</span></span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">UserObject</span>          <span class="o">//</span> <span class="o">❌</span> <span class="n">not</span> <span class="n">a</span> <span class="n">class</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-2-what-are-the-type-arguments">Inspector 2: What are the type arguments?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span> <span class="n">t</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="nc">AppliedType</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="n">args</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">args</span>  <span class="c1">// Extract the arguments
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">_</span>                    <span class="k">=&gt;</span> <span class="nc">Nil</span>   <span class="o">//</span> <span class="nc">No</span> <span class="n">arguments</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>AppliedType</code> - Pattern for generic types like <code>List[String]</code> or <code>Map[String, Int]</code></li>
<li><code>args</code> - The type parameters: <code>[String]</code> or <code>[String, Int]</code></li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>        <span class="c1">// AppliedType(List, [String])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>    <span class="c1">// AppliedType(Map, [String, Int])
</span></span></span><span class="line"><span class="cl">                    <span class="c1">// appliedArgs returns [String, Int]
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>              <span class="c1">// Not an AppliedType
</span></span></span><span class="line"><span class="cl">                    <span class="o">//</span> <span class="n">appliedArgs</span> <span class="n">returns</span> <span class="nc">Nil</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-3-is-this-an-option">Inspector 3: Is this an Option?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>  <span class="c1">// Is it a subtype of Option[Something]?
</span></span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span>           <span class="c1">// Yes! Extract the Something
</span></span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span>                             <span class="o">//</span> <span class="nc">Nope</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li><code>&lt;:&lt;</code> - Subtype check: &ldquo;Is this type compatible with Option[?]?&rdquo;</li>
<li><code>Option[?]</code> - The <code>?</code> means &ldquo;Option of anything&rdquo;</li>
<li><code>appliedArgs(t).headOption</code> - Get the first (and only) type argument</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>  <span class="c1">// ✅ Is subtype of Option[?]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// appliedArgs returns [String]
</span></span></span><span class="line"><span class="cl">                <span class="c1">// headOption returns Some(String)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">Some</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>       <span class="c1">// ✅ Some is subtype of Option
</span></span></span><span class="line"><span class="cl">                <span class="c1">// Returns Some(Int)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">String</span>          <span class="c1">// ❌ Not an Option
</span></span></span><span class="line"><span class="cl">                <span class="o">//</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why we need this:</strong> Field-level optionality. When we see <code>age: Option[Int]</code>, we need to know:</p>
<ol>
<li>The field itself is optional</li>
<li>The underlying type is <code>Int</code></li>
</ol>
<h3 id="inspector-4-is-this-a-sequence">Inspector 4: Is this a sequence?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">TypeRepr</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">isSeqLike</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Vector</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">isSeqLike</span> <span class="n">then</span> <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">headOption</span> <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s ANY kind of sequence type</li>
<li>If yes, extract what&rsquo;s inside (the element type)</li>
</ul>
<p><strong>Why check multiple types?</strong> Because Scala has many collection types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>    <span class="c1">// ✅ Is List[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Seq</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>        <span class="c1">// ✅ Is Seq[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Vector</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span>    <span class="c1">// ✅ Is Vector[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span>     <span class="c1">// ✅ Is Array[?]
</span></span></span><span class="line"><span class="cl"><span class="nc">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>     <span class="o">//</span> <span class="o">✅</span> <span class="nc">Is</span> <span class="nc">Set</span><span class="o">[</span><span class="kt">?</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p>All these should become <code>SequenceShape(element)</code>.</p>
<h3 id="inspector-5-is-this-a-map">Inspector 5: Is this a Map?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[(</span><span class="kt">TypeRepr</span>, <span class="kt">TypeRepr</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="n">t</span> <span class="o">&lt;:&lt;</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Map</span><span class="o">[</span><span class="kt">?</span>, <span class="kt">?</span><span class="o">]]</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">    <span class="n">appliedArgs</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="n">k</span> <span class="k">:</span><span class="kt">:</span> <span class="kt">v</span> <span class="kt">::</span> <span class="kt">Nil</span> <span class="o">=&gt;</span> <span class="nc">Some</span><span class="o">((</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">))</span>  <span class="c1">// Got key and value
</span></span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Map requires two type args: </span><span class="si">${</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">else</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening:</strong></p>
<ul>
<li>Check if it&rsquo;s a <code>Map[Something, Something]</code></li>
<li>Extract BOTH type arguments: key and value</li>
<li>If Map has wrong number of args, abort (compiler error)</li>
</ul>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">Int</span><span class="o">]</span>  <span class="c1">// ✅ Returns Some((String, Int))
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Long</span>, <span class="kt">User</span><span class="o">]</span>   <span class="c1">// ✅ Returns Some((Long, User))
</span></span></span><span class="line"><span class="cl"><span class="nc">String</span>            <span class="o">//</span> <span class="o">❌</span> <span class="nc">Returns</span> <span class="nc">None</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="inspector-6-can-this-be-a-map-key">Inspector 6: Can this be a Map key?</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">isAtomicKey</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Long</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Short</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">  <span class="n">t</span> <span class="o">=:=</span> <span class="nc">TypeRepr</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">Boolean</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Why this matters:</strong> Maps in Spark StructType only support primitive keys. You can&rsquo;t have <code>Map[User, String]</code> - User isn&rsquo;t a primitive.</p>
<p><strong>Examples:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">User</span><span class="o">]</span>  <span class="c1">// ✅ String is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Data</span><span class="o">]</span>     <span class="c1">// ✅ Int is atomic
</span></span></span><span class="line"><span class="cl"><span class="nc">Map</span><span class="o">[</span><span class="kt">User</span>, <span class="kt">String</span><span class="o">]</span>  <span class="o">//</span> <span class="o">❌</span> <span class="nc">User</span> <span class="n">is</span> <span class="nc">NOT</span> <span class="n">atomic</span> <span class="o">-</span> <span class="n">compile</span> <span class="n">error</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="the-pattern-recap">The pattern recap:</h3>
<p>Each inspector has ONE job:</p>
<ol>
<li><strong>isCaseClass</strong> - &ldquo;Is this a case class?&rdquo;</li>
<li><strong>appliedArgs</strong> - &ldquo;What are the generic type arguments?&rdquo;</li>
<li><strong>optionArg</strong> - &ldquo;Is this an Option, and what&rsquo;s inside?&rdquo;</li>
<li><strong>seqArg</strong> - &ldquo;Is this a sequence, and what&rsquo;s the element type?&rdquo;</li>
<li><strong>mapArgs</strong> - &ldquo;Is this a Map, and what are key/value types?&rdquo;</li>
<li><strong>isAtomicKey</strong> - &ldquo;Can this be a Map key?&rdquo;</li>
</ol>
<p>Compose these to answer complex questions. That&rsquo;s the TypeInspector pattern.</p>
<hr>
<h2 id="part-6---building-shapes-recursively-the-interpreter-pattern">Part 6 - Building shapes recursively: The interpreter pattern</h2>
<p>Now we use those inspectors to build a <code>TypeShape</code> - our internal representation of a type&rsquo;s structure.</p>
<p>This is recursive. For <code>List[Option[User]]</code>, we build:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">SequenceShape(
</span></span><span class="line"><span class="cl">  OptionalShape(
</span></span><span class="line"><span class="cl">    StructShape([field1, field2, ...])
</span></span><span class="line"><span class="cl">  )
</span></span><span class="line"><span class="cl">)</span></span></code></pre></td></tr></table>
</div>
</div><p>Let me show you the algorithm step by step.</p>
<h3 id="the-shape-building-algorithm">The shape building algorithm:</h3>
<pre class="mermaid">flowchart TD
    T[&#34;typeShapeOf(t)&#34;] --&gt; O{&#34;Option?&#34;}
    O -- &#34;Yes&#34; --&gt; OI[&#34;inner = optionArg(t)&#34;] --&gt; R1[&#34;typeShapeOf(inner)&#34;]
    O -- &#34;No&#34; --&gt; S{&#34;Sequence?&#34;}
    S -- &#34;Yes&#34; --&gt; SE[&#34;elem = seqArg(t)&#34;] --&gt; R2[&#34;SequenceShape(typeShapeOf(elem))&#34;]
    S -- &#34;No&#34; --&gt; M{&#34;Map?&#34;}
    M -- &#34;Yes&#34; --&gt; MKV[&#34;(k,v) = mapArgs(t)&#34;]
    MKV --&gt; K{&#34;isAtomicKey(k)?&#34;}
    K -- &#34;No&#34; --&gt; ERR[&#34;abort: unsupported key&#34;]
    K -- &#34;Yes&#34; --&gt; MV[&#34;MapShape(PrimitiveShape(k), typeShapeOf(v))&#34;]
    M -- &#34;No&#34; --&gt; CC{&#34;Case class?&#34;}
    CC -- &#34;Yes&#34; --&gt; FS[&#34;StructShape(collect FieldShape...)&#34;]
    CC -- &#34;No&#34; --&gt; P[&#34;PrimitiveShape(t.show)&#34;]
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">def</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">t</span><span class="k">:</span> <span class="kt">TypeRepr</span><span class="o">)</span><span class="k">:</span> <span class="kt">TypeShape</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Step 1: Is it an Option?
</span></span></span><span class="line"><span class="cl">  <span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Step 2: Is it a sequence?
</span></span></span><span class="line"><span class="cl">    <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">      <span class="c1">// Step 3: Is it a Map?
</span></span></span><span class="line"><span class="cl">      <span class="n">mapArgs</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="o">!</span><span class="n">isAtomicKey</span><span class="o">(</span><span class="n">k</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Unsupported Map key: </span><span class="si">${</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="nc">MapShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">k</span><span class="o">.</span><span class="n">show</span><span class="o">),</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">v</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="c1">// Step 4: Is it a case class?
</span></span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// ... handle case class
</span></span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="c1">// Step 5: Fallback - primitive
</span></span></span><span class="line"><span class="cl">          <span class="nc">PrimitiveShape</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">show</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>The order matters!</strong> We check Option first, then sequences, then maps, then case classes, finally primitives.</p>
<h3 id="walk-through-an-example-listoptionstring">Walk through an example: <code>List[Option[String]]</code></h3>
<pre class="mermaid">flowchart TD
    A[&#34;List[Option[String]]&#34;] --&gt;|typeShapeOf| B{&#34;Option?&#34;}
    B -- &#34;No&#34; --&gt; C{&#34;Sequence?&#34;}
    C -- &#34;Yes&#34; --&gt; D[&#34;elem = Option[String]&#34;]
    D --&gt; E[&#34;typeShapeOf(Option[String])&#34;]
    E --&gt; F{&#34;Option?&#34;}
    F -- &#34;Yes&#34; --&gt; G[&#34;inner = String&#34;]
    G --&gt; H[&#34;typeShapeOf(String)&#34;]
    H --&gt; I{&#34;Case class?&#34;}
    I -- &#34;No&#34; --&gt; J[&#34;PrimitiveShape(String)&#34;]
    J --&gt; K[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><ul>
<li>
<p><strong>Step 1:</strong> Is <code>List[Option[String]]</code> an Option?</p>
<ul>
<li>No. <code>optionArg</code> returns <code>None</code>.</li>
<li>Continue to Step 2.</li>
</ul>
</li>
<li>
<p><strong>Step 2:</strong> Is it a sequence?</p>
<ul>
<li>Yes! <code>seqArg</code> returns <code>Some(Option[String])</code>.</li>
<li>Build <code>SequenceShape(typeShapeOf(Option[String]))</code>.</li>
<li><strong>Recurse</strong> on <code>Option[String]</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 1:</strong> Is <code>Option[String]</code> an Option?</p>
<ul>
<li>Yes! <code>optionArg</code> returns <code>Some(String)</code>.</li>
<li><strong>Recurse</strong> on <code>String</code>.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 2:</strong> Is <code>String</code> an Option?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 3:</strong> Is <code>String</code> a sequence?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 4:</strong> Is <code>String</code> a Map?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 5:</strong> Is <code>String</code> a case class?</p>
<ul>
<li>No.</li>
</ul>
</li>
<li>
<p><strong>Recursion Step 6:</strong> Fallback to primitive.</p>
<ul>
<li>Return <code>PrimitiveShape(&quot;String&quot;)</code>.</li>
</ul>
</li>
</ul>
<p><strong>Unwind the recursion:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="o">(</span><span class="n">from</span> <span class="nc">Option</span> <span class="n">recursion</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">→</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="nc">PrimitiveShape</span><span class="o">(</span><span class="s">&#34;String&#34;</span><span class="o">))</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, where did the <code>Option</code> go? It got consumed! The <code>typeShapeOf</code> for <code>Option</code> just recurses on the inner type. This is the edge case I mentioned - element-level optionality gets lost.</p>
<h3 id="handling-case-classes">Handling case classes:</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">if</span> <span class="n">isCaseClass</span><span class="o">(</span><span class="n">t</span><span class="o">)</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">sym</span>    <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">typeSymbol</span>                      <span class="c1">// Get the class symbol
</span></span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">params</span> <span class="k">=</span> <span class="n">sym</span><span class="o">.</span><span class="n">primaryConstructor</span><span class="o">.</span><span class="n">paramSymss</span><span class="o">.</span><span class="n">flatten</span>  <span class="c1">// Get constructor params
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">val</span> <span class="n">fields</span> <span class="k">=</span> <span class="n">params</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">p</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">name</span>       <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span>                      <span class="c1">// Field name
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">ptpe</span>       <span class="k">=</span> <span class="n">t</span><span class="o">.</span><span class="n">memberType</span><span class="o">(</span><span class="n">p</span><span class="o">)</span>             <span class="c1">// Field type
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">hasDefault</span> <span class="k">=</span> <span class="n">p</span><span class="o">.</span><span class="n">flags</span><span class="o">.</span><span class="n">is</span><span class="o">(</span><span class="nc">Flags</span><span class="o">.</span><span class="nc">HasDefault</span><span class="o">)</span> <span class="c1">// Has default value?
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nc">StructShape</span><span class="o">(</span><span class="n">fields</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Let&rsquo;s break this down line by line:</strong></p>
<ol>
<li><strong><code>t.typeSymbol</code></strong> - Get the symbol for the case class</li>
<li><strong><code>sym.primaryConstructor</code></strong> - Case classes have a primary constructor</li>
<li><strong><code>.paramSymss.flatten</code></strong> - Get all parameters (flatten handles multiple param lists)</li>
<li><strong>For each parameter <code>p</code>:</strong>
<ul>
<li><strong><code>p.name</code></strong> - The field name: <code>&quot;id&quot;</code>, <code>&quot;email&quot;</code>, etc.</li>
<li><strong><code>t.memberType(p)</code></strong> - The field&rsquo;s type as TypeRepr</li>
<li><strong><code>p.flags.is(Flags.HasDefault)</code></strong> - Check if it has a default value like <code>age: Int = 0</code></li>
<li><strong><code>optionArg(ptpe).fold(...)</code></strong> - Check if the field type is Option[T]:
<ul>
<li>If yes: extract T, set <code>isOpt = true</code></li>
<li>If no: use the type as-is, set <code>isOpt = false</code></li>
</ul>
</li>
<li><strong><code>typeShapeOf(uT)</code></strong> - <strong>Recurse!</strong> Build the shape for the field&rsquo;s underlying type</li>
</ul>
</li>
</ol>
<h3 id="example-case-class-userid-long-email-string-age-optionint--none">Example: <code>case class User(id: Long, email: String, age: Option[Int] = None)</code></h3>
<pre class="mermaid">flowchart TD
    U[&#34;User(id: Long, email: String, age: Option[Int] = None)&#34;]
    U --&gt; F1[&#34;Field &#39;id&#39; : Long&#34;]
    F1 --&gt; S1[&#34;PrimitiveShape(Long)&#34;]
    U --&gt; F2[&#34;Field &#39;email&#39; : String&#34;]
    F2 --&gt; S2[&#34;PrimitiveShape(String)&#34;]
    U --&gt; F3[&#34;Field &#39;age&#39; : Option[Int] (default None)&#34;]
    F3 --&gt; O1{&#34;Option?&#34;}
    O1 -- &#34;Yes&#34; --&gt; A1[&#34;inner = Int&#34;]
    A1 --&gt; S3[&#34;PrimitiveShape(Int)&#34;]
    S1 --&gt; COLLECT[&#34;Collect fields&#34;]
    S2 --&gt; COLLECT
    S3 --&gt; COLLECT
    COLLECT --&gt; FINAL[&#34;StructShape([ id:PrimitiveShape(Long), email:PrimitiveShape(String), age:PrimitiveShape(Int) (optional, default) ])&#34;]
</pre><p><strong>Processing:</strong></p>
<ul>
<li>
<p><strong>Field 1: id</strong></p>
<ul>
<li><code>name</code> = &ldquo;id&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Long</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(Long)</code> = None → <code>uT = Long</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(Long)</code> = <code>PrimitiveShape(&quot;Long&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;id&quot;, PrimitiveShape(&quot;Long&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 2: email</strong></p>
<ul>
<li><code>name</code> = &ldquo;email&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>String</code></li>
<li><code>hasDefault</code> = false</li>
<li><code>optionArg(String)</code> = None → <code>uT = String</code>, <code>isOpt = false</code></li>
<li><code>typeShapeOf(String)</code> = <code>PrimitiveShape(&quot;String&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;email&quot;, PrimitiveShape(&quot;String&quot;), hasDefault=false, isOptional=false)</code></li>
</ul>
</li>
<li>
<p><strong>Field 3: age</strong></p>
<ul>
<li><code>name</code> = &ldquo;age&rdquo;</li>
<li><code>ptpe</code> = TypeRepr for <code>Option[Int]</code></li>
<li><code>hasDefault</code> = <strong>true</strong> (has <code>= None</code>)</li>
<li><code>optionArg(Option[Int])</code> = Some(<code>Int</code>) → <code>uT = Int</code>, <code>isOpt = true</code></li>
<li><code>typeShapeOf(Int)</code> = <code>PrimitiveShape(&quot;Int&quot;)</code></li>
<li>Result: <code>FieldShape(&quot;age&quot;, PrimitiveShape(&quot;Int&quot;), hasDefault=true, isOptional=true)</code></li>
</ul>
</li>
<li>
<p><strong>Final shape:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">StructShape</span><span class="o">([</span>
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">id</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Long</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">email</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">String</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">false</span>, <span class="kt">false</span><span class="o">)</span>,
</span></span><span class="line"><span class="cl">  <span class="kt">FieldShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">age</span><span class="err">&#34;</span>, <span class="kt">PrimitiveShape</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Int</span><span class="err">&#34;</span><span class="o">)</span>, <span class="kt">true</span>, <span class="kt">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">])</span></span></span></code></pre></td></tr></table>
</div>
</div></li>
</ul>
<h3 id="why-this-matters-1">Why this matters:</h3>
<p>Now we have a normalized representation of the case class. We can compare two <code>StructShape</code>s to see if they match, regardless of the original Scala syntax.</p>
<p>The recursive structure handles arbitrarily nested types:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Order</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>           <span class="c1">// Recurses into List, then into LineItem
</span></span></span><span class="line"><span class="cl">  <span class="n">metadata</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">],</span>   <span class="c1">// Recurses into Map, then String (twice)
</span></span></span><span class="line"><span class="cl">  <span class="n">address</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">]</span>         <span class="c1">// Recurses into Option, then Address
</span></span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Each recursion builds a piece of the tree. The tree is the schema.</p>
<hr>
<h2 id="part-7---field-vs-element-optionality-a-critical-distinction">Part 7 - Field vs Element optionality: A critical distinction</h2>
<p>Here&rsquo;s something subtle that most guides miss. The code in github handles two kinds of optionality:</p>
<h3 id="field-level-optionality-option-on-the-field-itself">Field-level optionality (Option on the field itself)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                                   <span class="o">^^^^^^^^^^^</span> <span class="n">field</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The macro detects this and sets <code>isOptional = true</code> on the FieldShape:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="o">(</span><span class="n">uT</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span> <span class="k">=</span> <span class="n">optionArg</span><span class="o">(</span><span class="n">ptpe</span><span class="o">).</span><span class="n">fold</span><span class="o">(</span><span class="n">ptpe</span> <span class="o">-&gt;</span> <span class="kc">false</span><span class="o">)(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">FieldShape</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">typeShapeOf</span><span class="o">(</span><span class="n">uT</span><span class="o">),</span> <span class="n">hasDefault</span><span class="o">,</span> <span class="n">isOpt</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="element-level-optionality-option-inside-a-collection">Element-level optionality (Option inside a collection)</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Data</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span>                          <span class="o">^^^^^^^^^^^^^^</span> <span class="n">element</span><span class="o">-</span><span class="n">level</span> <span class="nc">Option</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This creates a nested shape: <code>SequenceShape(OptionShape(PrimitiveShape(&quot;String&quot;)))</code></p>
<p>Wait, that&rsquo;s not in the code in github! Let me check&hellip; Actually, the code in github&rsquo;s <code>typeShapeOf</code> handles this by recursing:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">optionArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">).</span><span class="n">getOrElse</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">seqArg</span><span class="o">(</span><span class="n">t</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="n">a</span> <span class="k">=&gt;</span> <span class="nc">SequenceShape</span><span class="o">(</span><span class="n">typeShapeOf</span><span class="o">(</span><span class="n">a</span><span class="o">))).</span><span class="n">getOrElse</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>So <code>List[Option[String]]</code> becomes:</p>
<ol>
<li><code>seqArg</code> detects List → extract <code>Option[String]</code></li>
<li>Recurse on <code>Option[String]</code> → <code>typeShapeOf(Option[String])</code></li>
<li><code>optionArg</code> detects Option → extract <code>String</code></li>
<li>Base case: <code>PrimitiveShape(&quot;String&quot;)</code></li>
</ol>
<p>But the final shape is <code>SequenceShape(PrimitiveShape(&quot;String&quot;))</code> - the intermediate Option is consumed!</p>
<p><strong>Why this matters for contracts</strong>: When you write:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">Producer</span><span class="o">(</span><span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]])</span></span></span></code></pre></td></tr></table>
</div>
</div><p>These should probably NOT conform under Exact. The producer allows null elements, the contract doesn&rsquo;t. But the current code in github might allow it because the Option gets unwrapped.</p>
<p>This is the kind of edge case you discover when shipping to production. The fix would be to add an <code>OptionalShape</code> wrapper in the model.</p>
<div class="admonition caution">
  <div class="admonition-title">Optionality edge case</div>
  <div class="admonition-content">
    The element-level optionality issue (List[Option[String]] getting flattened) is real and subtle. If your schemas use nullable collections, test that the compile-time check actually catches the difference between List[String] and List[Option[String]]. The current implementation may allow mismatches here. Add explicit test cases for your nullable collection patterns before relying on them in production.
  </div>
</div>
<p>Visual (field vs element optionality):

<figure >
    
        <img src="/images/posts/compile-time-data-contracts/optionality.svg"
            alt="Field vs Element Optionality"
            
             />
    

    
        <figcaption>
            

            
                <p style="margin: -0.5rem 0 0 0;">
                    Field optionality vs element optionality - they&#39;re not the same.


                
                
                

            </p>
            
        </figcaption>
    
</figure>

</p>
<pre class="mermaid">flowchart LR
    A[&#34;List[Option[String]]&#34;] --&gt; B{&#34;Element optionality?&#34;}
    B -- &#34;intended shape&#34; --&gt; C[&#34;SequenceShape(OptionalShape(PrimitiveShape(String)))&#34;]
    B -- &#34;current code in github\n(consumes Option)&#34; --&gt; D[&#34;SequenceShape(PrimitiveShape(String))&#34;]
</pre><hr>
<h2 id="part-8---compiletime--evidence-the-full-picture">Part 8 - Compile‑time <strong>&ldquo;conforms&rdquo;</strong> evidence (the full picture)</h2>
<p>Now let&rsquo;s put it all together. Here&rsquo;s the complete macro from the code in github:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/Conforms.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.quoted.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">class</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">object</span> <span class="nc">Conforms</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">materialize</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">$</span><span class="o">{</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">P:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>, <span class="kt">String</span>, <span class="kt">Boolean</span>, <span class="kt">Boolean</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">fs</span> <span class="k">=</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span>
</span></span><span class="line"><span class="cl">        <span class="n">fs</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">hasDefault</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">isOptional</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">shapeOf</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1">// Compare fields by name (unordered, case-sensitive for now)
</span></span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">outMap</span>      <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contractMap</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">missing</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">outMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s">:</span><span class="si">$t$</span><span class="s">{if opt then &#34;</span> <span class="o">(</span><span class="n">optional</span><span class="o">)</span><span class="s">&#34; else &#34;&#34;}&#34;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">extra</span>   <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span>      <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">contractMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mism</span>    <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span> <span class="o">!=</span> <span class="n">t</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> expected </span><span class="si">$t</span><span class="s">, found </span><span class="si">${</span><span class="n">outMap</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">_1</span><span class="si">}</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">!=</span> <span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> optionality mismatch&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="o">(</span><span class="n">okMissing</span><span class="o">,</span> <span class="n">okExtra</span><span class="o">,</span> <span class="n">okMism</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Exact</span><span class="o">]</span>    <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Backward</span><span class="o">]</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">.</span><span class="n">filterNot</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="s">&#34;(optional)&#34;</span><span class="o">)),</span> <span class="nc">Nil</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">Policy.Forward</span><span class="o">]</span>  <span class="k">=&gt;</span> <span class="o">(</span><span class="nc">Nil</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span>                  <span class="k">=&gt;</span> <span class="o">(</span><span class="n">missing</span><span class="o">,</span> <span class="n">extra</span><span class="o">,</span> <span class="n">mism</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">okMissing</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okExtra</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">okMism</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="s">s&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">        |Compile‑time contract drift (policy: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">P</span><span class="o">]</span><span class="si">}</span><span class="s">).
</span></span></span><span class="line"><span class="cl"><span class="s">        |Out: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span><span class="si">}</span><span class="s"> vs Contract: </span><span class="si">${</span><span class="nc">Type</span><span class="o">.</span><span class="n">show</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Missing: </span><span class="si">${</span><span class="n">okMissing</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Extra: </span><span class="si">${</span><span class="n">okExtra</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;, &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |Mismatched: </span><span class="si">${</span><span class="n">okMism</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">        |&#34;&#34;&#34;</span><span class="o">.</span><span class="n">stripMargin</span>
</span></span><span class="line"><span class="cl">      <span class="n">quotes</span><span class="o">.</span><span class="n">reflect</span><span class="o">.</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="n">msg</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="k">new</span> <span class="nc">Conforms</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Let me break down what&rsquo;s happening:</p>
<ol>
<li>
<p><strong>Extract shapes</strong>: We summon <code>Shape[Out]</code> and <code>Shape[Contract]</code> and get their field lists. Note the <code>.valueOrAbort</code> - this forces compile-time evaluation.</p>
</li>
<li>
<p><strong>Compare</strong>: We build maps of field names → (type, optionality) and find differences.</p>
</li>
<li>
<p><strong>Apply policy</strong>:</p>
<ul>
<li>Exact: all differences are errors</li>
<li>Backward: missing optional fields are OK, extras are OK</li>
<li>Forward: extras are OK (but missing required fields are not)</li>
</ul>
</li>
<li>
<p><strong>Abort or succeed</strong>: If there are violations, we call <code>report.errorAndAbort</code> with a detailed message showing exactly what&rsquo;s wrong. Otherwise, we return the evidence.</p>
</li>
</ol>
<p>The beauty here: <strong>this runs during compilation</strong>. If schemas don&rsquo;t match, your code won&rsquo;t compile. Period.</p>
<div class="admonition danger">
  <div class="admonition-title">Build breakage alert</div>
  <div class="admonition-content">
    Compile-time contract checks will break your build when schemas drift. That&rsquo;s the point, but coordinate with your team. When upstream adds a field, every downstream pipeline build fails until they update contracts or transforms. This forces alignment, but can block deployments if not managed. Use CI branch builds to preview impact before merging schema changes.
  </div>
</div>
<p>Use it like this:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/ConformsSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Contract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutOk</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OutMissing</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@main</span> <span class="k">def</span> <span class="n">checkConformance</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutOk</span>,      <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>   <span class="c1">// ✅ compiles
</span></span></span><span class="line"><span class="cl">  <span class="c1">// summon[Conforms[OutMissing, Contract, Policy.Exact]] // ❌ compile‑time error (Missing: email:String)
</span></span></span><span class="line"><span class="cl">  <span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span> <span class="o">//</span> <span class="o">✅</span> <span class="n">compiles</span> <span class="o">(</span><span class="n">migration</span> <span class="n">mode</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>That&rsquo;s it. The entire core. In production, you&rsquo;d want to handle the element optionality issue I mentioned, but the structure stays the same.</p>
<hr>
<h2 id="part-9---developer-ergonomics-what-using-this-actually-feels-like">Part 9 - Developer ergonomics: What using this actually feels like</h2>
<p>Let&rsquo;s talk about the practical side - what happens when you actually use this in your daily work?</p>
<h3 id="compile-times">Compile times</h3>
<p><strong>Question: Does this slow down compilation?</strong></p>
<p>In practice, not noticeably. The macro runs once per <code>summon[Conforms[...]]</code> call during compilation. For a typical pipeline with 5-10 contract checks, you&rsquo;re adding maybe 100-200ms to your build. Compare that to the hours you&rsquo;d spend debugging a production schema mismatch.</p>
<h3 id="error-messages">Error messages</h3>
<p>When schemas drift, you get this:</p>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[error] Compile-time contract drift (policy: Policy.Exact).
</span></span><span class="line"><span class="cl">[error] Out: CustomerProducer vs Contract: CustomerContract
</span></span><span class="line"><span class="cl">[error] Extra: segment
</span></span><span class="line"><span class="cl">[error] Missing:
</span></span><span class="line"><span class="cl">[error] Mismatched:</span></span></code></pre></td></tr></table>
</div>
</div><p>Clear, actionable, and it stops your build. No cryptic macro errors, no stack traces. Just: &ldquo;Hey, you have an extra field called <code>segment</code>.&rdquo;</p>
<h3 id="integration-patterns">Integration patterns</h3>
<p><strong>With Spark:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Your Spark job
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">df</span> <span class="k">=</span> <span class="n">spark</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">parquet</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">typed</span> <span class="k">=</span> <span class="n">df</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">]</span>  <span class="c1">// Spark encoder
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Compile-time check before transformation
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerProducer</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">output</span> <span class="k">=</span> <span class="n">typed</span><span class="o">.</span><span class="n">transform</span><span class="o">(</span><span class="n">dropSegment</span><span class="o">).</span><span class="n">as</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">]</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>With schema registries:</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Avro schema in registry → case class → contract check
</span></span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerAvro</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">amt</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">CustomerAvro</span>, <span class="kt">CustomerContract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>
</span></span><span class="line"><span class="cl"><span class="o">//</span> <span class="nc">Build</span> <span class="n">fails</span> <span class="k">if</span> <span class="nc">Avro</span> <span class="n">schema</span> <span class="n">drift</span><span class="o">!</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>CI/CD integration:</strong>
Just run <code>sbt compile</code>. If contracts drift, the build fails. No special tooling needed. Works with Jenkins, GitHub Actions, GitLab CI - anything that runs sbt.</p>
<h3 id="onboarding">Onboarding</h3>
<p><strong>New team members ask</strong>: &ldquo;Why won&rsquo;t this compile?&rdquo;</p>
<p><strong>Answer</strong>: &ldquo;Check the error. You&rsquo;re trying to use <code>CustomerProducer</code> but the contract expects <code>CustomerContract</code>. Either adjust your transform or update the contract.&rdquo;</p>
<p>That&rsquo;s it. The compiler guides them. After one or two cases, they get it.</p>
<h3 id="when-to-use-vs-skip">When to use vs skip</h3>
<p><strong>Use compile-time contracts when</strong>:</p>
<ul>
<li>Multiple teams work on the same pipeline codebase</li>
<li>Schema changes break production regularly</li>
<li>You want fast feedback (compile vs deploy-and-test)</li>
</ul>
<p><strong>Skip when</strong>:</p>
<ul>
<li>One-off scripts or exploratory work</li>
<li>External APIs you can&rsquo;t control (use runtime validation)</li>
<li>Schema is genuinely dynamic (JSON with unknown keys)</li>
</ul>
<hr>
<h2 id="part-10---nested-types-maps-lists-and-deep-structures">Part 10 - Nested types: Maps, Lists, and deep structures</h2>
<p>The code in github handles nested types beautifully. Look at this example from <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Deep / Nested structures
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">LineItem</span><span class="o">(</span><span class="n">sku</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">qty</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">attrs</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">String</span>, <span class="kt">String</span><span class="o">])</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Address</span><span class="o">(</span><span class="n">street</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">zip</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderOut</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>
</span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">OrderContract</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">items</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">LineItem</span><span class="o">],</span>        <span class="c1">// List vs Seq - both are sequences
</span></span></span><span class="line"><span class="cl">  <span class="n">shipTo</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Address</span><span class="o">],</span>      <span class="c1">// Nested case class
</span></span></span><span class="line"><span class="cl">  <span class="n">tags</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Nil</span>       <span class="c1">// Set vs Seq, has default
</span></span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">evDeepOk</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">OrderOut</span>, <span class="kt">OrderContract</span>, <span class="kt">SchemaPolicy.Backward.</span><span class="k">type</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>What&rsquo;s happening here:</strong></p>
<ol>
<li>
<p><strong>Nested case classes</strong> - <code>Address</code> inside <code>OrderOut</code>. The macro recurses: when it sees <code>Option[Address]</code>, it unwraps to <code>Address</code>, then checks if <code>Address</code> is a case class, and recurses again to get its fields.</p>
</li>
<li>
<p><strong>Collections</strong> - <code>List[LineItem]</code> vs <code>Seq[LineItem]</code>. Both match because <code>seqArg</code> treats them equivalently. The macro sees &ldquo;sequence of LineItem&rdquo; and recurses on <code>LineItem</code>.</p>
</li>
<li>
<p><strong>Maps</strong> - <code>Map[String, String]</code> in <code>attrs</code>. The macro checks key type (must be atomic), then recurses on value type.</p>
</li>
<li>
<p><strong>Default values</strong> - <code>tags: Seq[String] = Nil</code> has a default. Under Backward policy, if <code>OrderOut</code> is missing <code>tags</code>, it&rsquo;s OK because the contract has a default.</p>
</li>
</ol>
<p>This is why the TypeInspector pattern matters - it handles arbitrary nesting without special cases.</p>
<hr>
<h2 id="part-10---policy-modifiers-in-two-minutes-ordered-ci-byposition">Part 10 - Policy modifiers in two minutes (Ordered, CI, By‑Position)</h2>
<p>Sometimes you need stricter comparison. Or different rules. Here are three quick policy extensions:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/main/scala/com/example/PolicyMods.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">PolicyMod</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">sealed</span> <span class="kt">trait</span> <span class="kt">ExactOrdered</span>   <span class="kt">extends</span> <span class="kt">PolicyMod</span> <span class="c1">// names + order + types must match
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactCI</span>        <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// case‑insensitive field names
</span></span></span><span class="line"><span class="cl">  <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">PolicyMod</span> <span class="c1">// ignore names; types by index
</span></span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactOrdered</span>    <span class="k">extends</span> <span class="nc">ExactOrdered</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactCI</span>         <span class="k">extends</span> <span class="nc">ExactCI</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="k">object</span> <span class="nc">ExactByPosition</span> <span class="k">extends</span> <span class="nc">ExactByPosition</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">ConformsMod</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">import</span> <span class="kt">scala.quoted.*</span>
</span></span><span class="line"><span class="cl">  <span class="n">inline</span> <span class="k">def</span> <span class="n">apply</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span> <span class="k">&lt;:</span> <span class="kt">PolicyMod</span><span class="o">](</span><span class="n">using</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="nc">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">M</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">impl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">Type</span>, <span class="kt">Contract:</span> <span class="kt">Type</span>, <span class="kt">M:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">quotes.reflect.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">fields</span><span class="o">[</span><span class="kt">A:</span> <span class="kt">Type</span><span class="o">]</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span><span class="o">)]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="k">val</span> <span class="n">sh</span> <span class="k">=</span> <span class="nc">Expr</span><span class="o">.</span><span class="n">summon</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">A</span><span class="o">]].</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;No Shape available&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">      <span class="err">&#39;</span><span class="o">{</span> <span class="nc">$sh</span><span class="o">.</span><span class="n">fields</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=&gt;</span> <span class="o">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">f</span><span class="o">.</span><span class="n">tpe</span><span class="o">))</span> <span class="o">}.</span><span class="n">valueOrAbort</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Out</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">fields</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mismatches</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Type</span><span class="o">.</span><span class="n">of</span><span class="o">[</span><span class="kt">M</span><span class="o">]</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">n</span> <span class="k">=</span> <span class="o">(</span><span class="n">s</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">toLowerCase</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">om</span> <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="k">val</span> <span class="n">cm</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n0</span><span class="o">,</span><span class="n">t</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span><span class="o">(</span><span class="n">n0</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">t</span> <span class="o">}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">        <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">keySet</span> <span class="n">union</span> <span class="n">om</span><span class="o">.</span><span class="n">keySet</span><span class="o">).</span><span class="n">toList</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span> <span class="n">k</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">          <span class="o">(</span><span class="n">cm</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">),</span> <span class="n">om</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">k</span><span class="o">))</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="n">t1</span><span class="o">),</span> <span class="nc">Some</span><span class="o">(</span><span class="n">t2</span><span class="o">))</span> <span class="k">if</span> <span class="n">t1</span> <span class="o">!=</span> <span class="n">t2</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;</span><span class="si">$k</span><span class="s"> expected </span><span class="si">$t1</span><span class="s">, found </span><span class="si">$t2</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">),</span> <span class="nc">None</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;missing </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(</span><span class="nc">None</span><span class="o">,</span> <span class="nc">Some</span><span class="o">(</span><span class="k">_</span><span class="o">))</span> <span class="k">=&gt;</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;extra </span><span class="si">${</span><span class="n">k</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="nc">Nil</span>
</span></span><span class="line"><span class="cl">        <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="k">_</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="k">_</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">!=</span> <span class="n">tC</span> <span class="k">=&gt;</span> <span class="s">s&#34;@</span><span class="si">$i</span><span class="s"> expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="err">&#39;</span><span class="o">[</span><span class="kt">PolicyMod.ExactOrdered</span><span class="o">]</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">out</span><span class="o">.</span><span class="n">length</span> <span class="o">!=</span> <span class="n">contract</span><span class="o">.</span><span class="n">length</span> <span class="n">then</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;length mismatch: </span><span class="si">${</span><span class="n">out</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s"> vs </span><span class="si">${</span><span class="n">contract</span><span class="o">.</span><span class="n">length</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span>
</span></span><span class="line"><span class="cl">          <span class="n">out</span><span class="o">.</span><span class="n">zip</span><span class="o">(</span><span class="n">contract</span><span class="o">).</span><span class="n">zipWithIndex</span><span class="o">.</span><span class="n">flatMap</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">            <span class="k">case</span> <span class="o">(((</span><span class="n">nO</span><span class="o">,</span><span class="n">tO</span><span class="o">),(</span><span class="n">nC</span><span class="o">,</span><span class="n">tC</span><span class="o">)),</span> <span class="n">i</span><span class="o">)</span> <span class="k">=&gt;</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">nameOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">nO</span> <span class="o">==</span> <span class="n">nC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(name) expected </span><span class="si">$nC</span><span class="s">, found </span><span class="si">$nO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="k">val</span> <span class="n">typeOk</span> <span class="k">=</span> <span class="k">if</span> <span class="n">tO</span> <span class="o">==</span> <span class="n">tC</span> <span class="n">then</span> <span class="nc">Nil</span> <span class="k">else</span> <span class="nc">List</span><span class="o">(</span><span class="s">s&#34;@</span><span class="si">$i</span><span class="s">(type) expected </span><span class="si">$tC</span><span class="s">, found </span><span class="si">$tO</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">              <span class="n">nameOk</span> <span class="o">++</span> <span class="n">typeOk</span>
</span></span><span class="line"><span class="cl">          <span class="o">}</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="k">_</span> <span class="k">=&gt;</span> <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">&#34;Unknown policy mod&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">mismatches</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="n">then</span>
</span></span><span class="line"><span class="cl">      <span class="n">report</span><span class="o">.</span><span class="n">errorAndAbort</span><span class="o">(</span><span class="s">s&#34;Modifier drift: </span><span class="si">${</span><span class="n">mismatches</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;; &#34;</span><span class="o">)</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="err">&#39;</span><span class="o">{</span> <span class="o">()</span> <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Try these:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">com.example.</span><span class="o">*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">A</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">B</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">A</span>, <span class="kt">B</span>, <span class="kt">PolicyMod.ExactByPosition</span><span class="o">]</span> <span class="c1">// ✅ same types by index
</span></span></span><span class="line"><span class="cl"><span class="c1">// ConformsMod[A, B, PolicyMod.ExactOrdered] // ❌ order matters → name mismatch
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIa</span><span class="o">(</span><span class="nc">ID</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CIb</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nc">ConformsMod</span><span class="o">[</span><span class="kt">CIa</span>, <span class="kt">CIb</span>, <span class="kt">PolicyMod.ExactCI</span><span class="o">]</span> <span class="o">//</span> <span class="o">✅</span> <span class="k">case</span><span class="err">‑</span><span class="n">insensitive</span> <span class="n">names</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Why these three? Because real systems need them:</p>
<ul>
<li><strong>ExactOrdered</strong>: For formats where position matters (some CSV parsers, protobuf with field numbers)</li>
<li><strong>ExactCI</strong>: Because someone always uses <code>UserId</code> when the contract says <code>userId</code></li>
<li><strong>ExactByPosition</strong>: For legacy systems that ignore field names entirely</li>
</ul>
<hr>
<h2 id="part-11---phantom-types-building-impossible-to-misuse-apis">Part 11 - Phantom types: Building impossible-to-misuse APIs</h2>
<p>Here&rsquo;s where things get interesting. The code, also uses phantom types, type-indexing / typestate builder patterns to build a pipeline builder that&rsquo;s <strong>impossible to misuse</strong>.
What are phantom types? They&rsquo;re type parameters that don&rsquo;t appear in the class body but control what you can do with an instance. Think of them as compile-time state machines.</p>
<pre class="mermaid">stateDiagram-v2
    [*] --&gt; Empty
    Empty --&gt; WithSource: addSource
    WithSource --&gt; WithTransform: transformAs[Next]
    WithTransform --&gt; Complete: addSink[Contract,P] + conforms
    Complete --&gt; [*]: build

    note right of WithTransform
      Evidence constraints:
      - S &lt;: WithSource to transform
      - S &lt;: WithTransform to sink
      - S = Complete to build
    end note
</pre><p>From the codebase <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Phantom type states
</span></span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Empty</span>         <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithSource</span>    <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">WithTransform</span> <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Complete</span>      <span class="k">extends</span> <span class="nc">BuilderState</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">S</span> <span class="k">&lt;:</span> <span class="kt">BuilderState</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="nc">private</span> <span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">steps</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">PipelineStep</span><span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add source when Empty
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSource</span><span class="o">[</span><span class="kt">C</span><span class="o">](</span><span class="n">src</span><span class="k">:</span> <span class="kt">TypedSource</span><span class="o">[</span><span class="kt">C</span><span class="o">])(</span><span class="n">using</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">])</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithSource</span>, <span class="kt">C</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only transform after adding source
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">f</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span><span class="o">)(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span><span class="o">,</span>  <span class="c1">// This constraint enforces order!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">Next</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">WithTransform</span>, <span class="kt">Next</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only add sink after transform
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">addSink</span><span class="o">[</span><span class="kt">R</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">](</span><span class="n">sink</span><span class="k">:</span> <span class="kt">TypedSink</span><span class="o">[</span><span class="kt">R</span><span class="o">])(</span><span class="n">using</span>
</span></span><span class="line"><span class="cl">    <span class="n">ev0</span><span class="k">:</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithTransform</span><span class="o">,</span>  <span class="c1">// Must have transform
</span></span></span><span class="line"><span class="cl">    <span class="n">ev1</span><span class="k">:</span> <span class="kt">SchemaConforms</span><span class="o">[</span><span class="kt">CurContract</span>, <span class="kt">R</span>, <span class="kt">P</span><span class="o">],</span>  <span class="c1">// Compile-time contract check!
</span></span></span><span class="line"><span class="cl">    <span class="n">sch</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">R</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">  <span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Complete</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="n">steps</span> <span class="k">:</span><span class="kt">+</span> <span class="kt">...</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Can only build when Complete
</span></span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">build</span><span class="o">(</span><span class="n">using</span> <span class="n">ev</span><span class="k">:</span> <span class="kt">S</span> <span class="o">=</span><span class="k">:</span><span class="o">=</span> <span class="nc">Complete</span><span class="o">)</span><span class="k">:</span> <span class="kt">SparkSession</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="o">(</span><span class="n">spark</span><span class="k">:</span> <span class="kt">SparkSession</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PipelineBuilder</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">apply</span><span class="o">[</span><span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Empty</span>, <span class="kt">CurContract</span><span class="o">](</span><span class="n">name</span><span class="o">,</span> <span class="nc">Nil</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="how-this-works">How this works:</h3>
<ol>
<li>
<p><strong>State transitions as types</strong> - Each method returns a new type parameter <code>S</code>. The compiler tracks which state you&rsquo;re in.</p>
</li>
<li>
<p><strong>Evidence constraints</strong> - <code>ev: S &lt;:&lt; WithSource</code> means &ldquo;S must be a subtype of WithSource&rdquo;. If it&rsquo;s not, this method doesn&rsquo;t exist.</p>
</li>
<li>
<p><strong>Compile-time state machine</strong> - You literally cannot call methods in the wrong order:</p>
</li>
</ol>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// ✅ This compiles
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;good&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">src</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// ❌ This doesn&#39;t compile - can&#39;t transform before adding source
</span></span></span><span class="line"><span class="cl"><span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="s">&#34;bad&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">Next</span><span class="o">](</span><span class="n">transform</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">ERROR</span><span class="k">:</span> <span class="kt">No</span> <span class="kt">implicit</span> <span class="kt">evidence</span> <span class="kt">S</span> <span class="k">&lt;:</span><span class="kt">&lt;</span> <span class="kt">WithSource</span></span></span></code></pre></td></tr></table>
</div>
</div><ol start="4">
<li><strong>Contract checking embedded</strong> - Notice <code>ev1: SchemaConforms[CurContract, R, P]</code> in <code>addSink</code>? That&rsquo;s where the compile-time contract check happens. The pipeline literally won&rsquo;t build if schemas drift.</li>
</ol>
<p>This pattern is called the <strong>Phantom Type Builder Pattern</strong>. For more details, see:</p>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala (Rhetorical Musings)</a>
</li>
</ul>
<p><strong>Why this matters</strong>: In production data pipelines, the order of operations matters. Read → Transform → Validate → Write. With phantom types, the compiler enforces this order. You can&rsquo;t accidentally write before transforming. You can&rsquo;t transform without a source.</p>
<p>This is the difference between &ldquo;here&rsquo;s how macros work&rdquo; and &ldquo;here&rsquo;s how to build production contract systems.&rdquo;</p>
<hr>
<h2 id="part-12---schema-evolution-and-versioning-handling-change-over-time">Part 12 - Schema evolution and versioning: Handling change over time</h2>
<p>Here&rsquo;s the reality: schemas evolve. Fields get added, deprecated, renamed. Teams roll out changes incrementally. The question isn&rsquo;t whether schemas will change - it&rsquo;s how you manage those changes without breaking production.</p>
<h3 id="the-schema-evolution-problem">The schema evolution problem</h3>
<p><strong>Scenario</strong>: You have <code>CustomerV1</code> running in production. Marketing wants to add a <code>segment</code> field for targeting. But you have 10 pipelines reading <code>CustomerV1</code>. How do you evolve without breaking everything?</p>
<p><strong>Wrong approach</strong>: Add <code>segment</code> to <code>CustomerV1</code>, deploy, hope for the best. Result: Some pipelines break because they don&rsquo;t handle the new field.</p>
<p><strong>Right approach</strong>: Version your contracts and use policies to manage transitions.</p>
<h3 id="versioning-strategy">Versioning strategy</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala - Current production
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala - New version with segment
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Notice <code>segment</code> is:</p>
<ol>
<li><strong>Optional</strong> (<code>Option[String]</code>)</li>
<li><strong>Has a default</strong> (<code>= None</code>)</li>
</ol>
<p>This makes it backward-compatible - old code can work with new data.</p>
<h3 id="migration-phases">Migration phases</h3>
<p><strong>Phase 1: Add the field (Backward policy)</strong></p>
<p>Deploy new producers writing <code>CustomerV2</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">producer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;write-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">rawData</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addSegment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">Policy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Backward allows optional fields
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Old consumers still read <code>CustomerV1</code>. They ignore the <code>segment</code> field. No breakage.</p>
<p><strong>Phase 2: Migrate consumers</strong></p>
<p>Update each consumer one by one:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">val</span> <span class="n">consumer</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;read-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">source</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">EnrichedCustomer</span><span class="o">](</span><span class="n">useSegment</span><span class="o">)</span>  <span class="c1">// Now uses segment field
</span></span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">EnrichedCustomer</span>, <span class="kt">Policy.Exact</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The compile-time contract ensures: &ldquo;Does this consumer handle the new schema?&rdquo; If you forget to update the case class, it won&rsquo;t compile.</p>
<p><strong>Phase 3: Deprecate V1</strong></p>
<p>Once all pipelines use <code>CustomerV2</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Mark V1 as deprecated
</span></span></span><span class="line"><span class="cl"><span class="nd">@deprecated</span><span class="o">(</span><span class="s">&#34;Use CustomerV2&#34;</span><span class="o">,</span> <span class="s">&#34;2025-10-01&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The compiler warns any remaining V1 usage. After a grace period, delete V1 entirely.</p>
<h3 id="handling-breaking-changes">Handling breaking changes</h3>
<p>What if you need to rename a field? Say, <code>email</code> → <code>emailAddress</code>?</p>
<p><strong>Non-breaking approach</strong>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add the new field, keep the old
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>              <span class="c1">// Old field
</span></span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New field
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Wait, that&rsquo;s duplication. Better:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span><span class="lnt">9
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Step 1: Add alias constructor
</span></span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV3</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span>       <span class="c1">// New canonical name
</span></span></span><span class="line"><span class="cl">  <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">CustomerV3</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">def</span> <span class="kt">fromV2</span><span class="o">(</span><span class="kt">v2:</span> <span class="kt">CustomerV2</span><span class="o">)</span><span class="kt">:</span> <span class="kt">CustomerV3</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">    <span class="nc">CustomerV3</span><span class="o">(</span><span class="n">v2</span><span class="o">.</span><span class="n">id</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">email</span><span class="o">,</span> <span class="n">v2</span><span class="o">.</span><span class="n">segment</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Step 2</strong>: Migrate producers to write <code>emailAddress</code>
<strong>Step 3</strong>: Migrate consumers to read <code>emailAddress</code>
<strong>Step 4</strong>: Remove the <code>fromV2</code> constructor</p>
<p>At each step, compile-time contracts verify: &ldquo;Does this transformation produce the expected schema?&rdquo;</p>
<h3 id="coexistence-during-migration">Coexistence during migration</h3>
<p>During migration, you have both V2 and V3 running. How to handle?</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span><span class="lnt">8
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">sealed</span> <span class="k">trait</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">V3</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">emailAddress</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">CustomerSchema</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Transformation handles both
</span></span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="n">normalize</span><span class="o">(</span><span class="n">schema</span><span class="k">:</span> <span class="kt">CustomerSchema</span><span class="o">)</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=</span> <span class="n">schema</span> <span class="k">match</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">V2</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">V3</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">email</span><span class="o">,</span> <span class="n">segment</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">case</span> <span class="n">v3</span><span class="k">:</span> <span class="kt">V3</span> <span class="o">=&gt;</span> <span class="n">v3</span></span></span></code></pre></td></tr></table>
</div>
</div><p>The contract checks both paths:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V2</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Backward</span><span class="o">]]</span>  <span class="c1">// V2 → V3 allowed
</span></span></span><span class="line"><span class="cl"><span class="n">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">V3</span>, <span class="kt">V3Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span>     <span class="o">//</span> <span class="n">V3</span> <span class="o">→</span> <span class="n">V3</span> <span class="n">exact</span> <span class="k">match</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="real-world-migration-backward--exact">Real-world migration: Backward → Exact</h3>
<p>Let&rsquo;s look at a complete migration scenario from the code in github. This is exactly how you&rsquo;d roll out schema changes in production.</p>
<p>From <code>CtdcPoc.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="cm">/** Sink contract: the target schema we promise to write. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerContract</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Producer: imagine an upstream source adds an extra field `segment`. */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerProducer</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">segment</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cm">/** Declared &#34;Next&#34; schema after a transform (dropping `segment`). */</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerNext</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Step 1: Migration phase - use Backward policy
</span></span></span><span class="line"><span class="cl"><span class="c1">// Producer has extra field, but contract allows it during migration
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">srcB</span> <span class="k">=</span> <span class="nc">TypedSource</span><span class="o">[</span><span class="kt">CustomerProducer</span><span class="o">](</span><span class="s">&#34;csv&#34;</span><span class="o">,</span> <span class="n">inPath</span><span class="o">,</span> <span class="nc">Map</span><span class="o">(</span><span class="s">&#34;header&#34;</span> <span class="o">-&gt;</span> <span class="s">&#34;true&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">sinkB</span> <span class="k">=</span> <span class="nc">TypedSink</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="n">tmpOutB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">dropExtras</span><span class="k">:</span> <span class="kt">DataFrame</span> <span class="o">=&gt;</span> <span class="nc">DataFrame</span> <span class="k">=</span> <span class="k">_</span><span class="o">.</span><span class="n">select</span><span class="o">(</span><span class="n">$</span><span class="s">&#34;id&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;email&#34;</span><span class="o">,</span> <span class="n">$</span><span class="s">&#34;age&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planB</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;CSV -&gt; Parquet B: transformAs[CustomerNext], Exact&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">srcB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerNext</span><span class="o">](</span><span class="s">&#34;drop segment&#34;</span><span class="o">)(</span><span class="n">dropExtras</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sinkB</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="whats-happening-here">What&rsquo;s happening here:</h3>
<p><strong>Phase 1: Producer adds field</strong></p>
<ul>
<li>Upstream adds <code>segment</code> field to <code>CustomerProducer</code></li>
<li>Our contract is still <code>CustomerContract</code> (no <code>segment</code>)</li>
<li>We use <code>transformAs[CustomerNext]</code> to explicitly drop the extra field</li>
<li>Then check <code>CustomerNext</code> conforms to <code>CustomerContract</code> under <strong>Exact</strong></li>
</ul>
<p><strong>Why this works:</strong></p>
<ol>
<li>The transform explicitly declares the output schema (<code>CustomerNext</code>)</li>
<li>The compiler checks: Does <code>CustomerNext</code> match <code>CustomerContract</code>? Yes (both have id, email, age)</li>
<li>If we later change <code>CustomerNext</code> by accident, compilation fails</li>
</ol>
<p><strong>Phase 2: Stabilization</strong>
Once the migration is done:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Later: Everyone uses CustomerContract, no transform needed
</span></span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">planStable</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">  <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerContract</span><span class="o">](</span><span class="s">&#34;stable&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSource</span><span class="o">(</span><span class="n">contractSource</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">noTransform</span>  <span class="c1">// Direct pass-through
</span></span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerContract</span>, <span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This is real-world stuff. The code shows you exactly how to migrate schemas safely.</p>
<pre class="mermaid">flowchart TD
    P1[&#34;Producer adds field &#39;segment&#39;&#34;] --&gt; T1[&#34;transformAs[CustomerNext]\n(drop extras)&#34;]
    T1 --&gt; V1[&#34;summon Conforms[CustomerNext,CustomerContract,Policy.Exact]&#34;]
    V1 --&gt; W1[&#34;write sink&#34;]

    subgraph Phase 1 [&#34;Migration window&#34;]
      direction TB
      P1 --&gt; T1 --&gt; V1 --&gt; W1
    end

    W1 --&gt; P2[&#34;Stabilize: sources match contract&#34;]
    P2 --&gt; V2[&#34;Conforms[CustomerContract,CustomerContract,Policy.Exact]&#34;]
    V2 --&gt; W2[&#34;noTransform + write&#34;]
</pre><hr>
<h2 id="part-13---testing-compiletime-for-real-copypaste-tests">Part 13 - Testing compile‑time, for real (copy/paste tests)</h2>
<p>How do you test that code fails to compile? With <code>assertDoesNotCompile</code>:</p>
<pre class="mermaid">sequenceDiagram
    participant Dev
    participant CI as &#34;GitHub Actions&#34;
    participant SBT as &#34;sbt testOnly *CompileTimeSpec&#34;
    Dev-&gt;&gt;CI: open PR
    CI-&gt;&gt;SBT: run compile-fail suite
    SBT-&gt;&gt;SBT: assertCompiles / assertDoesNotCompile
    alt all pass
        SBT--&gt;&gt;CI: success
        CI--&gt;&gt;Dev: green check
    else drift detected
        SBT--&gt;&gt;CI: failure with errorAndAbort msg
        CI--&gt;&gt;Dev: red X + compiler diff
    end
</pre><div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// src/test/scala/com/example/CompileTimeSpec.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">org.scalatest.funsuite.AnyFunSuite</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CompileTimeSpec</span> <span class="k">extends</span> <span class="nc">AnyFunSuite</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">test</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">Exact</span> <span class="kt">should</span> <span class="kt">fail</span> <span class="kt">when</span> <span class="kt">a</span> <span class="kt">required</span> <span class="kt">field</span> <span class="kt">is</span> <span class="kt">missing</span><span class="err">&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="kt">assertDoesNotCompile</span><span class="o">(</span><span class="err">&#34;</span><span class="kt">summon</span><span class="o">[</span><span class="kt">Conforms</span><span class="o">[</span><span class="kt">OutMissing</span>, <span class="kt">Contract</span>, <span class="kt">Policy.Exact</span><span class="o">]]</span><span class="err">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Backward should pass for the same mismatch&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OutMissing, Contract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;By‑Position should accept re‑ordered names with same types&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactByPosition]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Ordered should reject the same re‑ordering&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;ConformsMod[A, B, PolicyMod.ExactOrdered]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Pipeline builder enforces correct order&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">      PipelineBuilder[Contract](&#34;bad&#34;)
</span></span></span><span class="line"><span class="cl"><span class="s">        .transformAs[Next](identity)  // Can&#39;t transform without source
</span></span></span><span class="line"><span class="cl"><span class="s">    &#34;&#34;&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="n">test</span><span class="o">(</span><span class="s">&#34;Nested types conform correctly&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="n">assertCompiles</span><span class="o">(</span><span class="s">&#34;summon[Conforms[OrderOut, OrderContract, Policy.Backward]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>You can wire this into CI:</p>
<div class="highlight highlight-yaml" data-lang="yaml"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/compile-fail.yml (concept)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">compile-fail</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">pull_request]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cf</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-java@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">distribution</span><span class="p">:</span><span class="w"> </span><span class="nt">temurin, java-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;17&#39;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Compile‑fail suite</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">sbt -v &#34;testOnly *CompileTimeSpec&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><p>This is gold. Your CI literally proves that broken schemas can&rsquo;t be deployed.</p>
<hr>
<h2 id="part-14---scala-2-how-to-do-the-same-and-whats-different">Part 14 - Scala 2: how to do the same (and what&rsquo;s different)</h2>
<p>Scala 2 uses def macros with a <code>Context</code> (blackbox/whitebox). The idea is identical, but the implementation uses <code>c.universe</code> trees instead of <code>Expr</code>/quotes.</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// build.sbt (S2 module)
</span></span></span><span class="line"><span class="cl"><span class="c1">// scalaVersion := &#34;2.13.14&#34;
</span></span></span><span class="line"><span class="cl"><span class="c1">// libraryDependencies += &#34;org.scala-lang&#34; % &#34;scala-reflect&#34; % scalaVersion.value
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// src/main/scala-2/com/example/ConformsS2.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">com.example</span>
</span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">scala.reflect.macros.blackbox</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">class</span> <span class="nc">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nc">object</span> <span class="nc">ConformsS2</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">implicit</span> <span class="k">def</span> <span class="n">materialize</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">Policy</span><span class="o">](</span><span class="k">implicit</span> <span class="n">sOut</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">],</span> <span class="n">sCon</span><span class="k">:</span> <span class="kt">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">])</span><span class="k">:</span> <span class="kt">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">    <span class="n">macro</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">conformsImpl</span><span class="o">[</span><span class="kt">Out:</span> <span class="kt">c.WeakTypeTag</span>, <span class="kt">Contract:</span> <span class="kt">c.WeakTypeTag</span>, <span class="kt">P:</span> <span class="kt">c.WeakTypeTag</span><span class="o">](</span><span class="n">c</span><span class="k">:</span> <span class="kt">blackbox.Context</span><span class="o">)(</span><span class="n">sOut</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">Out</span><span class="o">]],</span> <span class="n">sCon</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">Contract</span><span class="o">]])</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">import</span> <span class="nn">c.universe._</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">fieldsOf</span><span class="o">[</span><span class="kt">T:</span> <span class="kt">c.WeakTypeTag</span><span class="o">](</span><span class="n">sh</span><span class="k">:</span> <span class="kt">c.Expr</span><span class="o">[</span><span class="kt">Shape</span><span class="o">[</span><span class="kt">T</span><span class="o">]])</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span>,<span class="kt">Boolean</span>,<span class="kt">Boolean</span><span class="o">)]</span> <span class="k">=</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="n">c</span><span class="o">.</span><span class="n">eval</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="nc">Expr</span><span class="o">[</span><span class="kt">List</span><span class="o">[(</span><span class="kt">String</span>,<span class="kt">String</span>,<span class="kt">Boolean</span>,<span class="kt">Boolean</span><span class="o">)]](</span><span class="n">q</span><span class="s">&#34;$sh.fields.map(f =&gt; (f.name, f.tpe, f.hasDefault, f.isOptional))&#34;</span><span class="o">))</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">out</span>      <span class="k">=</span> <span class="n">fieldsOf</span><span class="o">[</span><span class="kt">Out</span><span class="o">](</span><span class="n">sOut</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contract</span> <span class="k">=</span> <span class="n">fieldsOf</span><span class="o">[</span><span class="kt">Contract</span><span class="o">](</span><span class="n">sCon</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">outMap</span>      <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">contractMap</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">map</span><span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span><span class="n">opt</span><span class="o">)}.</span><span class="n">toMap</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">missing</span> <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">opt</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">outMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s">:</span><span class="si">$t$</span><span class="s">{if (opt) &#34;</span> <span class="o">(</span><span class="n">optional</span><span class="o">)</span><span class="s">&#34; else &#34;&#34;}&#34;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">extra</span>   <span class="k">=</span> <span class="n">out</span><span class="o">.</span><span class="n">collect</span>      <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">contractMap</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">n</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">    <span class="k">val</span> <span class="n">mism</span>    <span class="k">=</span> <span class="n">contract</span><span class="o">.</span><span class="n">collect</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="n">t</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_1</span> <span class="o">!=</span> <span class="n">t</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> expected </span><span class="si">$t</span><span class="s">, found </span><span class="si">${</span><span class="n">outMap</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">_1</span><span class="si">}</span><span class="s">&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="k">case</span> <span class="o">(</span><span class="n">n</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="k">_</span><span class="o">,</span><span class="n">o</span><span class="o">)</span> <span class="k">if</span> <span class="n">outMap</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">n</span><span class="o">).</span><span class="n">exists</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">!=</span> <span class="n">o</span><span class="o">)</span>  <span class="k">=&gt;</span> <span class="s">s&#34;</span><span class="si">$n</span><span class="s"> optionality mismatch&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="o">(</span><span class="n">missing</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">extra</span><span class="o">.</span><span class="n">nonEmpty</span> <span class="o">||</span> <span class="n">mism</span><span class="o">.</span><span class="n">nonEmpty</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">      <span class="n">c</span><span class="o">.</span><span class="n">abort</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="n">enclosingPosition</span><span class="o">,</span> <span class="s">s&#34;Drift: missing=</span><span class="si">${</span><span class="n">missing</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;,&#34;</span><span class="o">)</span><span class="si">}</span><span class="s"> extra=</span><span class="si">${</span><span class="n">extra</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;,&#34;</span><span class="o">)</span><span class="si">}</span><span class="s"> mism=</span><span class="si">${</span><span class="n">mism</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">&#34;;&#34;</span><span class="o">)</span><span class="si">}</span><span class="s">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span>
</span></span><span class="line"><span class="cl">      <span class="n">c</span><span class="o">.</span><span class="nc">Expr</span><span class="o">[</span><span class="kt">ConformsS2</span><span class="o">[</span><span class="kt">Out</span>, <span class="kt">Contract</span>, <span class="kt">P</span><span class="o">]](</span><span class="n">q</span><span class="s">&#34;new _root_.com.example.ConformsS2[${weakTypeOf[Out]}, ${weakTypeOf[Contract]}, ${weakTypeOf[P]}]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Key differences:</p>
<ul>
<li>Scala 3: <code>inline</code> + <code>${ ... }</code> splices; Scala 2: <code>macro def</code> with a <code>Context</code></li>
<li>Error reporting: <code>report.errorAndAbort</code> (Scala 3) vs <code>c.abort</code> (Scala 2)</li>
<li>Trees: <code>Expr[T]</code> (Scala 3) vs raw <code>Tree</code> (Scala 2)</li>
<li>Type inspection: <code>TypeRepr</code> (Scala 3) vs <code>Type</code> (Scala 2)</li>
</ul>
<p>Scala 3 macros are safer. The quoted API prevents many common macro bugs. But Scala 2 macros work fine if you&rsquo;re careful.</p>
<p>For derivation, check out <a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia</a>
 - it works across both versions.</p>
<hr>
<h2 id="part-15---integration-with-spark-runtime-defense-in-depth">Part 15 - Integration with Spark: Runtime defense-in-depth</h2>
<p>The code in github also shows how to mirror compile-time policies with Spark&rsquo;s built-in structural comparators for runtime validation.</p>
<p>From <code>SparkCore.scala</code>:</p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Derive Spark StructType from case class at compile time
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">struct</span><span class="k">:</span> <span class="kt">StructType</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">SparkSchema</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">inline</span> <span class="kt">given</span> <span class="kt">derived</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span><span class="k">:</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="k">=</span> <span class="n">$</span><span class="o">{</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C</span><span class="o">]</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Macro that converts case class → StructType
</span></span></span><span class="line"><span class="cl">  <span class="k">private</span> <span class="k">def</span> <span class="n">sparkSchemaImpl</span><span class="o">[</span><span class="kt">C:</span> <span class="kt">Type</span><span class="o">](</span><span class="n">using</span> <span class="nc">Quotes</span><span class="o">)</span><span class="k">:</span> <span class="kt">Expr</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">C</span><span class="o">]]</span> <span class="k">=</span> <span class="o">...</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Runtime policy mapping to Spark comparators
</span></span></span><span class="line"><span class="cl"><span class="k">trait</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">P</span> <span class="k">&lt;:</span> <span class="kt">SchemaPolicy</span><span class="o">]</span><span class="nc">:</span>
</span></span><span class="line"><span class="cl">  <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">PolicyRuntime</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.Exact.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsIgnoreCaseAndNullability</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactByPosition.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurally</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="n">ignoreNullability</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">given</span> <span class="nc">PolicyRuntime</span><span class="o">[</span><span class="kt">SchemaPolicy.ExactOrdered.</span><span class="k">type</span><span class="o">]</span> <span class="k">with</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="n">ok</span><span class="o">(</span><span class="n">found</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">,</span> <span class="n">expected</span><span class="k">:</span> <span class="kt">StructType</span><span class="o">)</span> <span class="k">=</span>
</span></span><span class="line"><span class="cl">      <span class="nc">DataType</span><span class="o">.</span><span class="n">equalsStructurallyByName</span><span class="o">(</span><span class="n">found</span><span class="o">,</span> <span class="n">expected</span><span class="o">,</span> <span class="k">_</span> <span class="o">==</span> <span class="k">_</span><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p><strong>Two-layer validation strategy</strong>:</p>
<ol>
<li><strong>Compile-time</strong> (macros) - Catches schema drift before deployment</li>
<li><strong>Runtime</strong> (Spark comparators) - Defensive check for external data sources</li>
</ol>
<pre class="mermaid">flowchart TD
    F[&#34;found: StructType&#34;] --- E[&#34;expected: StructType&#34;]
    F --&gt; EX[&#34;Policy.Exact&#34;] --&gt; C1[&#34;equalsIgnoreCaseAndNullability(found,expected)&#34;]
    F --&gt; EP[&#34;Policy.ExactByPosition&#34;] --&gt; C2[&#34;equalsStructurally(found,expected,true)&#34;]
    F --&gt; EO[&#34;Policy.ExactOrdered&#34;] --&gt; C3[&#34;equalsStructurallyByName(found,expected,resolver)&#34;]
</pre><p>For more on Spark&rsquo;s structural comparators, see the <a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">DataType API docs</a>
.</p>
<hr>
<h2 id="part-16---a-tiny-migration-playbook-you-can-adopt-tomorrow">Part 16 - A tiny migration playbook you can adopt tomorrow</h2>
<p>Here&rsquo;s how I use these policies in practice:</p>
<ul>
<li><strong>Critical paths → Exact</strong>. No surprises. If schemas don&rsquo;t match exactly, builds fail.</li>
<li><strong>Producer adding optional fields → Backward</strong> during rollout. Allows new optional fields on the producer side.</li>
<li><strong>Consumer that can ignore extras → Forward</strong> during rollout. Lets the consumer tolerate extra fields.</li>
<li><strong>After stabilization → tighten back to Exact</strong>.</li>
</ul>
<p>And remember:</p>
<ul>
<li>Keep transformations pure (no side effects inside map/filter)</li>
<li>Place I/O at edges (read once, write once)</li>
<li>Make side-effects idempotent (so retries don&rsquo;t duplicate writes)</li>
</ul>
<blockquote>
  <p>Compile-time prevents drift; runtime manages reality. Tests catch behavior, not shapes.</p>

</blockquote><hr>
<h2 id="part-17---production-patterns-what-i-learned-shipping-this">Part 17 - Production patterns: What I learned shipping this</h2>
<h3 id="pattern-1-organize-contracts-by-version">Pattern 1: Organize contracts by version</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV1.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV1</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// contracts/CustomerV2.scala
</span></span></span><span class="line"><span class="cl"><span class="k">package</span> <span class="nn">contracts</span>
</span></span><span class="line"><span class="cl"><span class="k">final</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomerV2</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// pipelines/CustomerMigration.scala
</span></span></span><span class="line"><span class="cl"><span class="k">import</span> <span class="nn">contracts._</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">val</span> <span class="n">migration</span> <span class="k">=</span> <span class="nc">PipelineBuilder</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="s">&#34;v1-to-v2&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSource</span><span class="o">[</span><span class="kt">CustomerV1</span><span class="o">](</span><span class="n">sourceV1</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">transformAs</span><span class="o">[</span><span class="kt">CustomerV2</span><span class="o">](</span><span class="n">addNameField</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">CustomerV2</span>, <span class="kt">SchemaPolicy.Exact</span><span class="o">](</span><span class="n">sinkV2</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="o">.</span><span class="n">build</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-2-use-companion-objects-for-schema-caching">Pattern 2: Use companion objects for schema caching</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Long</span><span class="o">,</span> <span class="n">email</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="k">object</span> <span class="nc">User</span><span class="k">:</span>
</span></span><span class="line"><span class="cl">  <span class="kt">given</span> <span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">summon</span><span class="o">[</span><span class="kt">SparkSchema</span><span class="o">[</span><span class="kt">User</span><span class="o">]]</span>  <span class="o">//</span> <span class="nc">Compute</span> <span class="n">once</span><span class="o">,</span> <span class="n">reuse</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-3-document-policy-choices-inline">Pattern 3: Document policy choices inline</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Good: Explicit reasoning
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Backward</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="c1">// Allow optional fields during Q2 migration
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Bad: No context
</span></span></span><span class="line"><span class="cl"><span class="o">.</span><span class="n">addSink</span><span class="o">[</span><span class="kt">Contract</span>, <span class="kt">SchemaPolicy.Full</span><span class="o">](</span><span class="n">sink</span><span class="o">)</span>  <span class="o">//</span> <span class="nc">Why</span> <span class="nc">Full</span><span class="o">?</span> <span class="nc">When</span> <span class="n">will</span> <span class="n">we</span> <span class="n">tighten</span><span class="o">?</span></span></span></code></pre></td></tr></table>
</div>
</div><h3 id="pattern-4-test-your-compile-fail-cases">Pattern 4: Test your compile-fail cases</h3>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="c1">// Keep these in version control as documentation
</span></span></span><span class="line"><span class="cl"><span class="n">test</span><span class="o">(</span><span class="s">&#34;CustomerV1 should not conform to CustomerV2 under Exact&#34;</span><span class="o">)</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">assertDoesNotCompile</span><span class="o">(</span><span class="s">&#34;summon[Conforms[CustomerV1, CustomerV2, Policy.Exact]]&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></td></tr></table>
</div>
</div><hr>
<h2 id="appendix---full-demo-project-skeleton-copy--run">Appendix - Full demo project skeleton (copy &amp; run)</h2>
<div class="highlight highlight-text" data-lang="text"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">├── build.sbt
</span></span><span class="line"><span class="cl">└── src
</span></span><span class="line"><span class="cl">    ├── main
</span></span><span class="line"><span class="cl">    │   └── scala
</span></span><span class="line"><span class="cl">    │       └── com/example/
</span></span><span class="line"><span class="cl">    │           ├── IntroMacro.scala
</span></span><span class="line"><span class="cl">    │           ├── Shape.scala
</span></span><span class="line"><span class="cl">    │           ├── Policy.scala
</span></span><span class="line"><span class="cl">    │           ├── Conforms.scala
</span></span><span class="line"><span class="cl">    │           └── SparkCore.scala
</span></span><span class="line"><span class="cl">    └── test
</span></span><span class="line"><span class="cl">        └── scala
</span></span><span class="line"><span class="cl">            └── com/example/
</span></span><span class="line"><span class="cl">                ├── ShapeSpec.scala
</span></span><span class="line"><span class="cl">                ├── ConformsSpec.scala
</span></span><span class="line"><span class="cl">                └── CompileTimeSpec.scala</span></span></code></pre></td></tr></table>
</div>
</div><p><strong>build.sbt (Scala 3)</strong></p>
<div class="highlight highlight-scala" data-lang="scala"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-scala" data-lang="scala"><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">scalaVersion</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;3.3.3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nc">ThisBuild</span> <span class="o">/</span> <span class="n">organization</span> <span class="k">:</span><span class="o">=</span> <span class="s">&#34;com.example&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.apache.spark&#34;</span> <span class="o">%%</span> <span class="s">&#34;spark-sql&#34;</span> <span class="o">%</span> <span class="s">&#34;3.5.1&#34;</span> <span class="o">%</span> <span class="s">&#34;provided&#34;</span><span class="o">,</span>
</span></span><span class="line"><span class="cl">  <span class="s">&#34;org.scalatest&#34;</span> <span class="o">%%</span> <span class="s">&#34;scalatest&#34;</span> <span class="o">%</span> <span class="s">&#34;3.2.17&#34;</span> <span class="o">%</span> <span class="nc">Test</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></td></tr></table>
</div>
</div><p>Run it:</p>
<div class="highlight highlight-bash" data-lang="bash"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sbt <span class="s2">&#34;runMain ctdc.CtdcPoc&#34;</span></span></span></code></pre></td></tr></table>
</div>
</div><blockquote>
  <p>Tip: For Scala 2, add a sibling module with scala‑reflect and copy the S2 file from above.</p>

</blockquote><hr>
<h2 id="references">References</h2>
<h3 id="scala-3-macros--metaprogramming">Scala 3 Macros &amp; Metaprogramming</h3>
<ul>
<li><a href="https://rockthejvm.com/courses/scala-macros-and-metaprogramming" rel="noopener external" target="_blank">Rock the JVM - Scala Macros and Metaprogramming (course)</a>
</li>
<li><a href="https://rockthejvm.com/articles/scala-3-macros-comprehensive-guide" rel="noopener external" target="_blank">Rock the JVM - Scala 3 Macros comprehensive guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/macros.html" rel="noopener external" target="_blank">Scala 3 Macros - Official overview</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/reflection.html" rel="noopener external" target="_blank">Scala 3 Reflection guide</a>
</li>
<li><a href="https://docs.scala-lang.org/scala3/guides/macros/best-practices.html" rel="noopener external" target="_blank">Scala 3 Macro best practices</a>
</li>
</ul>
<h3 id="apache-spark-integration">Apache Spark Integration</h3>
<ul>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/types/DataType%24.html" rel="noopener external" target="_blank">Spark DataType structural comparators</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/api/scala/org/apache/spark/sql/DatasetHolder.html" rel="noopener external" target="_blank">Spark Dataset implicits (toDF/toDS)</a>
</li>
<li><a href="https://spark.apache.org/docs/3.5.1/sql-data-sources-csv.html" rel="noopener external" target="_blank">Spark CSV data source</a>
</li>
<li><a href="https://index.scala-lang.org/vincenzobaz/spark-scala3-encoders" rel="noopener external" target="_blank">Scala 3 encoders for Spark (community)</a>
</li>
</ul>
<h3 id="phantom-types--type-level-programming">Phantom Types &amp; Type-Level Programming</h3>
<ul>
<li><a href="https://xebia.com/blog/compile-safe-builder-pattern-using-phantom-types-in-scala/" rel="noopener external" target="_blank">Compile-safe builder pattern using phantom types (Xebia)</a>
</li>
<li><a href="https://blog.rhetoricalmusings.com/posts/builder1/" rel="noopener external" target="_blank">Phantom Types in Scala - Builder Pattern (Rhetorical Musings)</a>
</li>
</ul>
<h3 id="other-resources">Other Resources</h3>
<ul>
<li><a href="https://github.com/softwaremill/magnolia" rel="noopener external" target="_blank">Magnolia - Generic derivation library</a>
</li>
</ul>
<hr>
<h2 id="what-compile-time-contracts-cant-catch">What compile-time contracts can&rsquo;t catch</h2>
<p>Let&rsquo;s be honest about the boundaries. Compile-time contracts handle a lot, but they&rsquo;re not magic. Here&rsquo;s what they <strong>can&rsquo;t</strong> do:</p>
<table>
	<thead>
			<tr>
					<th>Category</th>
					<th>What It Can&rsquo;t Catch</th>
					<th>Why</th>
					<th>Solution</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><strong>External data drift</strong></td>
					<td>• Third-party APIs that change without telling you; • Kafka topics from teams you don&rsquo;t coordinate with; • S3 buckets written by external vendors</td>
					<td>You don&rsquo;t control their build process</td>
					<td>Use runtime validation (schema registries, data contracts)</td>
			</tr>
			<tr>
					<td><strong>Late-arriving schema changes</strong></td>
					<td>• Schema registry updated after your deploy; • Database columns added while your job runs</td>
					<td>Compile-time happens before deploy</td>
					<td>Runtime checks with version monitoring</td>
			</tr>
			<tr>
					<td><strong>Data quality issues</strong></td>
					<td>• Nulls where you expect values (even if field is non-optional); • Out-of-range numbers (age = -5); • Malformed strings (email without @)</td>
					<td>Compile-time checks structure, not content</td>
					<td>Great Expectations, Deequ, or custom validators</td>
			</tr>
			<tr>
					<td><strong>Non-breaking additions</strong></td>
					<td>• Upstream adds optional field you don&rsquo;t use yet</td>
					<td>This is often fine! <code>Forward</code> policy handles it</td>
					<td>Policy-based awareness or stricter monitoring</td>
			</tr>
			<tr>
					<td><strong>Partial batch failures</strong></td>
					<td>• 1000 records match schema, 5 don&rsquo;t</td>
					<td>Compile-time is binary (compiles or doesn&rsquo;t)</td>
					<td>Runtime validation with error tables/quarantine</td>
			</tr>
	</tbody>
</table>
<p><strong>The takeaway</strong>: Compile-time contracts catch drift <strong>within your controlled codebase</strong> during development and deployment. For everything else - external sources, runtime changes, data quality - layer in runtime validation. Think defense-in-depth: compile-time as the first gate, runtime as the safety net.</p>
<hr>
<p><strong>TL;DR</strong></p>
<ul>
<li>Compile-time evidence + policy types make schema intent explicit and enforceable</li>
<li>TypeInspector pattern organizes type inspection into composable utilities</li>
<li>Phantom types with type-indexing/type-state builders create impossible-to-misuse APIs through compile-time state machines</li>
<li>Field vs element optionality matters for nested types</li>
<li>Nested structures (Maps, Lists, case classes) work through recursive shape building</li>
<li>Real migrations: use Backward during rollout, tighten to Exact after stabilization</li>
<li>Spark&rsquo;s comparators provide runtime defense-in-depth</li>
<li>Schema evolution needs versioning, coexistence strategies, and policy-based transitions</li>
<li>Developer ergonomics matter: clear errors, fast compile times, easy onboarding</li>
<li>Compile-time doesn&rsquo;t replace runtime validation - it complements it for controlled systems</li>
<li>This catches many drift classes early, but you still need runtime checks for external data</li>
<li>Compile-time contracts don&rsquo;t make pipelines unbreakable - they make breakage predictable</li>
</ul>
]]></content:encoded><enclosure url="https://vitthalmirji.com/images/og/2025-09-compile-time-data-contracts.png" type="image/png"/></item></channel></rss>