Vitthal Mirji

Engineer building platforms - Data, Software & OSS | Staff Software Engineer

Scala · Spark · Python · Rust · Data Engineering · OSS · Functional programming · Databricks

🇩🇪Germany

+49-151-28738427

[email protected]

vitthalmirji.com

github.com/vim89

linkedin.com/in/vitthal10

Download PDF resume

Summary	Staff Software Engineer with 12+ years building internal developer platforms, type-safe SDKs, and data infrastructure at Walmart-scale and Deloitte Fortune 500 clients. Led architecture for platforms processing 100M+ daily records on Spark/Databricks and Kafka/Flink, with sub-2h data freshness. Mentored 8 senior engineers on functional Scala and staff-level readiness. Current OSS/R&D: datacrate, llm4s, flowforge, DataFusion/Comet/Ballista. Creator of flowforge and toon4s, core contributor to llm4s, cask, and scalasql. International speaker (Scala.io Paris, Data Makers Fest, Lambda Days, OAISYS).
Key Accomplishments	Designed internal developer platform adopted by 150+ engineers across 40+ teams - six type-safe SDK libraries (contracts, archetypes, monitoring, quality, operations) that cut delivery from six weeks to two and reduced incidents by ~82%. Engineered closed-loop merchant analytics platform processing 100M+ daily records on Spark/Databricks, unlocking 75% business growth with sub-2h data freshness. Modernized Deloitte Fortune 500 estates (Aflac, Marriott, Eli Lilly), trimming infra spend by 62% (US$2.4M→US$900K) and boosting SLAs to 99.5%. Built observability platform (DPAT automation + data-quality engines) validating 5TB+/day at 99.9% accuracy, reducing MTTR by 60% and shrinking QA sign-off from three days to four hours.
Core Skills	Architecture: API design, SDK authorship, system design, data contracts, streaming observability, data governance, platform engineering, data warehousing, data modeling. Expert: Scala (Typelevel, Cats Effect, meta-programming), Python, SQL, Java. Productive / Actively Growing: Kotlin, Rust, Bash. Foundational: C++. Engines & Cloud: Spark, Databricks, Flink, Kafka, Beam, DataFusion, Comet, Ballista · GCP (BigQuery, Dataproc, GCS, Pub/Sub), AWS (S3, Glue, EMR) · Docker, Kubernetes. AI & Platform: llm4s, LLM agents, RAG, embeddings, evals, tracing, typed errors, provider abstractions, SDK design, observability, AI-assisted engineering workflows. Leadership: Staff+ design reviews, roadmap definition, cross-team adoption playbooks, senior IC mentoring (8 engineers), incident/SLO programs, product-to-architecture translation.
Work	llm4s foundation and Vimastra London, UK (Remote) · Mar 2026 – Present Senior Software Engineer - Data & Platforms llm4s core contributor and co-maintainer for type-safe LLM agents, RAG, tracing, evals, typed errors, and provider abstractions. Built flowforge, toon4s, and datacrate around compile-time data contracts, token-efficient formats, DataFusion/Comet/Ballista, and predictable APIs. Walmart Global Tech Bangalore · Nov 2023 – Mar 2026 Staff Software Engineer (IC5 / SDE5 - Platform Architect) Built flowforge - a compile-time schema-contract Scala SDK - adopted by 40+ teams / 150+ engineers; invalid schemas failed in CI before production. Validated 5TB+/day at 99.9% SLA; reduced incidents 82%, MTTR 60%, and QA sign-off from three days to four hours. Designed Scala/Python SDKs for contracts, monitoring, quality, and operations; reduced boilerplate 70%; mentored 8 senior engineers. Walmart Global Tech Bangalore · May 2021 – Oct 2023 Lead Software Engineer / Tech Lead (Platform & Data Infrastructure) Led 13-engineer squad delivering closed-loop analytics platform: 100M+ daily records with sub-2h freshness. Built Kafka supplier pipelines and reusable Scala/Python SDKs adopted by 12 teams: latency 48h to 2h, forecast accuracy +35%, Spark/Dataproc cost -40%. Deloitte Consulting (U.S. offices) Mumbai · Apr 2017 – May 2021 Specialist Senior - Data Platform Engineering Built enterprise data platforms for Aflac, Eli Lilly, and Marriott; 350+ table EDW; data quality SLA improved from 85% to 99.5%. Migrated 100+ TiB from on-premise Hadoop/RDBMS to AWS S3/Glue with zero data loss; automated reconciliation and CI/CD validation. Led engineering pods of 6-8 engineers; supported hiring, mentoring, and delivery across Fortune 500 engagements. Bista Solutions Inc. Mumbai · Feb 2016 – Apr 2017 Senior Software Engineer - Data Analytics Built Spark Streaming + Kafka real-time supply-chain telemetry for U.S. automotive clients: forecast accuracy +60%, inventory cost -10%. Nihilent Technologies Pune · Nov 2015 – Feb 2016 Software Engineer - Platform & Data Built ingestion framework for 1,000+ RDBMS tables into Hadoop HDFS, adopted across 50+ programs.
Open Source	Author and core contributor to Scala libraries spanning platform engineering, compile-time safety, LLM infrastructure, and upstream ecosystem contributions. cask (github.com/com-lihaoyi/cask): Added DB transaction support and application configuration handling to Li Haoyi's Scala web framework. scalasql (github.com/com-lihaoyi/scalasql): Added COUNT and COUNT(DISTINCT) aggregation support to Li Haoyi's type-safe SQL library. flowforge (github.com/vim89/flowforge): Type-safe Scala data-engineering SDK with compile-time contract validation. Prevents schema drift before deploy. datacrate: Rust typed data pipelines built on DataFusion/Comet/Ballista, bringing compile-time schema contracts to the Rust data-engineering ecosystem. toon4s (github.com/com-vitthalmirji/toon4s): Token-Oriented Object Notation for JVM. Reduces LLM token costs by 30–60% while maintaining JSON compatibility. 500+ tests, JMH benchmarks, Maven Central. llm4s (github.com/llm4s/llm4s): Core contributor and GSoC mentor. Type-safe LLM framework for Scala with multi-provider support, agents, RAG, and tracing. compile-time-data-contracts (github.com/vim89/compile-time-data-contracts): Scala 3 macro-based schema validation. Pipelines fail to compile on schema drift. datapipelines-essentials-python (github.com/vim89/datapipelines-essentials-python): Reusable Spark ETL library with SparkSession extensions, DataFrame validation, and SQL functions. abortable-bytes (github.com/vim89/abortable-bytes): Kyo effect-system POC for cancel-safe cloud uploads with automatic cleanup. spark-xml: Contributor to Databricks' Spark XML library. Rock the JVM (rockthejvm.com): Author of Scala technical articles.
Publications	Shift schema drift left: Policy-aware compile-time contracts for typed JVM and Apache Spark pipelines. arXiv:2604.16986
Talks / Speaking	Linux Foundation Open Source Summit 2026: Session details (YouTube) Scala.io Paris, France October 2025: Compile-time data contracts & Fiber-safe Data pipelines: Scala’s Effect system in action OAISYS - 2025 AI PRACTITIONERS CONFERENCE, Pune India November 2025: LLM4S: Build Reliable AI Systems in the JVM Ecosystem Scala India, September 2025: When Data pipelines fail early: Type-safe Data Engineering in Scala DroidFest 2026 by Kotlin Kolkata: Type Classes in Kotlin - Without the Magic. Lambda days J on the Beach, Spain: Hands-on workshop: Rust ate Spark's lunch: DataFusion pipelines without the JVM. Data Makers Fest International Conference 2026: Organising team & volunteer. May 4–6, 2026. GDG & JUGs: Regular speaker at Google Developer Groups, Google Developer Expert and Java/Kotlin User Group meetups on Scala, Spark, and Data Architecture. Bootcamps: Run internal training on data engineering, functional programming for Fortune 500 teams. Publish 25K+ view technical articles on vitthalmirji.com covering type-safe data engineering, Scala patterns, and production lessons.
Education	B.E. Computer Science - Visvesvaraya Technological University, First Class, Jun 2015 (Belgaum, India). Diploma Computer Science - Bangalore University, First Class Distinction, 2010.
Certifications	Google Professional Data Engineer - Feb 2021. Cloudera Certified Spark & Hadoop Developer - May 2020. Sun Certified Java Developer - Java 5.x (2009), 6.x (2010).
Recognition	Walmart Bravo Award - 2024 & 2021. Walmart Wall of Fame - 2021-2024. Deloitte Outstanding Performance - 2019. Multiple Deloitte Applause Awards - 2017-2021. Best Academic Project (1st/15) - 2015, Gogte Institute of Technology.