Apache Spark
Introduction
Apache Spark is the de facto standard for large-scale data processing, offering unified analytics for batch processing, streaming, machine learning, and graph processing. This hub covers practical Spark development, optimization techniques, and real-world deployment patterns.
Core components
Spark Core & RDDs
- Resilient Distributed Datasets: Fundamental data abstraction
- Transformations & actions: Lazy evaluation and computation
- Caching & persistence: Optimizing iterative algorithms
- Fault tolerance: Understanding Spark’s recovery mechanisms
DataFrames & Datasets
- Structured APIs: Working with DataFrames and Datasets
- Catalyst Optimizer: Understanding query optimization
- Schema evolution: Handling changing data structures
- Type safety: Benefits of strongly-typed Datasets
Spark SQL
- SQL analytics: Querying big data with familiar SQL
- Window functions: Advanced analytical queries
- User-defined functions: Extending Spark SQL capabilities
- Integration: Connecting to external data sources
Streaming
- Structured Streaming: Real-time data processing
- Watermarking: Handling late-arriving data
- Output modes: Append, complete, and update semantics
- Kafka integration: Building streaming data pipelines
MLlib
- Machine learning: Scalable ML algorithms
- Pipelines: Building reproducible ML workflows
- Feature engineering: Data preparation at scale
- Model serving: Deploying models in production
Featured articles
Below are the latest articles tagged with “spark”:
DQ is not Dairy Queen: Building a data quality framework (DQF + DPAT) in prod
Complete guide to building a data quality framework. Learn constraint validation in Scala, YAML checks (DPAT), Spark/Airflow/CI/CD integration, and automated failure reporting. Includes real code and architecture to catch issues early.
Functional utilities for data engineering: stop repeats in Scala pipelines now
Build reusable Scala utilities for data engineering on GCP, covering CDC, Delta maintenance, schema evolution, GCS helpers, affected-partition strategies, and error handling. Includes patterns for safe ops and repeatable pipelines.
How I cut delivery errors by 82%: DPAT, gated CRQ, and data contracts in prod
Production-tested strategy to cut pipeline failures from 12% to 2% using YAML-based data quality checks (DPAT), CI/CD gates, and strict data contracts. Includes before/after metrics and architecture for self-healing pipelines in prod.
Forecasting at scale: demand prediction with Random Forests and neural nets
Building a demand forecasting system with Random Forest, ANN, and hybrid time-series models on Hadoop and Spark. Covers data ingestion, feature engineering, model training, validation, and rollout for large SKU catalogs and seasonality.
Is Hadoop dead? Cloudera-Hortonworks, MapR layoffs, and Hadoop 3.0 reality now
Deep-dive into Hadoop's state in 2019: Cloudera+Hortonworks merger, MapR struggles, and what Hadoop 3.0 delivers (GPU scheduling, Docker, Hive ACID). Learn why AWS/GCP/Azure dominate but Hadoop is evolving into hybrid cloud with Spark.
Data lakes: Hive is not an RDBMS, HBase 141x faster, Spark's role in practice
Deep-dive performance comparison of Hive, HBase, and Spark on 9.6M NYC taxi records. Learn why Hive ACID falls short, when HBase is 141x faster for lookups, and what Spark actually optimizes for. Includes benchmarks and usage guidance.
Performance & optimization
Resource management
- Cluster configuration: Tuning executor and driver settings
- Memory management: Understanding Spark memory model
- Dynamic allocation: Scaling resources based on workload
- Resource isolation: Multi-tenant Spark deployments
Query optimization
- Join strategies: Broadcast vs shuffle joins
- Partitioning: Data layout for optimal performance
- Predicate pushdown: Minimizing data movement
- Columnar storage: Working with Parquet and Delta Lake
Monitoring & debugging
- Spark UI: Understanding execution plans and metrics
- Logging: Effective debugging strategies
- Performance profiling: Identifying bottlenecks
- Cost optimization: Reducing compute and storage costs
Deployment patterns
Cloud platforms
- Google Cloud Dataproc: Managed Spark on GCP
- AWS EMR: Elastic MapReduce for Spark workloads
- Azure HDInsight: Spark on Microsoft Azure
- Kubernetes: Container-native Spark deployments
Data formats & storage
- Parquet: Columnar storage for analytics
- Delta Lake: ACID transactions for data lakes
- Hudi: Incremental data processing
- Iceberg: Open table format for analytics
Integration patterns
- Data warehouses: Connecting to BigQuery, Snowflake
- Streaming sources: Kafka, Kinesis, Pub/Sub
- Orchestration: Airflow, Argo, and workflow management
- CI/CD: Testing and deploying Spark applications
Related topics
- Data engineering - Building data pipelines with Spark
- Scala - Functional programming for Spark development
- Google Cloud Platform - Running Spark on GCP
- Open source - Contributing to Spark ecosystem
Working with Apache Spark? Connect with me or explore the articles above for optimization tips and deployment strategies.