🎯 Why this post exists

Inventory decisions are where strategy meets survival. Demand forecasting isn’t just a data science problem—it’s the heartbeat of retail, logistics, and supply chain optimization.

This post tells the story of how we built a modular, scalable Demand Forecasting Platform that uses Random Forest, Artificial Neural Networks, and Hybrid Time Series models—all running on top of a robust Data Lake infrastructure powered by Apache Hadoop and Spark.

🚚 Real-World Chaos: The problem statement

Forecasting demand for every SKU across regions, seasons, and supply timelines is a nightmare. Every error costs time, money, and customer trust.

We tackled this by building models to answer:

• Will this product sell next month?
• How much should we stock?
• Will this item become obsolete?

🧱 System architecture at a glance

graph TD
    A[Raw Data] --> B[HDFS Storage]
    B --> C[Preprocessing on Spark]
    C --> D[Model Training]
    D --> E[Forecasting Output]
    E --> F[Dashboards, Alerts]

🧹 Phase 1: Raw data to gold

📥 Data Ingestion

We collected:

• Historical sales (3 years)
• Stock movement logs
• Product metadata
• Customer classification
• Seasonal promotion flags

🧪 Exploratory Data Analysis (EDA)

Sanity checks and patterns:

1
2
3

df.isnull().sum()
df.describe()
df['category'].value_counts()

We applied:

• Pearson/Spearman correlation checks
• Distribution analysis
• Outlier treatment (IQR-based)
• Stationarity tests (ADF)

🧠 Phase 2: Forecasting strategy

We didn’t pick just one model. We created an ensemble strategy combining:

• Statistical models (ETS, ARIMA, Holt-Winters)
• Machine Learning (Random Forest)
• Deep Learning (Artificial Neural Networks)

Each approach had its domain:

• Time series models → seasonal & trend patterns
• ML classifiers → product obsolescence
• ANNs → nonlinear patterns and demand spikes

🔍 Hybrid forecasting logic

1

hybrid_forecast = (arima_forecast * 0.4) + (ann_output * 0.6)

The model pipeline:

• Lag features (7, 14, 28 days)
• Sliding windows
• Forecast horizon: 30 days
• Evaluation: MAPE, RMSE

🧬 Data Engineering backbone

1
2
3

$ spark-submit clean_data.py
$ spark-submit train_random_forest.py
$ spark-submit predict_ann.py

All models ran in batch mode orchestrated by Oozie, and outputs were stored in Hive for dashboards.

🧪 Random Forest for obsolescence

1
2
3
4

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, max_depth=12)
model.fit(X_train, y_train)

Key Features:

• Time since last sale
• Category demand index
• Stockout frequency
• Geographic demand

🤖 ANN for demand prediction

1
2
3
4
5
6

from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, input_dim=10, activation='relu'))
model.add(Dense(1, activation='linear'))

Tuned using:

• ReLU activation
• Adam optimizer
• 30 epochs

🧪 Time series models

ETS and ARIMA used via statsmodels.

1
2
3

from statsmodels.tsa.holtwinters import ExponentialSmoothing
model = ExponentialSmoothing(data, trend='add', seasonal='add', seasonal_periods=12)
fit = model.fit()

📊 Use Case: Product B - Engine Oil Additive

Scenario:

• 18-month sales history
• Sudden spikes near monsoon
• Southern region dominates demand

Model predicted:

📈 Dashboard Integration

All output forecasts were published into:

• Tableau for visual insight
• Superset for stakeholder self-serve access

Forecast deviation alerts were published via email & SMS using custom logic.

📂 Modular design

Every step in the pipeline was modular:

• Ingestion node
• Preprocessing node
• Forecasting node
• Alerting node

Can be independently replaced or improved.

graph LR
    Ingestion[Ingestion Node] --> Preprocessing[Preprocessing Node]
    Preprocessing --> Forecasting[Forecasting Node]
    Forecasting --> Alerting[Alerting Node]

🧮 Evaluation metrics

📎 Appendix: Sample forecast output (JSON)

1
2
3
4
5
6
7

{
  "product_id": "ENG-OIL-ADDT123",
  "forecast_next_30_days": 138,
  "model_used": "Hybrid (RF + ANN)",
  "confidence_interval": "±4%",
  "obsolete_risk": "Low"
}

🧠 Technical learnings

• Feature engineering matters more than model choice
• Combining time-series and ML gives best of both worlds
• Spark + Hadoop made scaling data preprocessing easy
• Modular pipeline = easier model updates + CI/CD

💼 Business wins

• 28% drop in overstock
• 35% better accuracy than old Excel-based systems
• Zero stockouts during peak sales week
• Data science + supply chain teams now speak the same language

💡 Final thoughts

You don’t need a PhD to build world-class forecasting.

You need:

• Clean data
• Smart features
• Modular design
• Monitoring & alerting

And maybe, a sarcastic blog post to explain it.

🛠️ Future plans

• Integrate Facebook Prophet & DeepAR
• GPU training for ANN
• Dockerized microservices for forecasts
• Push-button retraining with Airflow

🙌 Feedback?

Email: [email protected]