All case studies

Case study 09 / 19

Automated Data Pipeline for Executive Power BI Dashboard

Containerized Airflow and Spark pipeline that extracts operational data, transforms metrics, loads a PostgreSQL data mart, and feeds executive Power BI dashboards.

Stack
Apache Airflow · Apache Spark · Docker · PostgreSQL · MSSQL · Power BI · SQL
Sanitized executive energy performance dashboard showing operational, risk, KPI, and financial metrics

The problem

Executive performance reporting depended on manual data preparation across multiple operational source systems. This made dashboard refreshes slower, harder to rerun safely, and less consistent across reporting periods.

What I built

Built a containerized data pipeline for operational performance dashboards using Apache Airflow, Apache Spark, Docker, PostgreSQL, and MSSQL. The workflow automates extraction from multiple source systems, transforms operational metrics, validates and aggregates business logic, and loads dashboard-ready datasets into a centralized PostgreSQL data mart for Power BI.

Architecture

Airflow schedules two runs a day. Spark extracts from three operational databases, transforms and validates 13 metrics across seven domains, and loads a PostgreSQL data mart with month snapshots, so any run can be repeated or backfilled safely. Power BI reads only the mart.

Data pipeline architecture: three operational databases feed Spark ETL jobs scheduled by Airflow twice a day, loading a PostgreSQL data mart of 13 metrics that Power BI reads

Open the image for full size

Proof

  1. 01 Designed the pipeline architecture, Airflow DAG scheduling, Spark ETL logic, SQL transformations, data validation, and rerun-safe refresh flow.
  2. 02 Pipeline supports scheduled refreshes, monthly updates, metric backfill handling, and centralized dashboard-ready tables.
  3. 03 Sanitized dashboard output shows executive-style performance views across operations, asset integrity, safety, risk, KPI, and financial metrics.

Result

Reduced manual data preparation and improved the reliability, freshness, and consistency of dashboard reporting, giving executives a cleaner view of performance across reporting periods.