Project Case Study

Automated Data Pipeline for Executive Power BI Dashboard

Containerized Airflow and Spark pipeline that extracts operational data, transforms metrics, loads a PostgreSQL data mart, and feeds executive Power BI dashboards.

Apache Airflow Apache Spark Docker PostgreSQL MSSQL Power BI SQL

Automated Data Pipeline for Executive Power BI Dashboard

Automated data pipeline architecture from source systems through Airflow, Spark, PostgreSQL data mart, and Power BI reporting

Problem

Executive performance reporting depended on manual data preparation across multiple operational source systems. This made dashboard refreshes slower, harder to rerun safely, and less consistent across reporting periods.

Solution

Built a containerized data pipeline for operational performance dashboards using Apache Airflow, Apache Spark, Docker, PostgreSQL, and MSSQL. The workflow automates extraction from multiple source systems, transforms operational metrics, validates and aggregates business logic, and loads dashboard-ready datasets into a centralized PostgreSQL data mart for Power BI.

Proof

Designed the pipeline architecture, Airflow DAG scheduling, Spark ETL logic, SQL transformations, data validation, and rerun-safe refresh flow.
Pipeline supports scheduled refreshes, monthly updates, metric backfill handling, and centralized dashboard-ready tables.
Sanitized dashboard output shows executive-style performance views across operations, asset integrity, safety, risk, KPI, and financial metrics.

Result

Reduced manual data preparation and improved the reliability, freshness, and consistency of dashboard reporting, giving executives a cleaner view of performance across reporting periods.