From MBTA data to validated Gold tables.
A complete Databricks implementation for transit analytics, from API ingestion to Gold tables.
An end-to-end Databricks demonstration using Massachusetts Bay Transportation Authority (MBTA) transit data from the Boston area. API ingestion, Lakeflow pipelines, Spark SQL medallion transformations, quality checks, and scheduled AI/BI dashboard runs.
Unity Catalog · validation · AI/BI
The pipeline, at a glance.
A simplified source-to-Gold view. The real Lakeflow dependency graph is available in the next section.
Historical stop events and live vehicle snapshots.
Parquet / JSON and source metadata in a Volume.
Incremental transforms and validated records.
Dimensions, facts, curated marts and dashboards.
Databricks skills, with proof.
One viewer, 11 real workspace screenshots. Select a skill to explore the implementation.
Lakeflow pipelines
ETL dependencies and completed runs
A managed pipeline connects source tables to Silver and Gold outputs; its graph shows 12 assets in the completed run.
metropulse_02_bronze_to_silver_testAll images are from the actual Databricks workspace. Use Enlarge for readable SQL, settings and run details.
Validated data outputs.
Two checks demonstrated in SQL. The result is reproducible from the workspace notebooks.
Silver → Gold
Fact ↔ mart
PASS here verifies fact-to-mart row-count reconciliation; it does not imply all possible data-quality checks passed.