Plant telemetry arrived as files from line controllers and ERP extracts, each site with its own scripts and its own definition of downtime. We built one lakehouse on Databricks: raw data landing as it arrives, refined by declarative pipelines that enforce quality rules, governed once in Unity Catalog, and read by every plant through the same measures.
Each plant ran its own extracts and notebooks. Line telemetry sat in files nobody reconciled; ERP data was joined to it by hand in spreadsheets. Downtime meant something slightly different at every site, so group reporting was weeks of reconciliation and numbers nobody quite trusted.
Telemetry and ERP land continuously into bronze Delta tables, are refined into silver and gold by declarative pipelines with quality expectations, governed once in Unity Catalog, and served through a serverless Databricks SQL warehouse. Every plant reads the same tables and the same measures.
Controller files and incremental ERP tables meet in Bronze. Validated Silver tables feed Gold plant measures; an explicit quality branch retains invalid records separately.
Silver persists the records accepted by the pipeline’s shared quality rules. Lakeflow Declarative Pipelines applies expectations while producing these Delta tables; an explicitly authored branch retains invalid Bronze records in quarantine.
Bronze, Silver and Gold are persisted Delta layers governed by Unity Catalog. Pipeline expectations define the accepted-record path and an explicitly authored quarantine branch; Lakeflow Jobs coordinates scheduled work.
Auto Loader picks up line controller files as they arrive; Lakeflow Connect pulls the ERP tables incrementally. Everything lands in bronze Delta tables with its source, its file and its arrival time. Nothing is transformed on the way in, so any silver rule can be replayed from bronze.
Lakeflow Declarative Pipelines builds the silver and gold layers as streaming tables and materialized views defined in SQL. Expectations validate the data; an explicitly authored quarantine branch retains invalid rows outside the reporting path. Downtime, the OEE inputs and the shift calendars are defined once, in code, and reviewed like code.
One metastore, catalogs per environment, schemas per domain. Row filters keep each plant to its own lines, column masks cover supplier prices, lineage shows where a gold number came from, and the system tables show what each pipeline costs.
A serverless Databricks SQL warehouse serves Power BI over the gold tables, and the plant dashboards read one semantic model. Liquid clustering on the largest telemetry tables keeps line-level queries fast without a maintenance calendar.
Lakeflow Jobs schedule what is not streaming. Databricks Asset Bundles carry pipelines, jobs and permissions from development to production through GitHub Actions, and Lakehouse Monitoring watches freshness and volume per table and alerts on drift.
The rules live in the pipeline code, reviewed and versioned; every plant reads the same gold tables, so group reporting is a query, not a reconciliation.
Streaming tables refresh continuously; a line stop shows in the dashboard within the pipeline’s interval, not the next morning.
Expectations catch malformed readings and duplicate ERP extracts; the run completes and the quarantine is reviewed.
Row filters and column masks govern access to the lakehouse. The Power BI serving path must carry the appropriate user identity or enforce its own report access rules.
Pipelines, jobs and permissions are deployed from Git; nothing is edited in production by hand.