A parcel carrier knew where a parcel had been the next morning, when the overnight load ran. We built the flow that knows where it is now: scanner and vehicle events read from Kafka by Structured Streaming into bronze, refined by declarative streaming tables that bound event-time state and handle late scans explicitly, joined statefully into one parcel status, and served both to the control room and to the customer tracking API while the parcel is still moving.
Scanner events and vehicle telemetry went to Kafka and from there to files, loaded overnight into a warehouse. The control room saw yesterday; the tracking API read a table refreshed a few times a day; an exception was found when a customer called.
Structured Streaming lands every topic in bronze with checkpointed Delta writes; declarative streaming tables parse, validate and join the scans to the vehicle positions within watermarks; a gold parcel status refreshes continuously; Databricks SQL serves the control room and a Lakebase synced table serves the tracking API’s point lookups.
Kafka events land in Bronze and are validated into Silver. Stateful joins produce Gold parcel status for SQL analysis and a separate Lakebase tracking path.
Structured Streaming joins validated Silver scans to vehicle positions within a time window, then aggregates the current status per parcel. Watermarks bound retained event-time state according to the configured rules; the resulting state updates the Gold parcel-status table.
The three persisted Delta layers remain separate from stateful processing. Watermarks bound event-time state; retained Bronze events support an explicitly authored late-event review path.
Structured Streaming reads each topic into its own bronze Delta table with the raw payload, key, partition, offset and ingest time. Checkpoints and Delta’s transactional commits support exactly-once recovery; event-key deduplication handles repeated business events separately. The raw payload retains fields that have not yet been promoted into the parsed schema.
Silver is Lakeflow Declarative Pipelines streaming tables: payloads parsed and validated with expectations, duplicates dropped on the event key, and a watermark that bounds the event-time state retained for processing. Raw events remain in bronze; explicit late-event handling flags scans outside the state horizon for review.
Scans are joined to vehicle positions in a stream-stream join within a time window, and a stateful aggregation keeps the current status per parcel: last scan, last known position, next expected event and its deadline. The gold table is the answer to “where is it”, kept by the pipeline rather than recomputed by every reader.
The control room reads the gold table through a serverless Databricks SQL warehouse, with SQL alerts on parcels past their expected event. The tracking API reads a Lakebase table synced from the same gold table, so a customer’s lookup is a Postgres point read and never touches the warehouse.
Lakehouse Monitoring watches freshness and volume per table and alerts when a topic goes quiet or a watermark starts closing too much state. Pipelines and their compute are declared in Asset Bundles and promoted through Git; a schema change is a reviewed pull request, not a hotfix at three in the morning.
Events reach the gold table within the pipeline’s interval, so the control room and the tracking API describe now rather than yesterday.
Checkpointed recovery and transactional Delta writes avoid duplicating committed output; event-key deduplication separately handles repeated scans.
Watermarks bound event-time state. Raw scans remain in bronze, and a separate late-event path flags records that need review.
SQL alerts fire on parcels past their expected event; the control room acts before the customer calls.
The tracking API reads the Lakebase synced table; the analysts’ queries and the customers’ lookups no longer compete.