This reference architecture brings ERP orders in Azure SQL, CRM records from a REST API and partner shipment files into one governed Azure Databricks design. Source contracts, Bronze–Silver–Gold processing and independent quality gates connect the data. Bundles and GitHub Actions define how reviewed code would reach development, staging and production.
An ERP database, a paginated CRM API and partner file deliveries expose different update, deletion and completion rules. A shared reporting model would need to reconcile those contracts and make each refresh and code change traceable.
Complete source batches would feed Bronze and Silver, with an independent quality gate before Gold. Reviewed code and resource definitions would move through isolated environments, while deployment, runtime and storage access would use deliberately assigned identities.
SQL snapshots, paginated REST responses and partner files land in ADLS before Auto Loader retains them in Bronze. Triggered Silver processing and a separate quality gate precede Gold reporting.
An independent task verifies that every required source and Silver table belongs to the expected completed batch, then checks freshness, key uniqueness, row counts and agreed reconciliation totals. It fails the job dependency when configured checks fail. Gold starts only after both the Silver update and this task succeed, so one successful Silver table cannot authorize publication on its own.
Reference architecture for Azure Databricks. Landing files and persisted Bronze, Silver and Gold tables are separate stages. Lakeflow Jobs requires successful Silver processing and the independent quality gate before starting Gold; a failing table does not atomically roll back other tables.
The proposal would extract complete Azure SQL snapshots through Spark JDBC into Parquet. A custom Python task would handle the CRM API’s pagination, rate limits, retries and documented cursor rules, landing JSON. Partner CSV deliveries would arrive in a separate ADLS path. Each source contract would define keys, schema, deletes and batch completion. Manifests and reconciliation counts would distinguish a complete extract from a partial delivery; source consistency requirements would determine the SQL extraction window.
Lakeflow Jobs would schedule Auto Loader with AvailableNow to ingest completed file deliveries into Bronze Delta tables. Each invocation would process files available before it starts, using persistent checkpoints and schema locations specific to each source and environment. Immutable landing files and source, file, batch and ingestion metadata would support investigation and replay. The API extractor would advance its cursor only after durable landing. Auto Loader’s checkpointed Delta ingestion guarantees would apply to files; business duplicates would require their own rules.
A triggered Lakeflow pipeline would standardize types, timestamps and business keys, select completed snapshots and reconcile source identifiers. Expectations would enforce row rules, with an explicit quarantine branch retaining invalid records. A separate job task would check source completion, freshness, uniqueness, relationships and agreed reconciliation tolerances before allowing the Gold pipeline to run. This separation matters: a failing expectation can roll back its table update while other parallel flows succeed. The quality gate would control downstream execution across those flows.
The proposed Gold model would define order and invoice facts alongside conformed customer, product and date dimensions, with shipment facts at their own grain. Revenue, backlog and delivery measures would have documented definitions and reconciliation checks. Databricks SQL would serve the approved tables to a Power BI semantic model after the quality gate succeeds. Access and report refresh would be configured for that serving path. A batch identifier and freshness record would help readers understand which completed source deliveries each result represents.
Declarative Automation Bundles, formerly Databricks Asset Bundles, would version the Python and SQL code, job and pipeline definitions, permissions and target settings together. Dev, stage and prod would resolve to separate workspaces, catalogs, landing paths and checkpoints. Stage and prod would use stable deployment roots; stage schedules would remain explicitly paused. The same reviewed commit and versioned artifact would be promoted. Bundles track deployed resources by workspace IDs and state, so changing bundle roots or adopting existing resources would require explicit state management.
GitHub Actions would run linting, unit tests and source-contract fixtures on pull requests. Trusted integration stages would validate the bundle configuration, deploy to the selected test environment, then explicitly run jobs against representative fixtures. Tests would cover partial deliveries, schema changes, duplicate records and replay, followed by Silver-to-Gold reconciliation. Bundle validation checks configuration; deployment creates or updates resources; neither proves data correctness. Production promotion would require successful stage results and its configured approval, followed by an explicit smoke check.
GitHub would authenticate through workload identity federation into a deployment service principal for each environment. The production policy would match the repository’s actual OIDC subject and audience, with branch restrictions and approval on the protected GitHub environment. Target run_as settings would assign jobs and pipelines a separate runtime principal, and the deployer would receive permission to use it. GitHub token claims and approval availability would be verified for the repository and plan. This design would remove long-lived Databricks deployment secrets.
Unity Catalog would govern the environment catalogs, tables and landing volumes. Catalog and external-location bindings would restrict production access to its intended workspace; workspace separation alone would not establish that boundary. An Azure Databricks Access Connector with a managed identity would provide ADLS access through a storage credential. Runtime principals would receive the required data, compute and secret permissions. API and database credentials would be referenced through governed connections or permissioned secret scopes, with values excluded from code and routine logs.
The proposed job would record source batches, extraction cursors, task outcomes, quality results and reporting freshness, with alerts on failures and late deliveries. Checkpoints would be protected from storage cleanup, and retention would preserve the agreed replay window. Retries would reuse batch identities and be tested for duplicate effects. An operations guide would distinguish restarting a failed task, replaying retained data and redeploying an earlier code version. Data or schema recovery would require its own reviewed procedure; code rollback would not restore either automatically.
Keys, delivery manifests and batch records would make incomplete extracts visible before they feed reporting.
Recorded row checks and cross-source reconciliation would determine whether the reporting refresh can proceed.
Reviewed code, versioned resources and executed stage tests would accompany the production release.
Assigned identities, environment boundaries and replay procedures would make the operating responsibilities concrete.
Official platform documentation used for this reference design. Reviewed 6 September 2026.