Cut Databricks compute cost by ~57% across a fleet of machine-log pipelines

Factories generate machine log files around the clock. Eleven legacy pipelines turned those logs into the tables engineers and analysts rely on. Each had been written differently, and each reprocessed whole tables on every run, so compute cost grew with the data and every deployment was a manual exercise.

• Standardized deployment with Databricks Asset Bundles and a Terraform-based pipeline, so every pipeline ships the same way.
• Converted legacy Python to PySpark and replaced full-table reloads with incremental processing (Delta change data feed with saved checkpoints), so each run touches only new data.
• Added parallel execution, job-specific Spark configuration and right-sized clusters, with logging and documentation.
• Built a follow-on AI-assisted workflow: four cooperating agents draft configuration, generate pipeline code, review it and orchestrate changes, while engineers approve every promotion.

• ~57% lower average daily compute cost across the eleven pipelines (first days after the final migration vs. a five-month baseline).
• 88% lower daily cost on the single most expensive pipeline in the fleet.
• A new pipeline delivered alongside, covering thousands of machines with billions of historical rows backfilled.