Databricks Optimization Techniques That Actually Move the Needle

Fix the files first

Most slow Delta tables are suffering from small files. Thousands of tiny files mean thousands of tasks and a lot of metadata overhead. OPTIMIZE compacts them, and liquid clustering (or Z-ORDER on older runtimes) co-locates data so queries skip files they do not need.

• Target file sizes in the low hundreds of MB for analytical tables
• Cluster by the columns used most in filters and joins
• Let predictive optimisation handle maintenance where available

Process incrementally

The cheapest data to process is data you have already processed. Auto Loader and checkpointed streams with availableNow triggers pick up only new files and then shut down. For gold tables, INSERT … REPLACE WHERE rebuilds just the affected date range instead of the whole table.

Right-size compute

Use job clusters rather than all-purpose clusters for scheduled work, enforce cluster policies, and read the Spark UI before adding nodes. Skewed joins and unnecessary shuffles are fixed by changing code, not by buying more workers.