Data Platform Optimization

Data platforms grow with every new source, every report and every team. At some point the bill rises and nobody can explain which applications cause it. Load runs take longer every month, reports load slowly, and for some tables nobody knows whether they are still needed. Nobody decided this. It is the sum of many decisions that were each reasonable.

We build on the platform you have. First we measure where consumption and runtime come from, then we remove the biggest causes one after another. Rebuilding a platform is covered under cloud data platform, and the rules for quality and access under data governance. This page is about costs, runtimes and overview.

Measure first, then change

It starts with an analysis of actual usage. All common platforms provide data for this, such as billing and usage views, system tables and utilization reports. We assign consumption by application, team, environment and purpose and ask: which jobs and queries cause the largest share? What runs with no visible benefit, at night or on weekends? Where does consumption grow faster than the data?

It usually concentrates in a few places, and priorities almost set themselves. Assigning it requires consistent tagging of resources, for example tags by team and project. If that is missing, we set it up first.

If you make changes without baseline values, you won’t know afterwards whether they helped. That applies in particular to the size of resources: a smaller instance saves money but can lengthen runtimes, and a larger one isn’t always faster. We test both.

What can be tuned on which platform

The levers differ by product. Here are the four we work with most often.

Snowflake

Warehouse size should match the task. According to the documentation, larger warehouses mainly help with complex queries, not with high concurrency. Additional or multi-cluster warehouses suit that case. A short auto-suspend avoids idle costs, though the minimum billing of 60 seconds per start has to be kept in mind. Resource monitors and budgets cap consumption.

Microsoft Fabric

Performance is measured in capacity units. Fabric smooths load peaks over time and only throttles when a capacity is overloaded. The Capacity Metrics app shows whether and when that happens. Depending on the finding, a larger capacity helps, or a split across several capacities, or tuning individual models and load runs.

Databricks

Serverless compute for notebooks, jobs and pipelines shortens start-up times and takes operational work off your plate. Whether it is cheaper depends on the workload, so we test with a representative one. The billing system table provides consumption data, tags assign costs, and compute policies limit the type and size of clusters.

Google BigQuery

Billing is by data processed or by reserved capacity (slots) in the Standard, Enterprise and Enterprise Plus editions, also with autoscaling. Which model fits depends on the query profile.

The same principle holds on other platforms: measure cost and runtime before the change, then check afterwards whether it worked.

Queries, load runs and storage

Expensive queries can often be simplified. Typical starting points are reading only the columns and rows you need, partitioning and clustering, caching costly intermediate results and dropping unnecessary joins. In the BI tool, aggregations and a lean semantic model help.

For load runs, incremental loading brings the most. Instead of reloading everything every day, only new or changed data is processed, for example through incremental dbt models or change data capture. We also check the order and start times of jobs, because jobs running at the same time compete for resources.

With storage, it comes down to retention. Rarely used data can move to cheaper storage classes. Retention periods for snapshots, time travel and raw data should be set deliberately, since every copy costs money. Deletion periods for personal data we clear with data protection.

What can go

Many savings come from leaving things out: tables that are never queried, pipelines without a consumer, reports nobody opens, duplicate copies of data. Nothing is switched off on suspicion. A report nobody seems to know can be needed for the annual closing. We ask the people involved and at first only deactivate objects instead of deleting them.

Then there is structure. Data layers, naming conventions and permissions are standardized, rights are granted by role rather than to individuals, and accounts of former employees disappear.

So it doesn’t get expensive again

A one-off exercise helps little. Without budgets, alerts and named owners, costs soon rise again. FinOps describes how engineering, business teams and finance handle cloud costs together. The FinOps Framework of the FinOps Foundation names these principles, among others: teams collaborate, each team takes ownership of its usage, cost data is accessible, timely and accurate, and technology decisions are driven by business value. It explicitly lists data platforms as an area of application.

In practice we keep this deliberately lean. Consumption is made visible and assigned, then measures are carried out, then someone regularly checks whether budgets and expectations are met. The framework calls these phases inform, optimize and operate. A monthly look at the biggest items, with named owners, usually achieves more than elaborate reporting. And whoever causes consumption should be able to see it, otherwise there is no incentive to save. Savings that worsen data quality or freshness come back later as extra work.

We show the team how to watch costs and runtimes in daily work, document the changes and train people in handling budgets and alerts.