Cloud Data Platform

A data platform is only as reliable as the infrastructure underneath it. Where the data lives, who can reach it, how changes are rolled out and what operations cost are often decided early in a project and almost in passing. Fixing those decisions later takes real effort.

This page is about the technical foundation: compute and storage services, networking, identities, encryption, environments, backup and costs. How data is structured inside the platform is covered under data warehouse and data lakehouse, and rules on quality and ownership belong to data governance. A platform that has become more expensive or slower over time is covered under data platform optimization.

Cloud, your own data center, or both

The cloud offers scalability, pay-as-you-go billing and a broad range of services. Your own data center can make sense when data should not leave the site, existing hardware is still in use, or the source systems are on premises anyway.

Hybrid architectures combine the two. Sensitive or very large sources stay local, analytics and platform services run in the cloud, and a secured link transfers only what is needed. We decide per type of data rather than for the whole platform. What counts is data protection requirements, latency, data volumes, the skills in the team and how easily you could move again later.

Providers, and what sovereign can mean

When choosing a provider, existing systems, range of services, location and operator, pricing model and team skills all count. Besides the large US providers there are European clouds, for example from Germany and France. We advise without a vendor preference.

There are now dedicated offerings for location and control requirements. AWS describes its European Sovereign Cloud, with data centers in Germany, as separate from its other regions and operated by staff based in the EU. Microsoft groups under Sovereign Cloud a public variant in European regions, a private one for running in your own environment, and partner clouds in individual countries. Google also works with partners, in France with S3NS, for example.

What sovereign means in a given case differs a lot: location of the data, operator and jurisdiction, control over the keys, range of services. Certificates, too, often cover only part of an offering. STACKIT states, by its own account, that it holds BSI C5 and ISO 27001 among others, while for IONOS Cloud the C5 attestation covers individual services such as Compute Engine, Cloud Cubes and S3 Object Storage. So we don’t rely on the brochure. We check whether the services we need are actually available in the offering. Whether it meets the legal requirements is for legal counsel to assess.

Who gets to the data

Networking, identities and encryption decide who gets to the data. They are costly to change afterwards, so planning belongs at the start.

Data services shouldn’t be openly reachable from the internet. We use separate network segments, private connections to the services, via private endpoints for example, and secured links to your own site by VPN or dedicated line. People sign in through the company’s central identity system, with single sign-on and multi-factor authentication. Services and pipelines get identities of their own instead of shared passwords, and we grant permissions through groups, following the principle of least privilege. Shared accounts and long-lived keys are hard to trace and a frequent starting point for security incidents.

Data is encrypted at rest and in transit with most providers anyway. What needs settling is who controls the keys. Your own keys in a key management service or hardware security module give more control but come with tasks such as rotation and recovery. Passwords and access keys belong in a secret store and never in the code.

Environments and infrastructure as code

A platform needs at least three separate environments: development, test and production. They are separated through accounts, subscriptions or projects with their own permissions. Production data reaches development only for a good reason, and then pseudonymized or anonymized wherever possible.

The most common mistake is a platform someone clicked together in the portal. Nobody knows later how it is configured, and rebuilding it is hardly possible. With infrastructure as code, the environment lives in files that sit in a repository and are rolled out automatically. Changes are traceable, environments can be rebuilt identically, and manual interventions stand out.

With several clouds or European providers, we usually pick Terraform or OpenTofu. Terraform is widely used, with providers for the major clouds and many others, but it is now under HashiCorp’s Business Source License. OpenTofu is the open fork under the Linux Foundation and, by its own account, meant as a drop-in replacement. In Azure-only environments Bicep is often enough, Microsoft’s language for Azure alone, which compiles to ARM templates and leaves state management to Azure itself.

The infrastructure is treated like software. Changes go through pull requests, are checked automatically, rolled out to test first and to production after approval. The same pipeline logic applies to data pipelines, transformations and reports.

Backup, monitoring and costs

The configuration is in code, but the data needs a backup of its own: versioning, snapshots, copies in a second region. Retention and recovery time are defined. And recovery gets tested regularly, because otherwise you only learn in an emergency whether the backup is any good.

Services, pipeline runs, storage and access are monitored. Alerts go to people who are responsible, not to a mailbox nobody reads.

We assign costs from the start. Resources are tagged by project, environment and team, each environment has budgets and alerts, and development and test environments shut down outside working hours. That follows FinOps principles. Provider-specific services are often worth using, but they should be chosen deliberately, so you don’t tie yourself to one provider too early.