Data Governance & Data Quality

When numbers differ between reports, trust in data erodes quickly. On top of that come data protection and security requirements that grow with every new source. In many companies the rules exist on paper, but nobody checks them in operation.

The data strategy sets where a company wants to go with its data. Data governance makes sure the data is up to the job day to day, and that operational part is what we implement technically. Governance should make daily work easier, so we keep the effort small and start where an error costs the most.

First: who is responsible?

Without owners, the rest stays on paper. Two roles matter most in daily work. The data owner is responsible for a data area from the business side, customer or finance data for example, and decides on definitions, quality requirements and access. The data steward maintains descriptions and master data day to day, handles quality alerts and is the contact person. How such roles are found and distributed across the organization is part of the data strategy.

Both roles only work with time and a mandate. An entry in the catalog without real ownership changes nothing. Alerts from quality checks therefore go straight to the named person. Metrics such as revenue or active customer are defined in data modeling. Governance makes sure each definition has an owner and can be found.

Measure quality instead of asserting it

Data quality can only be improved if it is measured. With the business teams we set rules and check them automatically on every load. Are values missing that should be there? Are keys and customer numbers unique? Do values match the expected format, and are relationships between tables intact? Did the data arrive on time, and are totals, quantities and trends within the expected range?

This is implemented with what is usually in use anyway: dbt tests or a testing framework, and some platforms and catalogs have quality features of their own. If a check fails, the responsible person is notified, and depending on severity the data is blocked or passed on with a flag.

It is important not to check too much at once. Anyone who gets dozens of irrelevant alerts a day soon reads none of them. It is better to start with the rules that prevent the most damage.

Catalog and lineage

A data catalog lists tables, reports and metrics with description, owners, classification and origin. It answers where which data lives and what it means. Lineage adds the path of the data from the source through transformations to the report. That lets you estimate what a change affects and narrow down the origin of errors.

There are plenty of products, commercial and open source alike. For a start, the catalog your own platform already includes is usually enough. A cross-platform tool pays off more with several platforms. What matters is less the product than upkeep: outdated descriptions are worse than none because they give false assurance. Here too, it is better to describe a few areas with high value thoroughly than to touch all the data at once and finish nowhere.

Access, pseudonymization, deletion

We grant access by role and by the principle of least privilege, not to individuals. Rights for individuals are hard to oversee and rarely adjusted when the team changes. Where needed, restrictions apply at row and column level, for regions or salary data, for example. Sensitive data is classified, as personal or confidential, say, and the classification drives rules such as masking or restricted access. Access is logged. When approvals take too long, teams turn to solutions of their own, and self-service with clear guardrails works better than a request process.

For personal data, two more points apply. Pseudonymization replaces direct identifiers such as names with keys, so analyses are possible without a direct link to a person. Pseudonymized data still counts as personal data. Deletion concepts define when which data is deleted or anonymized. Technically that is demanding, because data can sit in copies, history tables, backups and derived tables. If you delete only at the source, the rest stays. So we plan up front where deletions have to take effect. What is legally required is for legal counsel to assess, and we implement it technically.

Data contracts and data for AI

A data contract is an agreement between the side that supplies data and the side that uses it. It describes structure, quality requirements, freshness and contacts. With a contract, a structural change at the source gets noticed before it breaks reports. There are open standards for this, such as the Open Data Contract Standard, which describes contracts as YAML files, and contract checks in tools such as dbt. The files live in the repository and are checked automatically in the pipeline. The idea fits data products, where business teams provide data with clear ownership.

The same principles apply to AI. Assistants and agents should only see data that the requesting person is entitled to. Descriptions, quality and origin of the data determine how good the answers are. The legal requirements for AI systems are set by the EU AI Act, which we cover under the topic of the same name. Some catalog tools already expose descriptions, lineage and permissions to assistants through interfaces such as the Model Context Protocol. That is one more reason to keep metadata in good order.