Research · Data for AI

Data Governance & Lineage for AI

Governance defines who owns data, what it means and how it may be used. Lineage traces where data came from and how it changed. For AI, both are needed to explain outputs, meet obligations and fix problems.

The AI Data Accountability Stack

Governance and lineage for AI must answer questions that traditional data programs rarely faced: which data shaped this model or this answer, and can we prove a deleted record is gone from everywhere it was copied? The AI Data Accountability Stack is a CorpExcellence framework that organizes the answer into five layers. Each layer depends on the one beneath it, so a gap near the bottom limits what the top can prove to an auditor or an incident team.

Foundation: catalog, owners and versions

Catalog and owners extends the data catalog beyond tables to the document libraries, tickets, transcripts and knowledge bases that feed retrieval and agents. Each asset gets an owner, a classification and a list of permitted uses. Dataset versions captures fixed snapshots of training, evaluation and retrieval sets, so a team can reproduce what a model saw on a given date and apply retention rules to each snapshot rather than to a moving target. Without versions, the honest answer to the question of what a model learned from is usually a guess.

Middle: lineage events

Lineage events record, for every run, which job read which inputs and wrote which outputs. Collecting these events automatically from pipelines, orchestrators and embedding jobs is far more reliable than hand-drawn diagrams, which go stale as soon as a pipeline changes. Open standards make this practical across tools. OpenLineage, a graduated LF AI & Data Foundation project, defines a common model of jobs, runs and datasets with extensible facets that many platforms can emit and consume.

Top: AI system records and use approval

AI system records link each model, prompt template, retrieval index and agent configuration to the dataset versions and lineage behind it. Use approval records which purposes are approved for each data asset, who approved them and under what conditions. With both layers in place, a request such as “show every AI system that used last year’s customer complaints” becomes a query rather than a project.

Lineage, metadata and documentation that AI actually needs

AI-Ready Data sets out the metadata that matters most for AI, which dataset documentation to require and when, and a minimum metadata standard you can adopt.

Get the full report →

Where the industry is now

As of September 2026, Article 10 of the EU AI Act requires providers of high-risk systems to apply data governance practices covering data origin, the original purpose of personal data, and preparation steps such as labeling and cleaning. The AI Omnibus, Regulation (EU) 2026/1744, moved the main high-risk deadline to December 2, 2027, which gives teams time to build lineage now instead of reconstructing it for an audit later. Agentic AI adds pressure: an agent can read and write across several systems in one task, and lineage needs to capture those actions alongside batch pipelines.

Common pitfalls

• Cataloging structured data while retrieval systems index unmanaged documents.
• Tracking lineage to the warehouse and stopping before embeddings and vector indexes.
• Deleting a source record without tracing its copies in training sets, indexes and caches.
• Treating approval as a one-time gate when the use of the data later changes.