Docs menu
Data lineage
Where each dataset and column comes from, and what depends on it.
In short
Lineage shows where your data comes from and what depends on it: from the sources to the gold datasets, and on to the KPIs, figures and dashboards that use them. For datasets built by Northstar, it goes down to the column: you can see how each column is computed. Use it before changing or deleting something, to know what would be affected.
Reading the lineage page
Pick a source or a dataset to start from, or search for a column. The page lays out the chain in five lanes:
Sources → Bronze → Silver → Gold → Usages
- Usages are the KPIs, figures and dashboards built on a dataset.
- Upstream and Downstream set how far the chain extends on each side. Expand a node to show what feeds it or what uses it.
- Select a node or a link to see its details.
The links have four styles:
| Link | Meaning |
|---|---|
| Built or fed | A source feeds a dataset, or a transformation builds a dataset from another one. Set by Northstar. |
| Declared by hand | A link an owner or editor declared between two datasets. |
| Used by | A KPI, figure or dashboard reads the dataset. |
| Column to column | A column is computed from another column. |
You only see what you are allowed to see. A dataset you cannot access appears as such, without its details.
Automatic and declared links
- Automatic links come from what Northstar runs: the tables an ingestion loads, and the datasets a transformation reads. They are updated at every build, and they drive the refresh chain: when an input changes, what is built on it is rebuilt.
- Declared links document a dependency Northstar did not build, for example a table produced by an external pipeline. The owner, an editor or a platform administrator adds them from the dataset page, with Link a dataset. They appear in the lineage but never trigger a refresh.
On a dataset page, Derived from and Used by list the datasets on each side.
Where a column comes from
For a dataset built by a Northstar transformation, each column shows how it is built. Open it from the schema to see Column origin, down to the data sources:
| Kind | Meaning |
|---|---|
| Copy | Copied as is from an upstream column, under the same name |
| Renamed | Copied as is, under another name |
| Computed | Computed by an expression over upstream columns |
| Aggregated | A sum, a count, an average or a window function over upstream columns |
| Constant | A fixed value, with no upstream column |
| Unknown | The query could not be analysed for this column |
For an external dataset, Northstar does not know the query that built it, so the chain stops at the dataset.
Before you change something
Lineage answers "what breaks if this changes?":
- Before editing a managed dataset, the preview of the SQL tab lists the datasets that will be rebuilt and flags a removed column that another dataset uses.
- Before deleting a dataset, look at its downstream lane: KPIs, figures and dashboards built on it would stop working. Northstar refuses to delete a dataset that a transformation still reads.
- When a source changes, follow it downstream to see every dataset, KPI and dashboard it feeds.
FAQ
Why don't I see the origin of a column? Column origin exists for datasets built by a Northstar transformation. For a new one, it is computed at its next build.
Why does a node say "A dataset you cannot access"? The chain goes through a dataset that is not shared with your teams. Ask its owner or an administrator.
Does a declared link refresh anything? No. It documents the dependency; only links built by Northstar trigger refreshes.