Docs menu
Datasets
The governed asset that every KPI, dashboard and answer is built on.
In short
A dataset is a table, view or file registered in Northstar, with its documentation, owner, quality rules and access rights. Every KPI, figure, dashboard and Chat answer reads datasets, and only the datasets you are allowed to see. A dataset belongs to a layer: bronze for raw data, silver for cleaned data, gold for data ready for the business.
Why everything starts from datasets
A number on a dashboard is only as reliable as the data behind it. Northstar therefore puts governance on the dataset itself, not on each report:
- Documented: description, column descriptions, owner and tags.
- Validated: business rules run at every analysis. See quality and validations.
- Monitored: a quality score, early warnings and your own alerts.
- Traced: where it comes from and what uses it. See data lineage.
- Governed: teams get access to it, action by action. See teams and resources.
Everything built on a dataset inherits this: a KPI shows the dataset it is computed from, and Chat only answers from datasets you can use.
Bronze, silver and gold
| Layer | What it holds | Typical origin |
|---|---|---|
| Bronze | Raw data, as loaded from the source | An ingestion |
| Silver | Cleaned, typed and joined data | A transformation from bronze |
| Gold | Data ready for the business: one row per customer, per month… | A transformation from bronze or silver |
A dataset without a layer is shown as Unclassified and is treated as bronze. The layer of an external dataset is set by its owner or an editor. The layer of a managed dataset is fixed by its transformation.
Managed or external
- Managed: a Northstar transformation builds and refreshes this dataset. Its card shows the Northstar badge. Its layer and SQL definition are changed through the transformation, in the SQL and Transformation tabs.
- External: the dataset was created outside Northstar and discovered in the warehouse. Northstar reads it but does not build it; its SQL definition, when there is one, is read-only.
How a dataset arrives
- An ingestion loads a source. Each loaded table becomes a bronze dataset. See sources and ingestion.
- A transformation builds it. Create dataset → Create with AI opens a session where the agent writes a silver or gold dataset from datasets you can see. See transformations.
- Someone registers an existing table or file. Create dataset → Browse my storage registers a table, a view or a tabular file that already exists.
- A data pipeline registers it through the API, with a developer key.
The dataset page
| Tab | What you find there |
|---|---|
| Schema | Columns, types and descriptions. |
| SQL | The query that defines the dataset, when it has one. Editable for a managed dataset. |
| Transformation | For a managed dataset: inputs, refresh, versions and runs. |
| Exploration | The rows, with filters and grouping. |
| Validations | The business rules and their results. |
| Quality | The quality score, its dimensions, early warnings and per-column statistics. |
| KPIs | The KPIs computed from this dataset. |
| My alerts | The alerts you created on this dataset. |
Each tab appears only if your team has the matching right, for example Explore for Exploration.
AI features by layer
Some AI features only make sense on curated data, so they depend on the layer:
| AI feature | Layers |
|---|---|
| Validation rule recommendations | Silver and gold |
| AI quality summary | Silver and gold |
| KPI recommendations | Gold |
| Chat | Gold first, then silver, then every dataset you can use |
The catalogue, the schema, profiling, the quality score and your alerts work on every layer.
Column descriptions
Good column descriptions make the catalogue, Chat answers and AI recommendations more accurate.
- You can write a description yourself, or use Generate metadata with AI for one column or for all missing ones.
- An AI description is marked AI suggestion until someone accepts it. You can edit it before accepting, or reject it.
- On a managed dataset, the AI relies on the column lineage: a column copied from a documented upstream column is described from it.
AI recommendations for validation rules and KPIs need at least 70% of the dataset and its columns to be documented.
Deprecated datasets
The owner, an editor or a platform administrator can mark a dataset as deprecated. It moves to the deprecated section and stays readable, clearly flagged. Chat no longer answers from it. Restoring it puts it back in the active catalogue.
FAQ
Why can't I change the layer of this dataset? It is managed: its layer is fixed by the transformation that builds it.
Why are there no AI recommendations on my dataset? Check its layer (see the table above) and its documentation: at least 70% of the dataset and its columns must be described.
Can I delete a dataset that a KPI uses? Check the KPIs tab and the lineage first: KPIs, figures and dashboards built on it would stop working.