Docs menu

Transformations

How silver and gold datasets are built from raw data, then kept up to date.

In short

A transformation builds a silver or gold dataset from other datasets, in your warehouse. The AI agent writes the query; you review it and apply it; Northstar creates the view or table and keeps it up to date. Nothing is built until you apply it, and a built dataset is never visible to more teams than its inputs.

From a draft to a built dataset

Transformations are prepared in a session, listed under In preparation on the Datasets page. A session holds up to 8 datasets.

You can open a session in three ways:

Create dataset → Create with AI on the Datasets page;
from the proposals posted in a source's conversation after a successful ingestion;
from a dataset, to build silver or gold datasets on top of it.

In the session:

You describe the dataset you need. The agent reads the schema and a sample of the input datasets. Inputs must be active, visible to you, and stored in the same warehouse.
The agent writes a query for each dataset and previews its result. A dataset can read another dataset of the same session, so a bronze → silver → gold chain can be prepared at once.
Northstar checks the target: that the warehouse account can create the view or table. A missing right is shown with the exact statement to give your warehouse administrator.
You review each dataset: query, preview, layer, target and refresh.
You apply. The selected datasets are built in the warehouse, in dependency order, then analysed.

View or table

	View	Table
Stored in the warehouse	The query only	The result
Up to date	Always: it is computed when read	After each refresh
Best for	Light queries, small volumes	Heavy joins and aggregations, data read often

The agent recommends one; you choose. After the first build, the layer, the target and the choice of view or table can no longer change.

Refresh

Mode	When it applies
After each upstream update	A table whose inputs are refreshed by Northstar (an ingestion or another transformation). It is rebuilt each time they are.
On a schedule	A table whose inputs are not refreshed by Northstar. Daily by default; you can change the schedule.
No refresh	A view: it is always current. It is only checked when the data upstream changes.

Versions

Each apply creates a version. Versions are kept, so you can see which query built which data.

On a managed dataset, the SQL tab lets you change the query:

Edit the query.
Preview: the first rows, the columns added, removed or retyped, and the datasets built from this one that will be rebuilt. A change that removes a column another dataset uses is flagged.
Apply: the dataset is rebuilt, then the datasets built from it.

Restore applies an earlier version again, as a new version.

Cascade

When an ingestion loads new data, everything built on it follows, in order:

the tables refreshed after each upstream update are rebuilt;
the views are checked;
each rebuilt dataset is analysed and its alerts are checked.

If a dataset fails, the datasets built from it are Skipped for this refresh: they keep their previous data instead of being built on a failed input.

When something changes upstream

If a column a transformation reads is removed or changes type, the refresh fails with Upstream change. Other failures are labelled Permission, SQL error, Target, Dependency or Platform error.

The dataset keeps its last successful build until the problem is fixed.
The transformation shows Error, and the dataset and its source show a badge.
An alert is posted in the session's conversation, once per new error. Repair with AI asks the agent for a fix. The fix is proposed for review; nothing is applied without you.

Who sees the result

A built dataset is never more visible than its inputs. At its first build, it receives the team rights that all its inputs share.

Example: if the input orders is shared with Sales and Finance, and the input customers with Sales only, the result is shared with Sales.
If the inputs have no team in common, only platform administrators see the result until it is given to a team.

The author of the version becomes the owner of the dataset.

FAQ

Can I build a transformation on any warehouse? On PostgreSQL, MySQL, ClickHouse and Snowflake. See storage systems.

Why can't I change a built dataset from view to table? The target is fixed after the first build. Create a new dataset with the other choice.

A refresh failed overnight. Is my dashboard showing wrong numbers? No. The dataset keeps its last successful build, and its badge shows it is behind. The figures are those of the last successful refresh.