Skip to content
northstar
Docs menu

What updates on its own

The chain of refreshes that follows new data, and where to monitor it.

In short

When new data arrives, Northstar updates what depends on it, in order: datasets, analyses, alerts. KPIs and the Smart Cockpit are computed every day, and AI recommendations are refreshed in the background. The Workflows page shows the scheduled jobs linked to the datasets you can see, and their last runs.

The refresh chain

Here is what happens after an ingestion loads new data, step by step:

  1. The ingestion runs, on its schedule or when someone starts it.
  2. The loaded tables are registered as bronze datasets.
  3. The datasets built on them are rebuilt: the tables refreshed after each upstream update, in dependency order. Views are checked.
  4. Each rebuilt dataset is analysed: validation rules, quality score and early warnings.
  5. The alerts of those datasets are checked, and their owners get an email when a condition is met.

If a step fails, what depends on it is skipped and keeps its last successful data. See cascade.

Other refreshes:

  • Scheduled transformations: tables whose inputs are not refreshed by Northstar are rebuilt on their own schedule. Due refreshes are checked every 5 minutes.
  • Datasets not built by Northstar are analysed when someone runs an analysis from the Quality tab, or when a data pipeline requests it through the API.
  • Document collections are synced when someone clicks Resync now, or when a pipeline requests it.

KPIs and the Smart Cockpit

  • A KPI is computed when it is created or edited.
  • Every day, all KPIs are computed again, then the Smart Cockpit insights and executive summary are written for each person who has a cockpit.

A KPI therefore shows the data as it was at its last computation. See KPIs.

AI recommendations

  • KPI and validation recommendations are refreshed every 10 minutes for the datasets whose documentation is complete enough.
  • Column descriptions requested with Generate all missing metadata are written in the background. They appear as suggestions when they are ready.
  • Silver and gold proposals are posted after a successful ingestion, when nothing is built on the source's tables yet.

The Workflows page

Workflows lists the scheduled jobs and pipelines linked to the datasets and sources you can see: data ingestions, and the pipelines of external datasets.

For each workflow you see its schedule, whether it is paused, its triggers and its last runs. A run is running, completed, failed, crashed or cancelled.

Depending on your role, you can:

  • Pause or Resume its schedule;
  • Trigger Run to start it now;
  • Refresh configuration to read the latest schedules and triggers. This also happens automatically every week.

FAQ

My dashboard shows yesterday's numbers. Why? Follow the chain back: was the source's last ingestion successful? Was the dataset rebuilt? KPIs are computed once a day, so a KPI can lag behind its dataset.

Can I make a KPI refresh more often? Editing a KPI computes it right away. Otherwise, KPIs follow the daily computation.

Where do I see why a refresh failed? On the source's runs for an ingestion, on the dataset's Transformation tab for a transformation, and on the Workflows page for other jobs.

See also