Sources and ingestion

In short

A source is where data comes from: a CRM, an internal API, a SaaS tool. An ingestion is the pipeline that loads it into one of your warehouses. You can describe the data you need and let the ingestion agent write and test the pipeline, or register a pipeline that already runs elsewhere. Every table a pipeline loads becomes a bronze dataset, linked to its source.

Source and ingestion

• The source is the catalogue entry: name, logo, category and health.
• The ingestion is the pipeline that loads the source: its code, its configuration, its versions and every run.

A source has at most one ingestion.

Managed, external and draft sources

The New source menu offers both ways in: With the AI agent, or Register an existing pipeline. An external source can be taken over later with Manage with the AI agent.

The ingestion agent

You describe the data you need, for example "Salesforce opportunities and accounts". You can paste links to the documentation of an internal API. The agent then works in steps:

1. It researches the source and asks you to decide where it matters, each time with a recommended option.
2. It requests credentials through a form outside the conversation. Values are encrypted as soon as you send them: neither the agent nor the conversation ever sees them. Only their names are shown afterwards.
3. It writes the pipeline and its configuration: which tables to load, how to write each one (Append, Replace or Merge on a key), and the schedule.
4. It tests the pipeline on a sample, in an isolated environment, and shows the tables, row counts and a preview.
5. You review the configuration, the code and the preview. You can edit them; any edit requires a new test.
6. You deploy with Approve and deploy. This creates a version, schedules it and starts a first run.

Runs and schedules

Each ingestion run has a mode:

• Test: started by the agent on a sample, before deployment.
• Manual: started with Run now.
• Scheduled: started by the ingestion's schedule. Without a schedule, the ingestion runs manually only.

A run is Queued, Running, then Succeeded or Failed. A run still queued after 30 minutes is marked as failed. Pause suspends scheduled runs; Resume restarts them.

Every deployment creates a new version. Versions cannot be changed afterwards, so you can always see which code loaded which data.

After a successful run

After each successful manual or scheduled run, Northstar does three things:

1. It registers each loaded table as a bronze dataset, linked to the source.
2. It refreshes what is built on those tables: the silver and gold datasets that depend on them are rebuilt, then analysed. See the refresh chain.
3. It may propose silver and gold datasets in the source's conversation when nothing is built on its tables yet. You can open a session from these proposals, or set them aside with Later: new proposals come when the source's tables change.

Health

Each source shows the health of its last run: Healthy, Unhealthy, Syncing or Unknown. The home page shows the share of successful syncs over the last 24 hours.

FAQ

What happens to the data if I delete a source? The datasets already loaded are kept. Deleting a managed source also deletes its ingestion: versions, runs and conversation.

Who can fill in the credentials form? An editor of the ingestion. The values are never visible, not even to the person who entered them.

Can I edit the code the agent wrote? Yes, in the review card. The edited code must pass a new test before it can be deployed.

See also

• Storage systems
• Datasets and their layers
• Transformations
• What updates on its own