Docs menu
Storage systems
Where your datasets and document collections are stored, and how credentials are protected.
In short
A storage system is a connection to a place where your data lives: a SQL warehouse, or a file store such as a drive or a bucket. Northstar reads datasets and documents from it, and writes the silver and gold datasets it builds there. Tables stay in your warehouse: queries run where the data is.
Supported kinds
| Family | Kinds | What it is used for |
|---|---|---|
| SQL warehouses | PostgreSQL, MySQL, ClickHouse, Snowflake, Amazon Redshift, Databricks, Google BigQuery, Microsoft Fabric | Datasets, KPIs, dashboards, Chat and the SQL Executor |
| File stores | Amazon S3, Google Drive, OneDrive, SharePoint | Document collections (PDF), and tabular files (CSV, TSV, Parquet) registered as datasets |
Not every kind does everything:
- Ingestion destinations. A pipeline written with the ingestion agent can load into PostgreSQL, MySQL, ClickHouse, Snowflake, Amazon Redshift, Databricks, Google BigQuery or Amazon S3.
- Transformations. Northstar builds silver and gold datasets on PostgreSQL, MySQL, ClickHouse and Snowflake.
Credentials
You enter credentials when you register the storage system.
- Passwords, keys and connection secrets are encrypted before they are stored. They are never shown again after you save: to change them, you enter new ones.
- Each system shows its status: Credentials valid, Credentials invalid or Credentials missing.
- Test connection checks the credentials. It needs credentials first.
- Deleting a storage system also deletes the credentials stored for it.
Where silver and gold datasets are written
Each warehouse that supports transformations has a Silver schema and a Gold schema. On ClickHouse, these are databases.
- Leave a field empty to use the default:
silverandgold. - Changing them later does not move anything: datasets already built keep their location.
- Before a transformation is applied, Northstar checks that the account can write to the target and lists any missing right.
FAQ
Can I read a password I entered? No. Secrets are encrypted at rest and never displayed again. Enter new values to replace them.
Why can't I build a silver or gold dataset on this warehouse? Transformations run on PostgreSQL, MySQL, ClickHouse and Snowflake only. On the other kinds you can still register datasets, define KPIs and build dashboards.
Does Northstar copy my tables? No. Datasets are queried where they live. Ingestion pipelines load data into the warehouse you choose, and document collections keep an index of the PDF text so that Chat can search it.