# Data Hub (/docs/use/portal/data-hub)





Data Hub is the Portal workspace for data engineers and authorized tenant members who configure and operate managed ingestion. It brings connector registration, connections, ingestions and run diagnostics into one interface.

## Before you start [#before-you-start]

Select the intended tenant and open **Data Hub** from the Portal workspace switcher. You need permissions for the actions you perform and access to the relevant connections and ingestions. An administrator can assign functional or custom roles, while resource owners manage sharing.

Data Hub remains navigable even when you lack an operation permission. Individual pages explain missing access, and write actions remain permission-gated. A visible workspace or navigation item does not grant permission to list or change its resources.

For a first transfer, have source and destination connector definitions available in your tenant, connection details and appropriate credentials, and approved network access to both systems. If a connector is missing, ask its maintainer to [build and register it](/docs/develop/data-connectors). The catalogue contains registered connectors; it does not automatically contain every MeltanoHub or Singer connector.

## Understand the four resources [#understand-the-four-resources]

| Resource   | What it represents                                                                                                              |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Connector  | A source or destination implementation, its immutable image version, capabilities and configuration schemas.                    |
| Connection | A configured instance of a connector version, with credentials stored separately from ordinary settings.                        |
| Ingestion  | A source-to-destination transfer with selected streams, replication settings and destination options.                           |
| Run        | One execution, including its status, timing, result and available diagnostics. Connection tests and discovery also create runs. |

The connector catalogue is not the [lakehouse Data Catalog](/docs/data/data-catalog). Registering a connector does not publish datasets, grant table access or create catalog lineage.

## 1. Create and test connections [#1-create-and-test-connections]

1. Open **Connections** and create a source connection from an available source connector and version.
2. Complete the configuration fields generated from its schema. Put passwords, tokens and access keys in credential fields, not ordinary configuration.
3. Save the connection. If it supports testing, run a connection test and inspect the resulting run.
4. Create the destination connection using its destination connector and repeat the supported checks.

Some connectors do not provide a standalone connection test. Their capabilities determine which actions are available; a disabled test is not proof that the destination is unusable. Validate that destination with a small real ingestion.

Connection configuration can be edited, and another registered version can be selected. Credential replacement requires separate permission and resource access. Existing secret values are not returned by management reads. Coordinate connector-version changes with ingestion progress and pause automated execution before incompatible changes.

## 2. Configure an ingestion [#2-configure-an-ingestion]

1. Open **Ingestions** and create an ingestion with the source and destination connections.
2. Discover source streams where the connector supports discovery. Choose the streams to transfer.
3. Select the supported replication method. For incremental replication, choose the replication cursor separately from any primary-key fields. A cursor tracks progress; primary keys identify records for destinations that support merging or upserts.
4. Complete the destination settings and save the ingestion.

<img alt="Data Hub New ingestion form with source and destination connections, a selected stream, replication mode, and destination settings" src="__img0" />

The Portal keeps an ingestion's source, destination and stream configuration fixed after creation. Create a replacement ingestion to change that configuration. Its progress is separate; plan replay and destination continuity instead of assuming the old checkpoint is transferred.

## 3. Run and verify the transfer [#3-run-and-verify-the-transfer]

Run the ingestion manually before enabling automation. Open the run from the ingestion page or **Runs** to inspect its state and available source, transfer and destination diagnostics.

Verify the destination itself: expected records, schemas, key handling and repeat-run behavior. A successful run is evidence of execution, not a guarantee that every business-level data expectation was met.

For a failed run, use its failure code and stage diagnostics to correct the cause before starting another run. Diagnostics are sanitized summaries, not unrestricted connector logs. Cancellation stops execution; it does not roll back records already written. Retries and state replay can duplicate writes unless the destination implements suitable deduplication or upsert behavior.

## 4. Schedule an ingestion [#4-schedule-an-ingestion]

Open the ingestion's **Schedule** panel:

1. Enter a five-field cron expression, for example `0 6 * * *` for 06:00 daily.
2. Set the timezone explicitly, for example `Europe/Madrid` or `UTC`.
3. Choose your identity or an available tenant workload identity. The ingestion owner establishes or changes its execution identity; the selected identity also needs the required functional and resource access.
4. Save the schedule, clearing **Suspended** when it is ready to run.

Use a tenant workload identity for unattended operations when appropriate; an operator provisions that identity before it appears in the list. Authorization is checked again during execution, so revoked access can stop scheduled work. Overlapping schedule firings are skipped rather than creating concurrent scheduled transfers for the same ingestion.

Suspending a schedule prevents future scheduled starts. Handle any active run separately. Removing a schedule does not delete its ingestion or destination data.

## Upload and event triggers [#upload-and-event-triggers]

An ingestion can also expose an authenticated callback for an external event producer. The source connector must understand the event payload; enabling the trigger alone does not configure a bucket notification or make every connector event-aware.

1. Enable the ingestion's upload trigger and save its callback URL, trigger ID and one-time token in protected configuration.
2. Have the source-system owner configure the event producer to call that endpoint. Ceph bucket notifications and AWS S3 event delivery use different setup paths.
3. Send a representative event and verify both the resulting run and destination records.

Treat the trigger token as a credential. Rotating it requires updating the producer. Pause the trigger before maintenance or recovery. See [connector event integration](/docs/develop/data-connectors#event-aware-sources) for the API contract and examples.

## Access and sharing [#access-and-sharing]

Data Hub combines tenant membership, functional permissions and resource relationships. A role permitting ingestion execution does not automatically grant access to every connection.

| Task                                                 | Functional permissions and resource access                                                                           |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Browse definitions and authorized resources          | Corresponding `data.connector_definition`, `data.connection`, `data.ingestion` and `data.run` read/list permissions. |
| Configure and run transfers                          | Creation/update/test/discovery/use/run permissions, with editor or invoker access as appropriate.                    |
| Schedule or cancel work                              | Schedule/cancel permissions and the required operator relationship.                                                  |
| Replace connection credentials                       | `data.connection.credential.update` and operator access. This is separate from general builder permissions.          |
| Delete resources or reset/restore ingestion progress | Corresponding functional permissions and ownership.                                                                  |

The Data Consumer, Data Builder and Data Steward functional roles group common tasks. Credential management is a separate role. Custom roles can select individual permissions. Sharing exposes viewer, editor, invoker, operator and grant-manager relationships; delegating grant management and transferring ownership remain owner actions. New owners must be eligible users in the tenant.

Platform or tenant administration is not a blanket grant to private connection credentials or ingestion content. Check the active tenant, functional role and resource sharing when an item or action is unavailable.

## Recover progress [#recover-progress]

Recovery changes the ingestion's checkpoint, not the destination database or object store.

1. Suspend its schedule, pause its event trigger and wait for active execution to finish or cancel it.
2. Inspect the failure and destination state with the data owner. Decide whether replay is acceptable before changing progress.
3. In **Recovery**, load the latest snapshot before restoring, or use **Reset state** only when restarting progress is intended.
4. Inspect the recovery run, then perform a controlled ingestion and verify destination contents before resuming automation.

Restore validates snapshot identity and integrity and refuses incompatible existing progress. Do not reset simply to bypass that refusal: agree a recovery plan first. Reset and replay can produce duplicate records. Ask the platform operator to investigate missing or unreadable backups using the [managed-ingestion reference](/docs/data/components/managed-ingestion#checkpoint-backup-and-recovery).

## Troubleshooting [#troubleshooting]

| Symptom                               | Check first                                                                                          |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Missing connector or action           | Active tenant, functional permissions and the connector's advertised capabilities.                   |
| Cannot use a connection or ingestion  | Resource sharing as well as functional permissions; administration alone is not sufficient.          |
| Image pull or startup failure         | Registered digest, cluster registry access and the connector's runtime contract.                     |
| Connection failure                    | Credentials, TLS trust, approved endpoint access and native database/storage rights.                 |
| Discovery succeeds but transfer fails | Destination settings, selected streams, keys, write privileges and stage diagnostics.                |
| Scheduled work stops                  | Schedule suspension, execution identity, revoked permissions and backend worker health.              |
| State restore fails                   | Latest snapshot availability, ingestion identity, integrity and compatibility with current progress. |

## Related pages [#related-pages]

* [Working with data](/docs/data/working-with-data)
* [Build and register a connector](/docs/develop/data-connectors)
* [Managed data ingestion](/docs/data/components/managed-ingestion)
* [Build a data and ML workflow](/docs/get-started/build-data-workflow)