Upserting assets

Upsert assets operation

Integration Step	Description
Get External Data	In this step, the external system is queried to retrieve the data. This can be done through an SQL connector, a custom connector or something similar. Sometimes it is useful to create a Java Class (POJO), so that the retrieved data is instantiated as Java Objects. That makes it easier to test and map the data to the Collibra DGC format in future steps. It also improves the robustness of the solution. This step is identical as in the method where import to CSV is used.
Transform using DataWeave	This step describes the core business logic of the integration. Here you define how to transform the entity retrieved from the external system to an asset of chosen type in Collibra DGC.
Upsert to Collibra DGC	In this step, given assets are upserted into Collibra DGC. It is equal to the last three steps described in the import CSV section (Importing CSV data). Here each asset is enriched with mapping to check if it was previously synchronized into Collibra DGC. Conversion into CSV and import are also made automatically.

Integration Step

Description

Get External Data

In this step, the external system is queried to retrieve the data. This can be done through an SQL connector, a custom connector or something similar.

Sometimes it is useful to create a Java Class (POJO), so that the retrieved data is instantiated as Java Objects. That makes it easier to test and map the data to the Collibra DGC format in future steps. It also improves the robustness of the solution.

This step is identical as in the method where import to CSV is used.

Transform using DataWeave

This step describes the core business logic of the integration. Here you define how to transform the entity retrieved from the external system to an asset of chosen type in Collibra DGC.

Upsert to Collibra DGC

In this step, given assets are upserted into Collibra DGC. It is equal to the last three steps described in the import CSV section (Importing CSV data). Here each asset is enriched with mapping to check if it was previously synchronized into Collibra DGC. Conversion into CSV and import are also made automatically.

For more information about upserting assets, see About upserting assets.