Overview possible scenarios

When you are defining an integration flow to import data from a different application in Collibra DGC, there are a few scenarios that the integration has to support. Below is a list of the most common and important scenarios.

Scenario Description Solution
Only import the delta

You do not typically want to import the entire external system every time you run the integration. You only want to import the assets from the external system that are new or have changed since the last time the integration was running.

This is mainly for performance reasons.

To accomplish this, you have to:

  1. Uniquely identify the entity in the external system.
  2. Map the external entity to the Collibra DGC asset.
  3. Keep track of the last sync date.
  4. Include the last sync date in the query to get the external system entities.
Move asset in Collibra DGC When you have imported an external entity into Collibra DGC with the integration flow, you usually want to move that asset to a different community or domain. Other systems typically don't have the same concept of communities or domains, as they are not governance solutions. When you have moved an asset in Collibra DGC to a different community or domain, you do not want the asset to be created again the next time you run the integration flow. The integration flow has to be smart enough to recognize that the asset already exists in Collibra DGC, but that it's in a different location.

To accomplish this, you have to:

  1. Uniquely identify the entity in the external system.
  2. Map the external entity to the Collibra DGC asset.
  3. Update the (moved) asset instead of creating a new one when importing.
Update asset in external system When an entity is updated in the external system, you want the changes to be reflected in the asset in Collibra DGC. You do not want to create a completely new asset in Collibra DGC, because you might have added other attributes or relations to the asset that you do not want to lose.

To accomplish this, you have to:

  1. Uniquely identify the entity in the external system.
  2. Find the corresponding asset in Collibra DGC through the mapping.
    If the corresponding asset exists in Collibra DGC, update it instead of creating a new one.
Recreate asset in Collibra DGC When an imported asset in Collibra DGC is deleted by accident, you typically want it to be re-created the next time the integration flow is run.

To accomplish this, you have to:

  1. Uniquely identify the entity in the external system.
  2. Find the corresponding asset in Collibra DGC through the mapping.
  3. Re-create the corresponding asset if it no longer exists in Collibra DGC.
Delete asset in external system When entity is deleted in the external system you usually also want to delete the corresponding asset in Collibra DGC. Other approaches could be to change the status of the asset in Collibra DGC to Deleted or Deprecated for example.

To accomplish this, you have multiple options:

Sometimes external systems provide easy ways to retrieve entities that were removed since given point in time. If this is the case you can:

  1. Retrieve unique identifiers of the entities that were removed in the external system.
  2. Find corresponding assets in Collibra DGC through the mapping.
  3. Remove or change the status of the assets that were found.

If external system doesn't provide information about entities that were removed then you can:

  1. Review all the Collibra DGC assets that are mapped to an entities in an external system during the integration process .
  2. Check if the entity still exists in the external system for each mapped asset in Collibra DGC (ideally in batch instead of one by one).
  3. Delete the asset in Collibra DGC or change its status if the asset does no longer exist in the external system.

When using the upsert assets operation, you do not have to look manually for corresponding assets in Collibra DGC through mapping because it is done automatically. You only have to provide the unique identifier of the entity in the external system (here you can find more information about that).

The operation also takes care of recreating assets that have been removed from Collibra DGC and of checking if an asset already exists and creating or updating it accordingly. This way, you are free to move assets to different domains in Collibra DGC, as described in the Move asset in Collibra DGC scenario.