Concurrent import

To reduce the amount of time it takes to import data, you can use a concurrent import and run multiple import operations at the same time. Because each REST API call triggers one import job, you can perform multiple REST calls.

The number of import operations is limited by the configuration settings in Collibra Console. Due to machine restrictions, you can run up to 8 import operations concurrently.

Most concurrent import errors occur because of missing dependencies between import commands executed within different jobs at the same time. To ensure successful imports, data should be preprocessed first following these rules:

• A concurrent import of communities and domains is not supported. You must create them manually or import them in a dedicated job.
• You should not use two concurrent jobs to import assets with the same identifier.
• You should not use two concurrent jobs that specify the same relation between two assets.

By default, the whole import job is executed in one single database transaction. The bigger the transaction size, the higher the chance of database locks and job failure. To limit the database transaction size, you can use any of the following options:

• Use smaller input files, with no more than 10,000 assets. In case of concurrency errors, retry the failed job.
• Run import jobs with the continueOnError parameter set to true. With this option, Collibra automatically splits the input data into smaller chunks.

.