Query the Developer Portal from Claude, Codex, Cursor, or VS Code!
Connect with MCP
Ctrlk
Collibra sitesRequest a demo
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Create and run data quality jobs with the Data Quality REST API

In this tutorial, you learn how to use the Data Quality REST API to create a data quality job, run it, check its status, and get its results. Use this workflow to add data quality checks to your data pipelines, orchestration tools, and CI/CD processes.

This tutorial is only for Data Quality & Observability (cloud).

Prerequisites

Task
Resource permission

Create a job

Data Quality Job > Create

View jobs, job runs, and run results

Data Quality Job > View

Run or cancel a job

Data Quality Job > Run

Update a job

Data Quality Job > Edit

Delete a job or a job run

Data Quality Job > Delete

  • If you need access to all jobs regardless of resource permissions, ask your administrator to grant Product Rights > Data Quality combined with one of these global permissions:

    • View Permissions > View All, for read access to all jobs.

    • Resources > Manage all resources, for full access to all jobs.

  • You either created and installed an Edge site or were granted a Collibra Cloud site with a connection to the data source that you want to monitor. You need the names of the Edge site, Edge connection, data source, schema, and table.

  • You can authenticate with the Collibra REST API. This tutorial uses basic authentication with a username and password. For data pipelines and CI/CD tools, use token-based authentication with a JSON Web Token (JWT) instead.

About the Jobs resource

A data quality job defines which data to check and how to check it. A job contains:

  • A data location: the Edge site, Edge connection, data source, schema, and table.

  • An optional source query that selects the rows and columns to check.

  • Monitoring settings, notification settings, and an optional schedule.

Each time a job runs, it creates a job run with a unique jobRunId. Use the job run to track the status and get the results.

This tutorial uses these endpoints:

  • GET /jobs: searches for existing jobs.

  • POST /jobs: creates a job and, by default, queues its first run.

  • GET /jobRuns/{jobRunId}: returns the status and summary results of a run.

  • GET /jobRuns/{jobRunId}/monitors: returns the result of each monitor.

  • GET /jobRuns/{jobRunId}/profile: returns column-level profiling statistics.

  • POST /jobs/{jobName}/run: runs an existing job.

  • PATCH /jobs/{jobName}: updates an existing job.

  • POST /jobRuns/{jobRunId}/cancel: cancels an in-progress run.

  • DELETE /jobRuns/{jobRunId}: deletes a run and its results.

  • DELETE /jobs/{jobName}: deletes a job.

The jobType field sets the processing method of the job:

  • PUSHDOWN: the job runs its queries in your data source.

  • PULLUP: the job uses Spark to process the data. Pullup jobs are available only when the Pullup preview feature is enabled.

By default, POST /jobs queues a run after it creates the job. To create the job without a run, add "queueRun": false to the request body. In this case, the jobRunId in the response is empty.

Find existing jobs

Before you create a job, search for jobs on the same table. This prevents duplicate jobs.

The response is a paged list of jobs, with the most recently updated job first. Use the limit and offset query parameters to page through the results. The jobName query parameter is a case-insensitive partial match.

You can also filter by edgeSiteName, edgeConnectionName, dataSourceName, and jobType.

Create and run your first job

1

Send a POST request to /jobs with the job type and the data location. This example creates a Pushdown job with default settings and queues a run.

If you omit jobName, the API uses <schema>.<table> as the name. If a job with this name exists, the API adds a counter, for example SALES.CUSTOMER_ORDERS_1. Job names can only contain letters, digits, _, ., and -.

If you omit sourceQuery, the job checks a sample of approximately 20,000 rows, not the full table. To check all rows, provide your own sourceQuery. For an example, see Check a time slice of data.

If you omit monitoringSettings, the job uses the default adaptive monitors: row count, null values, empty fields, and uniqueness.

2

Save the jobRunId from the response. A 201 response contains the job definition and the ID of the queued run.

3

Send a GET request to /jobRuns/{jobRunId} to get the status of the run.

Send the request again at an interval, for example every 30 seconds, while status is one of these values:

  • WAITING

  • DISPATCHED

  • SETUP

  • RUNNING

  • SENDING

Treat any other status as final. Final statuses include FINISHED, FAILED, CANCELLED, and UNKNOWN. New statuses can be added in the future. While the run is in progress, the activity field shows the current phase, for example LOAD, PROFILE, or RULES.

When the run is finished, the response also contains the summary results:

The score is the percentage of passing monitors, as a whole number.

If the status is FAILED, the exception field contains the error message.

4

Send a GET request to /jobRuns/{jobRunId}/monitors to get the result of each monitor.

The response contains two lists: adaptiveMonitors and customMonitors. The state of each monitor is one of these values: LEARNING, PASSING, BREAKING, SUPPRESSED, USER_PASSED, EXCEPTION, STALE, or SKIPPED.

On a new job, adaptive monitors show the LEARNING state. During the first runs, the monitors learn a baseline of your data before they can pass or break. By default, the learning phase is 4 runs. To change it, set monitoringSettings.adaptiveMonitors.settings.learningPhase on the job.

The example response shows a run after the learning phase.

Common use cases

Check a time slice of data

To check only a part of a table, such as the orders of one day, add a sourceQuery with the ${rd} and ${rdEnd} placeholders. Set the values of the placeholders with runDate and runDateEnd.

Because the SQL contains single quotes and ${...} placeholders, save the request body to a file.

Do not put literal dates in the SQL. Always use ${rd} and ${rdEnd}, and keep them in single quotes. runDateEnd is exclusive and must be later than runDate.

If you omit jobSettings.dateFormat, the API uses the type of runDate. You do not need to set both. To use timestamps, use "kind": "TIMESTAMP" with an RFC 3339 value, for example 2026-09-22T00:00:00Z.

Run an existing job from a data pipeline

To run a job after your pipeline loads new data, send a POST request to /jobs/{jobName}/run. The request body is optional. If you omit it, the run date is the start of the current day in UTC, and the run has no end date.

If the source query of the job uses ${rdEnd}, always send runDate and runDateEnd.

A 202 response contains the new jobRunId. Use it to check the status and get the results, as in steps 3 and 4 of Create and run your first job.

To backfill previous periods, add a backrun object. For example, "backrun": { "timeBin": "DAY", "binValue": 7 } also runs the job for the 7 days before the run date. The timeBin can be DAY, MONTH, or YEAR. By default, the maximum binValue is 50.

Schedule a daily job

To run a job on a schedule, send a PATCH request to /jobs/{jobName} with schedulingSettings.

You cannot update a job while one of its runs is in progress. The job in this example already has runs from the previous sections. Before you send a PATCH request, make sure that the latest run has a final status. For more information, see step 3 of Create and run your first job.

A PATCH request changes only the top-level fields that you include. If you include schedulingSettings or notifications, the API replaces the full object. Always send the complete object for these two fields.

This job runs at 02:00 UTC from Monday to Friday. The ONE_DAY offset sets the run date to the day before the scheduled day, so each run checks the data of the previous day. The schedule also supports HOURLY and MONTHLY modes.

scheduledRunTime is always in UTC, in the 24-hour format HH:mm:ss.

Send email notifications

To send an email when a run fails or when the score is low, send a PATCH request with notifications. The API replaces the full notifications object, so include all the notification options that you want to keep.

Email is the only channel, so one EMAIL entry in channels is sufficient. The recipients must be existing Collibra usernames, not email addresses. The API skips names that do not match a user.

The quantity field is used only by threshold notification types, such as SCORE_LESS_THAN_LIMIT and ROWS_LESS_THAN_LIMIT.

Find failed runs

To find all failed runs of jobs that contain sales in the name, filter /jobRuns by status and name.

By default, jobName is a partial match. For an exact match, add nameMatchMode=EXACT. In both modes, the match is not case-sensitive. To filter by more than one status, repeat the parameter, for example status=FAILED&status=CANCELLED.

Get profiling statistics

To get column-level statistics for a run, such as null counts, distinct counts, and top value shapes, send a GET request to /jobRuns/{jobRunId}/profile.

The job applies masking when it runs, and stores only the masked values. By default, monitoringSettings.adaptiveMonitors.descriptiveStatistics is false. In this case:

  • The profile masks the minimum, maximum, mean, and top and bottom values.

  • The job skips histograms, correlation, and rules that use stat variables.

  • The profile still shows row counts, null and empty counts, unique counts, and data types.

To show the values, set this field to true on the job. The change applies only to new runs. Earlier runs stay masked. The values can contain sensitive data.

Cancel or delete runs and jobs

To stop a run that is in progress, send a POST request to /jobRuns/{jobRunId}/cancel.

To delete a run, send a DELETE request to /jobRuns/{jobRunId}. This also deletes all run results, such as the profile, monitor, rule, and alert results.

You cannot delete a run that is in progress. Cancel the run first. If the run is in progress, the API returns a 409 response.

To delete a job, send a DELETE request to /jobs/{jobName}.

Additional resources

PreviousExport Data Quality & Observability (cloud) data with the Data Egress API
NextRun data quality jobs from a PySpark notebook

Last updated

Was this helpful?