> For the complete documentation index, see [llms.txt](https://developer.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.collibra.com/tutorials/create-and-run-data-quality-jobs-with-the-data-quality-rest-api.md). # Create and run data quality jobs with the Data Quality REST API In this tutorial, you learn how to use the [Data Quality REST API](https://developer.collibra.com/api/references/data-quality) to create a data quality job, run it, check its status, and get its results. Use this workflow to add data quality checks to your data pipelines, orchestration tools, and CI/CD processes. {% hint style="info" %} This tutorial is only for Data Quality & Observability (cloud). {% endhint %} ### Prerequisites * You have the **Data Quality Pushdown Processing** or **Data Quality Pullup Processing** [Edge capability](https://productresources.collibra.com/docs/collibra/latest/Content/UnifiedDataQuality/ta_add-data-quality-capability.htm) enabled for your [data source](https://productresources.collibra.com/docs/collibra/latest/Content/UnifiedDataQuality/ta_create-jdbc-connection-for-dq.htm). * You have a global role with the **Product Rights > Data Quality** [global permission](https://productresources.collibra.com/docs/collibra/latest/Default.htm#cshid=DOC0630), plus the [resource permission](https://productresources.collibra.com/docs/collibra/latest/Content/UnifiedDataQuality/ref_data-quality-resource-roles-permissions.htm) for each task: | Task | Resource permission | | ------------------------------------ | ------------------------- | | Create a job | Data Quality Job > Create | | View jobs, job runs, and run results | Data Quality Job > View | | Run or cancel a job | Data Quality Job > Run | | Update a job | Data Quality Job > Edit | | Delete a job or a job run | Data Quality Job > Delete | * If you need access to all jobs regardless of resource permissions, ask your administrator to grant **Product Rights > Data Quality** combined with one of these global permissions: * **View Permissions > View All**, for read access to all jobs. * **Resources > Manage all resources**, for full access to all jobs. * You either created and installed an [Edge site](https://productresources.collibra.com/docs/collibra/latest/Content/Edge/EdgeSitesInstallation/ta_install-edge-site.htm) or were granted a [Collibra Cloud site](https://productresources.collibra.com/docs/collibra/latest/Content/Edge/co_collibra-cloud-site.htm) with a [connection](https://productresources.collibra.com/docs/collibra/latest/Content/UnifiedDataQuality/ta_create-jdbc-connection-for-dq.htm) to the data source that you want to monitor. You need the names of the Edge site, Edge connection, data source, schema, and table. * You can authenticate with the Collibra REST API. This tutorial uses basic authentication with a username and password. For data pipelines and CI/CD tools, use [token-based authentication with a JSON Web Token (JWT)](https://developer.collibra.com/tutorials/collibra-rest-api-authentication-with-json-web-token) instead. ### About the Jobs resource A data quality job defines which data to check and how to check it. A job contains: * A data location: the Edge site, Edge connection, data source, schema, and table. * An optional source query that selects the rows and columns to check. * Monitoring settings, notification settings, and an optional schedule. Each time a job runs, it creates a job run with a unique `jobRunId`. Use the job run to track the status and get the results. This tutorial uses these endpoints: * `GET /jobs`: searches for existing jobs. * `POST /jobs`: creates a job and, by default, queues its first run. * `GET /jobRuns/{jobRunId}`: returns the status and summary results of a run. * `GET /jobRuns/{jobRunId}/monitors`: returns the result of each monitor. * `GET /jobRuns/{jobRunId}/profile`: returns column-level profiling statistics. * `POST /jobs/{jobName}/run`: runs an existing job. * `PATCH /jobs/{jobName}`: updates an existing job. * `POST /jobRuns/{jobRunId}/cancel`: cancels an in-progress run. * `DELETE /jobRuns/{jobRunId}`: deletes a run and its results. * `DELETE /jobs/{jobName}`: deletes a job. The `jobType` field sets the processing method of the job: * `PUSHDOWN`: the job runs its queries in your data source. * `PULLUP`: the job uses Spark to process the data. Pullup jobs are available only when the Pullup preview feature is enabled. {% hint style="info" %} By default, `POST /jobs` queues a run after it creates the job. To create the job without a run, add `"queueRun": false` to the request body. In this case, the `jobRunId` in the response is empty. {% endhint %} ### Find existing jobs Before you create a job, search for jobs on the same table. This prevents duplicate jobs. ```bash curl "https://{your-collibra-host}/rest/dq/1.0/jobs?schemaName=SALES&tableName=CUSTOMER_ORDERS" \ -u "{username}:{password}" ``` The response is a paged list of jobs, with the most recently updated job first. Use the `limit` and `offset` query parameters to page through the results. The `jobName` query parameter is a case-insensitive partial match. You can also filter by `edgeSiteName`, `edgeConnectionName`, `dataSourceName`, and `jobType`. ### Create and run your first job {% stepper %} {% step %} Send a POST request to `/jobs` with the job type and the data location. This example creates a Pushdown job with default settings and queues a run. {% code title="Request" %} ```bash curl -X POST \ "https://{your-collibra-host}/rest/dq/1.0/jobs" \ -u "{username}:{password}" \ -H "Content-Type: application/json" \ -d '{ "jobName": "sales_customer_orders", "jobType": "PUSHDOWN", "dataLocation": { "edgeSiteName": "my-edge-site", "edgeConnectionName": "snowflake-prod", "dataSourceName": "ANALYTICS", "schemaName": "SALES", "tableName": "CUSTOMER_ORDERS" } }' ``` {% endcode %} {% hint style="info" %} If you omit `jobName`, the API uses `.` as the name. If a job with this name exists, the API adds a counter, for example `SALES.CUSTOMER_ORDERS_1`. Job names can only contain letters, digits, `_`, `.`, and `-`. If you omit `sourceQuery`, the job checks a sample of approximately 20,000 rows, not the full table. To check all rows, provide your own `sourceQuery`. For an example, see [Check a time slice of data](#check-a-time-slice-of-data). If you omit `monitoringSettings`, the job uses the default adaptive monitors: row count, null values, empty fields, and uniqueness. {% endhint %} {% endstep %} {% step %} Save the `jobRunId` from the response. A `201` response contains the job definition and the ID of the queued run. {% code title="Response (abbreviated)" %} ```json { "jobName": "sales_customer_orders", "jobType": "PUSHDOWN", "dataLocation": { "edgeSiteName": "my-edge-site", "edgeConnectionName": "snowflake-prod", "dataSourceName": "ANALYTICS", "schemaName": "SALES", "tableName": "CUSTOMER_ORDERS" }, "jobRunId": "3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d" } ``` {% endcode %} {% endstep %} {% step %} Send a GET request to `/jobRuns/{jobRunId}` to get the status of the run. ```bash curl "https://{your-collibra-host}/rest/dq/1.0/jobRuns/3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d" \ -u "{username}:{password}" ``` Send the request again at an interval, for example every 30 seconds, while `status` is one of these values: * `WAITING` * `DISPATCHED` * `SETUP` * `RUNNING` * `SENDING` Treat any other status as final. Final statuses include `FINISHED`, `FAILED`, `CANCELLED`, and `UNKNOWN`. New statuses can be added in the future. While the run is in progress, the `activity` field shows the current phase, for example `LOAD`, `PROFILE`, or `RULES`. When the run is finished, the response also contains the summary results: {% code title="Response (abbreviated)" %} ```json { "jobRunId": "3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d", "jobName": "sales_customer_orders", "status": "FINISHED", "startTime": "2026-09-23T08:00:04Z", "endTime": "2026-09-23T08:02:41Z", "executionTimeSeconds": 157, "score": 95.0, "activeMonitors": 42, "breakingMonitors": 2, "rowCount": 184320 } ``` {% endcode %} The `score` is the percentage of passing monitors, as a whole number. {% hint style="info" %} If the status is `FAILED`, the `exception` field contains the error message. {% endhint %} {% endstep %} {% step %} Send a GET request to `/jobRuns/{jobRunId}/monitors` to get the result of each monitor. ```bash curl "https://{your-collibra-host}/rest/dq/1.0/jobRuns/3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d/monitors" \ -u "{username}:{password}" ``` The response contains two lists: `adaptiveMonitors` and `customMonitors`. The `state` of each monitor is one of these values: `LEARNING`, `PASSING`, `BREAKING`, `SUPPRESSED`, `USER_PASSED`, `EXCEPTION`, `STALE`, or `SKIPPED`. {% hint style="info" %} On a new job, adaptive monitors show the `LEARNING` state. During the first runs, the monitors learn a baseline of your data before they can pass or break. By default, the learning phase is 4 runs. To change it, set `monitoringSettings.adaptiveMonitors.settings.learningPhase` on the job. The example response shows a run after the learning phase. {% endhint %} {% code title="Response (abbreviated)" %} ```json { "adaptiveMonitors": [ { "monitorName": "ROW COUNT", "monitorType": "ROW_COUNT", "primaryColumn": "ALL", "state": "BREAKING", "observedValue": "184320", "expectedMin": "190000", "expectedMax": "210000", "tolerance": "NEUTRAL", "dimensions": ["Completeness"] } ], "customMonitors": [] } ``` {% endcode %} {% endstep %} {% endstepper %} ### Common use cases #### Check a time slice of data To check only a part of a table, such as the orders of one day, add a `sourceQuery` with the `${rd}` and `${rdEnd}` placeholders. Set the values of the placeholders with `runDate` and `runDateEnd`. Because the SQL contains single quotes and `${...}` placeholders, save the request body to a file. {% code title="create-job.json" %} ```json { "jobName": "sales_customer_orders_daily", "jobType": "PUSHDOWN", "sourceQuery": "SELECT * FROM \"SALES\".\"CUSTOMER_ORDERS\" WHERE \"ORDER_DATE\" >= '${rd}' AND \"ORDER_DATE\" < '${rdEnd}'", "runDate": { "kind": "DATE", "value": "2026-09-22" }, "runDateEnd": { "kind": "DATE", "value": "2026-09-23" }, "dataLocation": { "edgeSiteName": "my-edge-site", "edgeConnectionName": "snowflake-prod", "dataSourceName": "ANALYTICS", "schemaName": "SALES", "tableName": "CUSTOMER_ORDERS" } } ``` {% endcode %} {% code title="Request" %} ```bash curl -X POST \ "https://{your-collibra-host}/rest/dq/1.0/jobs" \ -u "{username}:{password}" \ -H "Content-Type: application/json" \ -d @create-job.json ``` {% endcode %} {% hint style="info" %} Do not put literal dates in the SQL. Always use `${rd}` and `${rdEnd}`, and keep them in single quotes. `runDateEnd` is exclusive and must be later than `runDate`. If you omit `jobSettings.dateFormat`, the API uses the type of `runDate`. You do not need to set both. To use timestamps, use `"kind": "TIMESTAMP"` with an RFC 3339 value, for example `2026-09-22T00:00:00Z`. {% endhint %} #### Run an existing job from a data pipeline To run a job after your pipeline loads new data, send a POST request to `/jobs/{jobName}/run`. The request body is optional. If you omit it, the run date is the start of the current day in UTC, and the run has no end date. {% hint style="info" %} If the source query of the job uses `${rdEnd}`, always send `runDate` and `runDateEnd`. {% endhint %} {% code title="Request" %} ```bash curl -X POST \ "https://{your-collibra-host}/rest/dq/1.0/jobs/sales_customer_orders_daily/run" \ -u "{username}:{password}" \ -H "Content-Type: application/json" \ -d '{ "runDate": { "kind": "DATE", "value": "2026-09-23" }, "runDateEnd": { "kind": "DATE", "value": "2026-09-24" } }' ``` {% endcode %} A `202` response contains the new `jobRunId`. Use it to check the status and get the results, as in steps 3 and 4 of [Create and run your first job](#create-and-run-your-first-job). To backfill previous periods, add a `backrun` object. For example, `"backrun": { "timeBin": "DAY", "binValue": 7 }` also runs the job for the 7 days before the run date. The `timeBin` can be `DAY`, `MONTH`, or `YEAR`. By default, the maximum `binValue` is 50. #### Schedule a daily job To run a job on a schedule, send a PATCH request to `/jobs/{jobName}` with `schedulingSettings`. {% hint style="info" %} You cannot update a job while one of its runs is in progress. The job in this example already has runs from the previous sections. Before you send a PATCH request, make sure that the latest run has a final status. For more information, see step 3 of [Create and run your first job](#create-and-run-your-first-job). {% endhint %} A PATCH request changes only the top-level fields that you include. If you include `schedulingSettings` or `notifications`, the API replaces the full object. Always send the complete object for these two fields. {% code title="Request" %} ```bash curl -X PATCH \ "https://{your-collibra-host}/rest/dq/1.0/jobs/sales_customer_orders_daily" \ -u "{username}:{password}" \ -H "Content-Type: application/json" \ -d '{ "schedulingSettings": { "schedulerMode": "DAILY", "isActive": true, "scheduledRunTime": "02:00:00", "daily": { "dailyOffset": "ONE_DAY", "daysOfWeek": ["MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY"] } } }' ``` {% endcode %} This job runs at 02:00 UTC from Monday to Friday. The `ONE_DAY` offset sets the run date to the day before the scheduled day, so each run checks the data of the previous day. The schedule also supports `HOURLY` and `MONTHLY` modes. {% hint style="info" %} `scheduledRunTime` is always in UTC, in the 24-hour format `HH:mm:ss`. {% endhint %} #### Send email notifications To send an email when a run fails or when the score is low, send a PATCH request with `notifications`. The API replaces the full `notifications` object, so include all the notification options that you want to keep. {% code title="Request" %} ```bash curl -X PATCH \ "https://{your-collibra-host}/rest/dq/1.0/jobs/sales_customer_orders_daily" \ -u "{username}:{password}" \ -H "Content-Type: application/json" \ -d '{ "notifications": { "notificationOptions": [ { "notificationType": "JOB_FAILED", "enabled": true }, { "notificationType": "SCORE_LESS_THAN_LIMIT", "enabled": true, "quantity": 80 } ], "useIndividualMessages": false, "globalMessage": "Check the CUSTOMER_ORDERS data quality job.", "channels": [ { "channel": "EMAIL", "recipients": ["jsmith", "adoe"] } ] } }' ``` {% endcode %} {% hint style="info" %} Email is the only channel, so one `EMAIL` entry in `channels` is sufficient. The `recipients` must be existing Collibra usernames, not email addresses. The API skips names that do not match a user. The `quantity` field is used only by threshold notification types, such as `SCORE_LESS_THAN_LIMIT` and `ROWS_LESS_THAN_LIMIT`. {% endhint %} #### Find failed runs To find all failed runs of jobs that contain `sales` in the name, filter `/jobRuns` by status and name. ```bash curl "https://{your-collibra-host}/rest/dq/1.0/jobRuns?status=FAILED&jobName=sales" \ -u "{username}:{password}" ``` By default, `jobName` is a partial match. For an exact match, add `nameMatchMode=EXACT`. In both modes, the match is not case-sensitive. To filter by more than one status, repeat the parameter, for example `status=FAILED&status=CANCELLED`. #### Get profiling statistics To get column-level statistics for a run, such as null counts, distinct counts, and top value shapes, send a GET request to `/jobRuns/{jobRunId}/profile`. ```bash curl "https://{your-collibra-host}/rest/dq/1.0/jobRuns/3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d/profile?limit=100&includeTotal=true" \ -u "{username}:{password}" ``` {% hint style="warning" %} The job applies masking when it runs, and stores only the masked values. By default, `monitoringSettings.adaptiveMonitors.descriptiveStatistics` is `false`. In this case: * The profile masks the minimum, maximum, mean, and top and bottom values. * The job skips histograms, correlation, and rules that use stat variables. * The profile still shows row counts, null and empty counts, unique counts, and data types. To show the values, set this field to `true` on the job. The change applies only to new runs. Earlier runs stay masked. The values can contain sensitive data. {% endhint %} #### Cancel or delete runs and jobs To stop a run that is in progress, send a POST request to `/jobRuns/{jobRunId}/cancel`. ```bash curl -X POST \ "https://{your-collibra-host}/rest/dq/1.0/jobRuns/3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d/cancel" \ -u "{username}:{password}" ``` To delete a run, send a DELETE request to `/jobRuns/{jobRunId}`. This also deletes all run results, such as the profile, monitor, rule, and alert results. ```bash curl -X DELETE \ "https://{your-collibra-host}/rest/dq/1.0/jobRuns/3f2b8c1e-7a4d-4e9b-9c2f-1d5e6a7b8c9d" \ -u "{username}:{password}" ``` {% hint style="warning" %} You cannot delete a run that is in progress. Cancel the run first. If the run is in progress, the API returns a `409` response. {% endhint %} To delete a job, send a DELETE request to `/jobs/{jobName}`. ```bash curl -X DELETE \ "https://{your-collibra-host}/rest/dq/1.0/jobs/sales_customer_orders" \ -u "{username}:{password}" ``` #### Additional resources * [Jobs API reference](/api/references/data-quality/jobs.md) * [Job Runs API reference](/api/references/data-quality/job-runs.md) * [Export Data Quality & Observability (cloud) data with the Data Egress API](/tutorials/export-data-quality-and-observability-cloud-data-with-the-data-egress-api.md) * [Collibra REST API authentication with JSON Web Token](https://developer.collibra.com/tutorials/collibra-rest-api-authentication-with-json-web-token) * [About Data Quality jobs](https://productresources.collibra.com/docs/collibra/latest/Content/UnifiedDataQuality/co_create-data-quality-job.htm) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://developer.collibra.com/tutorials/create-and-run-data-quality-jobs-with-the-data-quality-rest-api.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.