Page the results

The Output Module also supports paging the results for the root node of the query. You can specify an offset and a length parameter to limit the results to a subset of the complete list.

JSON key	Default value	Description
displayStart	0	The offset in the list of results. This offset is a zero-based index value.
displayLength	-1	The maximum total number of results to return. A negative value means unlimited.
maxCountLimit	-1	The maximum count value. A count of all records can lead to performance problems. When paging, you can limit the max count to this value. Passing 0 means no count is done.

JSON

{
  "ViewConfig": {
    "displayStart": 10,
    "displayLength": 5,
    "maxCountLimit": 10000,
    "Resources": {
      "Community": {
        "Id": { "name": "communityId" },
        "Name": { "name": "communityName" },
        "Description": { "name": "communityDescription" },
        "Order": [ { "Field": { "name": "communityName", "order": "ASC" } } ]
      }
    }
  }
}

YAML

The example query above selects page 3 of all communities, with five results per page.

• Paged results should always be sorted, otherwise the results might seem inconsistent from page to page.

• The paged results list is recalculated upon each request.

• All entities that have been added or removed will appear/disappear from the list, modifying the indexes of the elements in the results list.

• The Collibra Console allows limiting the number of results returned by queries. The values range from 10,000 to 100,000. If enabled, and the limit is set, then:

  • The default displayLength value (-1) is overwritten by the limit set through the console.

  • If the displayLength set in the ViewConfig/TableViewConfig is larger than the limit value set in the Collibra Console, an exception is thrown.

Cursor-based pagination

Offset-based pagination does not scale for a big data set. The bigger the offset, the slower the response time. For a bulk export of many assets, it's better to use cursor-based pagination:

• Run the initial query sorting by ID. This is the recommended option to achieve maximum performance. Use a medium page size, such as 10,000.

• Read the results and keep the last ID value.

• Pass this ID as a filter to the next query.

• Repeat the process until the number of returned results is less than the page size. This means you retrieved all data.

{
  "ViewConfig": {
    "displayLength": 10,
    "Resources": {
      "Asset": {
        "Id": { "name": "assetId" },
        "Signifier": { "name": "assetName" },
        "Order": [
          {
            "Field": {
              "name": "assetId",
              "order": "ASC"
            }
          }
        ],
        "Filter": {
          "Field": {
            "name": "assetId",
            "operator": "GREATER",
            "value": "00000000-0000-0000-0000-000000005032"
          }
        }
      }
    }
  }
}