Under the hood – Designing a critical data process with Collibra Workflows

Note   This scenario was part of the interactive Data Citizens – Under the Hood technical day 2019.

Prerequisites

  • Java SE Development Kit:
  • Eclipse:
    • Download link: Eclipse .
    • Select the Eclipse IDE for Java Developers version.
  • Flowable designer:
    1. Open Eclipse.
    2. In the menu bar, click HelpInstall New Software….
    3. In the Available Software window, click Add.
    4. Enter the required information:
      • Name: Flowable BPMN 2.0 designer.
      • Location: https://www.flowable.com/open-source/designer/update/.

Each attendee has an individual Collibra Data Governance Center environment. Find yours at:

To check the setup:

  1. Navigate to your environemnt.
  2. Sign in with:
    • Username: Admin
    • Password: uth19
  3. Go to CatalogData Sources.
  4. Check if you can see a list of columns.

Tip   You need a sample column to test your workflow during this workshop.

Use-case overview

Generate a workflow project and diagram

  1. Open Eclipse.
  2. Select FileNewOther….
  3. Search for Flowable.

  4. Select Flowable Project.
  5. Enter the project name: uth-dc19.
  6. Click Finish.
  7. Navigate to ~/src/main/resources/diagrams.
  8. Right-click the diagrams folder and select NewOther….
  9. Search for Flowable and choose Flowable Diagram.
  10. Enter the process Name: PIITagApproval.
  11. Open the PIITagApproval workflow using the Flowable Diagram Editor.

Deploying a workflow

After each step of the use-case workshop you can deploy your workflow and test the changes you have implemented. To deploy your workflow:

  • Go to Collibra Data Governance Center.
  • Go to SettingsWorkflowsWorkflow definitions.
  • Click Upload a file and select the BPMN file to deploy.
  • If the deployment was successful, search for the PII Tag Approval workflow.
  • In the Applies To section, select Asset.
  • In the upper-right part of the Applies To section, click Add.
  • Search for the Column asset type and click Save.
  • To enable the workflow, click the play button in the upper-right part of the page.
  • Check the status to verify the workflow is enabled.

Use-case step 1

During use-case step 1, you create the initial process structure using pools and swim lanes for the different stakeholders. Next, you add a user task for the Technical Steward to check if there is a PII tag and manually mark the column as containing personal identifiable information.

Add the process pool

  1. Go to the Flowable Palette.

  2. Go to the Container section and drag a Pool inside the empty canvas.
  3. Select the pool and go to the Properties panel.
  4. Enter the required information:
    • Id: tagApprovalPool
    • Name: Tag Approval
  5. Go to the Process panel.
  6. Enter the required information:
    • Id: piiTagApprovalWorkflow

      Note   The process ID must be unique at the Collibra DGC environment level.

      Warning   Uploading a workflow with the same process ID as an existing workflow in Collibra DGC will replace the existing workflow.

    • Name: PII Tag Approval

      Note   The name will be displayed as the workflow name in Collibra DGC.

      Warning   Collibra DGC does not accept two workflows with the same display name.

    • Namespace: https://www.collibra.com/apiv2

      Note   By referencing version 2 of the Collibra Java API, the workflow will be validated against V2 methods and deprecated V1 methods will trigger a method does not exist error when called.

    • Documentation: The contents of the field will become the description of the workflow in Collibra DGC.

Set up the swim lanes

Each pool contains by default one lane. To add a second one and configure them:

  1. Go to the Flowable Palette.
  2. Go the Container section and drag a Lane inside the pool.
  3. Configure the first lane:
    • Select the lane.
    • Go to the Properties panel.
    • Enter the required information:
      • Id: systemLane
      • Name: System
  4. Configure the second lane:
    • Select the lane.
    • Go to the Properties panel.
    • Enter the required information:
      • Id: technicalStewardLane
      • Name: Technical Steward

Add start and end events

  1. Go to the Flowable Palette.
  2. Go to the Start event container and drag a StartEvent inside the System lane.
  3. Select the start event.
  4. Go to the Properties panel.
  5. Select the Form tab.
  6. Add an approverUserExpression variable:

    Note   The variable can be changed from the worfklow definition page in Collibra Data Governance Center.

    1. Click New.
    2. Enter the required information:
      • ID: approverUserExpression
      • Name: Approver User Expression
      • Type: string
      • Default: role(Technical Steward)
      • Required: true
      • Readable: false
      • Writeable: true
  7. Go to the End event container and drag an EndEvent inside the System lane.

Add the Check Tag user task for the technical steward

  1. Go to the Flowable Palette.
  2. Go to the Task container and drag a UserTask inside the Technical Steward lane.
  3. Select the user task.
  4. Go to the Properties pane.
  5. Enter the task name: Check Tag.
  6. Go to the Documentation tab and enter a task description.

    Note   This description will be displayed in task bar in Collibra DGC to give the user more context and guidance on how to complete the task.

  7. Go to the Main config tab.
  8. Enter a Canditate users expression as defined in the start form variables: ${approverUserExpression}.

Deploy your workflow by following the steps from the Deploying a workflow section. Test the workflow by going to a column asset and clicking the PII Tag Approval button.

Use-case step 2

During use-case step 2, you add a script task to automatically check if the column has a PII tag or not and route the workflow accordingly. When the column is not tagged as PII you end the workflow. When the column is tagged as PII you create a task for the technical steward to manually add the personal identifiable informationflag to the column asset.

Ad get and filter tags script

  1. Remove the connection between the strat event and the user task.
  2. Go to the Flowable Palette.
  3. Go to the Task container and drag ScriptTask inside System lane.
  4. Select the script task.
  5. Go to the Properties pane.
  6. Enter a script task Name: Get and filter tags.
  7. Go to the Main config tab.
  8. Select the Groovy script language.
  9. Add the Get and filter tags script:
    loggerApi.info("Get & filter tags script started...")
    
    //Variables
    def tags = []
    def piiTagName = execution.getVariable("piiTagName")
    def isPII = false
    def assetUuid = item.id 
    
    tags = tagApi.getTagsByAssetId(assetUuid)
    loggerApi.info("Number of tags found: " + tags.size())
    
    //Loop over the tag list and check for PII tag
    for(tag in tags){
    	if(tag.getName().equals(piiTagName)){
    		isPII = true
    	}
    }
    
    loggerApi.info("isPII status: " + isPII)
    
    //Set variables to be used in workflow
    execution.setVariable("isPII",isPII)
    
    loggerApi.info("Get & filter tags script ended.")

Route the workflow when PII data is found or not

  1. Go to the Flowable Palette.
  2. Go to the Gateway container and drag an ExclusiveGateway to the System lane, after the Get and filter tags script task.

  3. Go to the End event container and drag a second EndEvent to the System lane, above the exclusive gateway.
  4. Connect the second end event with the exclusive gateway using a SequenceFlow.
  5. Select the sequence flow and enter the Name: No PII Found.
  6. Go to the Properties pane.
  7. Click the Main config tab, enter the Condition for PII not found: ${!isPII}.
  8. Connect the exclusive gateway with the user task using a SequenceFlow.
  9. Select the sequence flow and enter the Name: PII Found.
  10. Go to the Properties pane.
  11. In the Main config tab, enter the Condition for PII found: ${isPII}.
  12. Change the user task name and documentation to reflect the change in behavior of use-case step 2:
    1. Name: Add PII Attribute
    2. Documentation: Manually flag this asset as PII by adding the PII boolean attribute.

Deploy your workflow by following the steps from the Deploying a workflow section. Test the workflow by going to a column asset and clicking the PII Tag Approval button.

Use-case step 3

During use-case step 3, you add an approval user task, which is created when the column is flagged as PII. The technical steward needs to complete the approval user task by approving or rejecting and providing a reason. The reason is automatically saved to the comments section in the Collibra DGC. When the technical steward approves the PII tag, the steward must add the PII attribute manually to the corresponding column.

Create the approval user task

  1. Select the user task from Use-case step 1.
  2. Go to the Properties pane and change the name and documentation:
    • Name: Approve PII Tag
    • Documentation: Approve whether this asset with tag “${piiTagName}” has personal identifiable information. If you approve, add the PII boolean attribute to this column.
  3. Add the form used to complete the approval user task:
    1. Click the Form tab and click New to add a new form field.
    2. Add a comment box to allow the technical steward to provide a reason:
      • Id: commentBox
      • Name: Reason
      • Type: textarea
      • Required: true
      • Readable: true
      • Writeable: true
    3. Add a task button to clearly define what action needs to be done in the task:
      • Id: taskButton1
      • Name: Approve/Reject
      • Type: taskButton
      • Required: false
      • Readable: true
      • Writeable: true
    4. Add an approve button to the form:
      • Id: approveButton
      • Name: Approve
      • Type: button
      • Required: false
      • Readable: true
      • Writeable: true
    5. Add a reject button to the form:
      • Id: rejectButton
      • Name: Reject
      • Type: button
      • Required: false
      • Readable: true
      • Writeable: true

Save the reason to the comments section

  1. Go to the Flowable Palette.
  2. Go to the Task container and drag a ScriptTask inside the System lane.
  3. Select the script task.
  4. Go to the Properties pane.
  5. Enter a script task Name: Save Comment.
  6. Go to the Main config tab.
  7. Select the Groovy script language.
  8. Add the Add comment script:
    import com.collibra.dgc.core.api.dto.instance.comment.AddCommentRequest
    import com.collibra.dgc.core.api.model.ResourceType
    
    loggerApi.info("Add comment script started...")
    
    //Variables
    def commentBox = execution.getVariable("commentBox")
    def commentBoxContent = commentBox.toString()
    def assetUuid = item.id
    
    loggerApi.info("Comment box content: " + commentBoxContent)
    
    commentApi.addComment(AddCommentRequest.builder()
    	.baseResourceId(assetUuid)
    	.baseResourceType(ResourceType.Asset)
    	.content(commentBoxContent)
    	.build()
    )
    
    loggerApi.info("Add comment script ended.")
  9. Connect the Save Comment script to the end event using a SequenceFlow.

Deploy your workflow by following the steps from the Deploying a workflow section. Test the workflow by going to a column asset and clicking the PII Tag Approval button.

Use-case step 4

During use-case step 4, you trigger the workflow automatically when a tag is added ta column and the technical steward automatically receives a task to tapprove or reject the PII tag. If the technical steward approves the tag, the PII boolean attribute is added to the corresponding column.

Route the workflow based on approval or rejection

  1. Go to the Flowable Palette.
  2. Gto the Gateway container and drag an ExclusiveGateway to the system lane right after the save comment script task.

  3. Go to the End event container and drag a third EndEvent to the System lane, above the exclusive gateway.
  4. Connect the third end event with the exclusive gateway using a SequenceFlow.
  5. Select the sequence flow and go to the Properties pane.
  6. In the Main config tab, enter the Condition for when the tag is rejected: ${rejectButton}.
  7. Go to the Flowable Palette.
  8. Go to the Task container and drag a ScriptTask after the exclusive gateway.
  9. Connect the script task with the exclusive gateway using a sequence flow.
  10. Select the sequence flow and go to the Properties pane.
  11. In the Main Config tab, enter the condition for when the tag is approved: ${approveButton}.
  12. Connect the script task with the last end event using a SequenceFlow.

Automatically set the PII attribute using a script task

  1. Select the last added script task.
  2. Go to the Properties pane.
  3. Enter aNscript name: Set PII Attribute.
  4. Go to the Main config tab.
  5. Select the Groovy script language.
  6. Add the Set PII Attribute script:
    import com.collibra.dgc.core.api.dto.instance.attribute.AddAttributeRequest
    
    loggerApi.info("Set PII Attribute script started...")
    
    //Variables
    def assetUuid = item.id
    def piiAttributeTypeId = execution.getVariable("piiAttributeTypeId")
    def piiAttributeTypeUuid = string2Uuid(piiAttributeTypeId)
    
    //Set PII Attribute
    attributeApi.addAttribute(AddAttributeRequest.builder()
    	.assetId(assetUuid)
    	.typeId(piiAttributeTypeUuid)
    	.value(true)
    	.build()
    )
    
    loggerApi.info("Set PII Attribute script ended")

Automatically trigger the workflow when a tag is added to a column

  1. Go to Collibra Data Governance Center.
  2. Go to SettingsWorkflowsWorkflow definitions.
  3. Select the PII Tag Approval workflow.
  4. Go to the Start Event section.
  5. Select the Asset Tag Event.
  6. Click Save.

Deploy your workflow by following the steps from the Deploying a workflow section. Test the workflow by going to a column asset and clicking the PII Tag Approval button.