Data Quality user documentation | Realtime Workflows

Realtime Workflows

A Realtime Workflow is a series of connected steps including Transform and Validate, much like your existing Aperture Workflows. However, rather than processing batches of data loaded from a file or database, a Realtime Workflow processes a single record of data submitted via an API request then outputs an API response.

The benefit of a Realtime Workflow is to move your data quality checks and transformation upstream, to the left, where the data is being collected. Wherever possible, preventing bad data from being submitted is the better option rather than identifying data issues and rectifying them later.

For example, a company is signing up new memberships at their branches via an online form. The company wants to ensure the details being captured are complete, consistent and meet the stringent business rules that have already been built out and are applied in an existing Validate Workflow. Once the form is submitted, the values are checked in real time and a response will return any errors that can be amended by the submitter before the new record is created in their CRM.

This feature is an Aperture Data Studio license add on, which can be useful for customers who want to process data, in real time, as and when it's received, rather than waiting for a scheduled batch process at the end of the day.

Workflow steps supported

Workflow step	Description
Export	Allows exporting of record to only an external database (JDBC).
Output	Returns the response to the request.
Request	This step is used to configure the expected JSON structure of your data.
Search Duplicate store	Allows searching for a matching record in the Duplicate store.
Split	Applies a True/False filter to the input, splitting it into passing and failing rows.
Transform	This step allows you to add/remove various Functions and arrange, rename or exclude columns.
Update Duplicate store	Allows inserting or updating a record in an existing Duplicate store.
Validate	Allows use of business rules to measure the quality, completeness and accuracy of your data.

Before you begin:

Ensure your license has the Realtime Workflows addon (top-right menu then About).

API Key Authentication

As you’re calling the Workflow via an API, you have the option to secure this using an API Key. Do the following to enable API Key authentication for your Realtime Workflows:

Check the “Require API key for all request” checkbox when creating your Realtime Workflow.
Next create a new API Key and select the Realtime Workflows that can be executed using this key.
Copy the generated key and paste it in the authorization header of your request. Your authorization header should look like this:

    "Authorization": "<Your API Key>"

Creating a Realtime Workflow

Click on “Create new Realtime Workflow” and fill in the details, including the External Label, which will be used in the API request URL.
Select ‘Require API key for all requests’ to prevent unauthorised requests
This Workflow editor is the same as the standard Workflow editor, but it only shows the supported steps.
Start with the Request step, where data from the API request is received, and define the expected JSON structure for your data here. Each field defined is interpreted as columns while the values for each field will be the values for the columns.

Note that you can enter multiple records in the Request step with different examples, which helps when building out the Workflow, but the actual API requests should only contain a single record per request.

Add a Transform or Validate step to manipulate or validate the data as needed.
Connect an Output step to signify what should be returned as a response to the API request.

Using a Realtime Workflow

Once your Workflow is set up, you can now make an API request.

To run draft Workflows:
POST http://<hostname>:<port>/dataapi/<tenancy id>/<external label of environment>/<external label of workflow>/draft

To run published Workflows:
POST http://<hostname>:<port>/dataapi/<tenancy id>/<external label of environment>/<external label of workflow>

The exact URL for your Workflow can be found in the Request step settings.
If you’ve enabled API key authentication, remember to paste the generated API Key in the authorization header of your request.

FAQ

Top-right user menu > About, look for add-on ‘Realtime Workflow’. If it is not shown, then reach out to your Experian contact or contact support to discuss.

Reach out to your Experian contact or contact support to discuss.

No, you should submit a single record per API request.

Make a POST API request using an application like Postman, a free, easy-to-use third-party application you can use to send API calls. Alternatively, you can also enter multiple test records in the Request step to test out all the different permutations.

Create a Ruleset containing the rules and use this in your Validate step so that any future rule updates can be applied everywhere they are used.

If the Realtime Workflow is depending on any associated objects, i.e. a Ruleset, re-usable Function, Find duplicates rules, you will need to either edit the Realtime Workflow or hit the Refresh action button on Workflow list screen to ensure these dependency changes are picked up.

Use the Transform step with the Functions Contains Match or Replace matches.

Yes, attach the Export step to export the record to an external database.

Yes, but it is difficult to give any exact numbers as it will depend on factors like your setup and other jobs being run concurrently. To give some indication, tests have processed thousands of requests per minute from hundreds of requestors.

Was this helpful?

Next: Use the REST API

Aperture Data Studio v2

Extend Data Studio functionality

Previous topic:
Monitor data