Auto-refresh sources

This feature allows Datasets to be automatically refreshed at the point of Workflow execution. The latest data will be fetched from the origin.

Source types that support Auto-refresh sources

Auto-refresh sources will work with External systems where the source table is still available. File uploads and dropzone files are excluded because once a Dataset is created, the origin file is not retained.

Valid data source types for Auto-refresh sources are JDBC, Azure BLOB, HDFS, Amazon S3, and SFTP.

Views will inherit the refresh setting from their base Dataset, so a Workflow that uses a View as source can also be configured to refresh that View's underlying Dataset on execution.

Enable Auto-refresh sources

There are three settings required in order to enable Auto-refresh sources:

When creating or editing a Dataset

These settings have a cumulative effect and you need all three to be enabled in order for data to be refreshed.
When all of these options are enabled, and the Workflow is executed, it will run with the latest data from the origin.

Provided your dataset is from a supported data source type:

  1. Go to Datasets.
  2. Find the Dataset or Add Dataset.
  3. On the Edit details page, tick the setting Allow auto-refresh.

Creating/editing a Workflow

On the Source step, tick Allow auto-refresh.

When executing the Workflow

Via the Run Workflow dialog or Schedule for execution:

  1. From the Workflow, press Run and tick Refresh sources.
  2. In the edit/ create page for a Schedule, tick Refresh sources.

Having this level of control allows for scenarios such as:

  • Ensuring the latest data is always used during scheduled or manual executions of a Workflow.
  • The creation of a Dataset that can never be automatically reloaded (e.g. definitive reference data).
  • The creation of a View that can never be automatically reloaded.
  • A one-off test execution of a Workflow that does not reload Datasets.
  • A Schedule that, regardless of the Workflow configuration, will never reload Datasets.