Pipeline Validation

StreamSets Cloud automatically validates pipelines as you design them and when you preview them.

Each validation type verifies different information:

Design validation
As you design a pipeline, StreamSets Cloud verifies that each stage is completely configured. The pipeline canvas displays errors when a stage is unconnected or when a required stage property has not been configured.
Preview validation
When you preview a pipeline, StreamSets Cloud verifies that all configured values are valid and verifies that the pipeline can run as configured. The pipeline canvas displays errors when a stage has an invalid value or cannot connect to an external system.

For example, design validation verifies that you have entered all of the required properties for the Azure Data Lake Storage Gen2 origin. Preview validation verifies that you have entered valid values by attempting to connect to the specified storage account in Azure.

StreamSets Cloud displays errors found by both design and preview validation as error icons on stages, with detailed messages displayed in a tooltip. For example, the following image displays an error icon for the Azure Data Lake Storage Gen2 origin found during design validation:

The tooltip for the error icon displays the error messages, as follows:

For errors found by design validation, StreamSets Cloud also displays the number of errors encountered in each configuration tab when you select a stage in the pipeline canvas. For example, the following image shows that design validation found a total of three errors for the Azure Data Lake Storage Gen2 origin, two errors in the Data Lake tab and one error in the Files tab: