Pipeline Designer Tips

The Control Hub Pipeline Designer is closely based on the Data Collector pipeline configuration canvas. Some functional differences are described below:
Authoring Data Collector
When configuring a pipeline in Pipeline Designer, you select the authoring Data Collector to use. You can use the system Data Collector provided with Control Hub for exploration and light development. To perform explicit validation or data preview, use one of your registered Data Collectors as the authoring Data Collector.
Choose a registered Data Collector that is the same version as the Data Collectors that you intend to use to run the pipeline in production. Using a different Data Collector version can result in developing a pipeline that is invalid for production Data Collectors.
For example, if you use a more recent version of Data Collector for development than for production, you might include a stage, stage library, or stage functionality does not exist in the production Data Collector.
The registered Data Collector must meet certain requirements to be used as the authoring Data Collector. For more information, see Authoring Data Collectors.
Create a Pipeline
When you create a pipeline, you specify whether the pipeline will run on Data Collector or SDC Edge, and you can start with a blank canvas or an existing template.
Pipeline Designer provides several Data Collector and SDC Edge templates as sample pipelines. You can use them to familiarize yourself with Pipeline Designer and pipeline designs or you can use them as a basis for pipeline development.
Edit a Published Pipeline
When viewing a published pipeline, Pipeline Designer displays the pipeline in read only mode. The pipeline mode displays above the canvas as shown:
To edit a published pipeline, click the Edit icon: . The pipeline then enters Edit mode.
Select Multiple Stages
When editing a pipeline, you can select multiple stages in the pipeline canvas and then move or delete the selected stages. To select multiple stages in the canvas, select the Shift key and then click each stage.
Update Stage Libraries for the Pipeline
When editing a pipeline, you can use the Update Stage Libraries dialog box to update the stage libraries for multiple stages in the pipeline.
This allows you to update all necessary stage libraries at one time when you change the authoring Data Collector for the pipeline. When preferred, you can also change stage libraries individually by editing each stage.
The stage libraries that display for each stage depends on the authoring Data Collector selected for the pipeline. For example, if the authoring Data Collector has the MapR 5.0 and 5.1 stage libraries installed, then these are the stage libraries that display for a MapR FS destination or MapR FS File Metadata executor.
To update multiple stage libraries at one time, click the More icon (), then select Update Stage Libraries.
The Update Stage Libraries dialog box displays the stage name and type for each stage in the pipeline. On the right is the corresponding list of stage libraries for the stage that are available on the authoring Data Collector.
Update the stage libraries as needed, then click Update to save your changes.
Work with Versions
When you have multiple versions of a pipeline, Pipeline Designer indicates the version of the pipeline that you are viewing. You can click the pipeline version to select a different version to view, as follows:

When you first create a pipeline in Pipeline Designer, the pipeline is in a draft state so that you can add and configure stages in the pipeline. Draft pipelines are versioned as <version>-DRAFT, for example, v1-DRAFT or v2-DRAFT. When you finish editing a draft pipeline, publish the pipeline to indicate that the pipeline is final and is available to be added to a job.

Published pipelines are versioned by number, for example, v1 or v2. You cannot edit published pipelines. When viewing a published version, you can click edit to create a new version based on the pipeline in the canvas.

For more information about working with versions, see Comparing Pipeline Versions.
Publish a Pipeline
When you have completed work on a pipeline, you publish the pipeline. After you publish the pipeline, you can create and run jobs based on the pipeline.
Use the Publish icon to publish a valid pipeline: . Enter a commit message stating what changed in this pipeline version so that you can track the commit history of the pipeline.
After you publish the pipeline, it enters read only mode and can be used to create a job.
Create a Job
After you publish a pipeline, you can create a job.
You can create a job using the Create Job icon in Pipeline Designer: . Or, you can create a job from the Jobs view.
Data Preview Requirement
You can preview data when the pipeline uses a registered Data Collector as the authoring Data Collector.
If the pipeline uses the system Data Collector or a selected registered Data Collector that is not accessible, the Preview Pipeline icon () is disabled.
Validation Requirement
You can perform explicit validation when the pipeline uses a registered Data Collector as the authoring Data Collector.
Use the Validate icon to perform explicit validation: .
Compare Versions
When there are multiple versions of the pipeline, you can use the Compare with Previous Version icon () to compare the current version with a previous version.

In the Comparison window, you can select different pipeline versions in each window and navigate through the pipelines.

View Pipeline History
You can view the history of the pipeline to see the commit history and the tags associated with the pipeline.
Use the History icon to view the pipeline history: .
For more information about working with the pipeline history, see Pipeline History.
Deleting a Pipeline Version
When a pipeline has multiple versions, you can delete the pipeline version that you are viewing. Or, you can delete all versions of the pipeline. Use the Delete Pipeline option available in the More menu:
You can only delete pipelines or pipeline versions when they are not used in jobs.