Pipeline Designer Tips

The Control Hub Pipeline Designer is closely based on the Data Collector pipeline configuration canvas. Some functional differences are described below:
Authoring Data Collector
When configuring a pipeline or pipeline fragment in Pipeline Designer, you must select the authoring Data Collector to use. You can select any accessible Data Collector, either the system Data Collector if available or a registered Data Collector. However, to perform explicit validation or data preview, you must select a registered Data Collector as the authoring Data Collector.
Always select an authoring Data Collector that is the same version as the execution Data Collector that you intend to use to run the pipeline in production. Using a different Data Collector version can result in developing a pipeline that is invalid for the production Data Collector.
For example, if you use a more recent version of Data Collector for development than for production, you might include a stage, stage library, or stage functionality that does not exist in the production Data Collector.
The registered Data Collector must meet certain requirements to be used as the authoring Data Collector. For more information, see Authoring Data Collectors.
Create a Pipeline or Fragment
When you create a pipeline or pipeline fragment, you specify whether the pipeline will run on Data Collector or SDC Edge. For pipelines, you can start with a blank canvas or an existing template.
Pipeline Designer provides several Data Collector and SDC Edge templates as sample pipelines. You can use them to familiarize yourself with Pipeline Designer and pipeline designs or you can use them as a basis for pipeline development.
Edit a Published Pipeline or Fragment
When viewing a published pipeline or pipeline fragment, Pipeline Designer displays the pipeline or fragment in read-only mode. The mode appears above the canvas as shown:
To edit a published pipeline or fragment, click the Edit icon: . The pipeline or fragment then enters Edit mode.
Select Multiple Stages
When editing a pipeline or pipeline fragment, you can select multiple stages in the pipeline canvas and then move or delete the selected stages. To select multiple stages in the canvas, press the Shift key and then click each stage.
Update Stage Libraries for the Pipeline
When editing a pipeline or pipeline fragment, you can use the Update Stage Libraries dialog box to update the stage libraries for multiple stages in the pipeline or fragment.
This allows you to update all necessary stage libraries at one time when you change the authoring Data Collector for the pipeline or fragment. When preferred, you can also change stage libraries individually by editing each stage.
The stage libraries that display for each stage depends on the authoring Data Collector selected for the pipeline or fragment. For example, if the authoring Data Collector has the MapR 5.0 and 5.1 stage libraries installed, then these are the stage libraries that display for a MapR FS destination or MapR FS File Metadata executor.
To update multiple stage libraries at one time, click the More icon (), then select Update Stage Libraries.
The Update Stage Libraries dialog box displays the stage name and type for each stage in the pipeline or fragment. On the right is the corresponding list of stage libraries for the stage that are available on the authoring Data Collector.
Update the stage libraries as needed, then click Update to save your changes.
Work with Versions
When you have multiple versions of a pipeline or pipeline fragment, Pipeline Designer indicates the version of the pipeline or fragment that you are viewing. You can click the pipeline or fragment version to select a different version to view, as follows:
For more information about working with versions, see Version History.
Run a Test of a Draft Pipeline
When editing a pipeline, you can perform a test run of the draft pipeline to quickly test the pipeline logic. You can perform a test run of a draft version of a fully configured pipeline.
Publish a Pipeline or Fragment
When you have completed work on a pipeline or fragment, you publish the pipeline or fragment. Publish a pipeline to create and run jobs based on the pipeline. Publish a fragment to make it available for testing or use in pipelines.
Use the Publish icon to publish a valid pipeline or fragment: . Enter a commit message stating what changed in this version so that you can track the commit history of the pipeline or fragment.
After you publish a pipeline, it enters read only mode and can be used to create a job. After you publish a fragment, it enters read only mode and can be included in pipelines.
Create a Job
After you publish a pipeline, you can create a job.
You can create a job using the Create Job icon in Pipeline Designer: . Or, you can create a job from the Jobs view.
Data Preview Requirement
You can preview data when the pipeline uses a registered Data Collector as the authoring Data Collector.
If the pipeline uses the system Data Collector or a selected registered Data Collector that is not accessible, the Preview Pipeline icon () is disabled.
Validation Requirement
You can perform explicit validation when the pipeline uses a registered Data Collector as the authoring Data Collector.
Use the Validate icon to perform explicit validation: .