Version History
A typical pipeline or pipeline fragment development cycle involves iterative changes. The pipeline repository maintains the version history for each pipeline and fragment.
When you first create a pipeline or fragment in Pipeline Designer, it starts in a draft state so that you can configure the object. Draft objects are versioned as <version>-DRAFT, for example, v1-DRAFT or v2-DRAFT. When you finish editing a pipeline, publish the pipeline to indicate that it is final and is available to be added to a job. Publish a fragment to make it available for testing or use in pipelines.
Published pipelines and fragments are versioned by number, for example, v1 or v2. You cannot edit published objects. When viewing a published version, you can click edit to create a new version based on the version that displays in the canvas.
To view the history of a pipeline, select a pipeline in the Pipelines view and then click
the History icon: . To view the history of a fragment, select the
fragment in the Pipeline Fragments view and then click the
History icon.
The version history displays the following information for the selected object:
- Pipeline commits
- Displays each version of the pipeline or fragment along with the version number
and commit message. Within Control Hub, you can view and compare versions.
You can add any pipeline version to a job. Or if you discover that you must roll back to a previous pipeline version, you can edit an existing job to use the previous version. You can get a previous version of a pipeline and then continue editing the previous version in Pipeline Designer.
Similarly, you can add any version of a pipeline fragment to a pipeline. And you can edit the pipeline to use a different version when needed. You can also edit a previous version of a fragment to create a new version.
If you edit and then publish a previous version of an object, Control Hub updates the minor version number rather than the major version number.
- Pipeline tags
- Displays all tags defined for the pipeline or fragment. Assign tags to pipeline or fragment versions to differentiate between the versions.
You might want to add tags to versions to mark release points or to separate
development and production environments.
For example, when you finish developing the first version of a fragment, you might add a "Ready to Test" tag to the version. Then, when the fragment passes muster, you can use a "Ready for Use" tag. These tags clearly inform the data engineer which fragments are ready to be tested in development pipelines or used in production pipelines.
Comparing Versions
You can compare two versions of a pipeline or pipeline fragment. When you compare versions, Control Hub presents the versions side-by-side in the canvas, highlighting the differences between the versions. You can compare the pipeline or fragment configuration properties and all of the stages in the pipeline or fragment.
Working with Pipeline Tags
You can create and assign tags to versions of pipelines and fragments to differentiate between the versions. You can also remove tags from versions of pipelines and fragments.