Version History

A typical pipeline or pipeline fragment development cycle involves iterative changes. The pipeline repository maintains the version history for each pipeline and fragment.

When you first create a pipeline or fragment in Pipeline Designer, it starts in a draft state so that you can configure the object. Draft objects are versioned as <version>-DRAFT, for example, v1-DRAFT or v2-DRAFT. When you finish editing a pipeline, publish the pipeline to indicate that it is final and is available to be added to a job. Publish a fragment to make it available for testing or use in pipelines.

Published pipelines and fragments are versioned by number, for example, v1 or v2. You cannot edit published objects. When viewing a published version, you can click edit to create a new version based on the version that displays in the canvas.

To view the history of a pipeline, select a pipeline in the Pipelines view and then click the History icon: . To view the history of a fragment, select the fragment in the Pipeline Fragments view and then click the History icon.

The version history displays the following information for the selected object:

Pipeline commits
Displays each version of the pipeline or fragment along with the version number and commit message. Within Control Hub, you can view and compare versions.

You can add any pipeline version to a job. Or if you discover that you must roll back to a previous pipeline version, you can edit an existing job to use the previous version. You can get a previous version of a pipeline and then continue editing the previous version in Pipeline Designer.

Similarly, you can add any version of a pipeline fragment to a pipeline. And you can edit the pipeline to use a different version when needed. You can also edit a previous version of a fragment to create a new version.

If you edit and then publish a previous version of an object, Control Hub updates the minor version number rather than the major version number.

For example, you have a pipeline with three versions. You get version 2 of the pipeline, edit the pipeline in Pipeline Designer, and then publish the pipeline again. Control Hub versions the current pipeline as v2.1. So you now have four versions of the pipeline, with version 2.1 of the pipeline marked as the current version:
  • v2.1
  • v3
  • v2
  • v1
Pipeline tags
Displays all tags defined for the pipeline or fragment. Assign tags to pipeline or fragment versions to differentiate between the versions. You might want to add tags to versions to mark release points or to separate development and production environments.

For example, when you finish developing the first version of a fragment, you might add a "Ready to Test" tag to the version. Then, when the fragment passes muster, you can use a "Ready for Use" tag. These tags clearly inform the data engineer which fragments are ready to be tested in development pipelines or used in production pipelines.

Similarly, you might use a "Ready to Deploy" tag for a pipeline when it is production-ready. The following image displays the history of a sample pipeline with three versions and a "Ready to Deploy" tag applied to the latest version:

Comparing Versions

You can compare two versions of a pipeline or pipeline fragment. When you compare versions, Control Hub presents the versions side-by-side in the canvas, highlighting the differences between the versions. You can compare the pipeline or fragment configuration properties and all of the stages in the pipeline or fragment.

  1. To compare pipeline versions, in the Navigation panel, click Pipeline Repository > Pipelines.
    To compare fragment versions, in the Navigation panel, click Pipeline Repository > Pipeline Fragments.
  2. To compare the two most recent versions, hover over a pipeline or fragment and then click the Compare with Previous Version icon: .
    Or to compare any two versions, select the pipeline or fragment in the list and then click the History icon: . Select the two versions that you want to compare, and then click the Compare with Previous Version icon: .

    Control Hub displays the two versions side-by-side in the comparison window, highlighting any differences with a deleted, added, or updated icon, as follows:

    You can drill into the details of each stage, and compare the configuration of each stage between the versions.

Working with Pipeline Tags

You can create and assign tags to versions of pipelines and fragments to differentiate between the versions. You can also remove tags from versions of pipelines and fragments.

  1. To create tags for pipeline versions, in the Navigation panel, click Pipeline Repository > Pipelines.
    To create tags for fragment versions, in the Navigation panel, click Pipeline Repository > Pipeline Fragments.
  2. Select a pipeline or fragment in the list and then click the History icon: .
  3. Under Pipeline Commits, hover the version that you want to create a tag for, and then click the Create Tag icon: .
  4. Enter a name and message for the tag.
    In the tag message, state the reason for adding the tag to this version.
  5. Click Create Tag.
    Control Hub displays the tag next to the version.
  6. To view all tags created for the selected pipeline, click the Pipeline Tags tab.
    To delete a tag from the selected pipeline, on the Pipeline Tags tab, click the Remove Tag icon ().
  7. To view all tags created for the selected fragment, click the Pipeline Fragment Tags tab.
    To delete a tag from the selected fragment, on the Pipeline Fragment Tags tab, click the Remove Tag icon.