Pipeline Fragments Overview

A pipeline fragment is a stage or set of connected stages that you can use in pipelines. Use pipeline fragments to easily add the same processing logic to multiple pipelines and to ensure that the logic is used as designed.

For example, to use the same set of processing logic for all pipelines that process log data, you create a pipeline fragment that includes the processors you want to use, such as the Field Merger and Field Remover to organize record fields. After you publish the fragment, you add the fragment to pipelines as needed, reusing the fragment processing logic.

Design pipeline fragments in the Control Hub Pipeline Designer. You cannot design fragments in the Data Collector UI.

You can use published fragments in pipelines. When you use a fragment in a pipeline, it displays in the canvas as a single fragment stage. You can expand the fragment stage to view the stages and stage configuration defined in the fragment, but you cannot edit the fragment from within the pipeline. This ensures that the fragment processing logic is used as designed.

You can edit and update published pipeline fragments. To prevent the invalidation of pipelines that use fragments, pipelines do not automatically inherit changes to published fragments. To configure a pipeline to use a different version of a fragment, you must edit the pipeline and update the fragment version.

When viewing a fragment, you can also view the list of pipelines that use the fragment. After you publish updates to a fragment, you can easily navigate to the various pipelines that use the fragment to determine whether they should use the latest version of the fragment. Then, you can update the fragment version that the pipelines use, as needed.