Using Fragments in Pipelines

After a pipeline fragment is published, you can use it in the pipeline. Here's how a fragment works in a pipeline...

Generally, you can use a fragment like any other stage. The fragment displays in the stage library as a fragment stage type. When appropriate, a fragment category also displays in the Pipeline Creation help bar. You can use multiple fragments in a pipeline and reuse the same fragment in a pipeline.

After you add a fragment to the canvas, it displays as a fragment stage by default. The fragment stage input and output streams represent the input and output streams of the fragment logic. You can use the Expand Fragments icon to view the stages in the fragment and to verify the fragment logic. The Expand Fragments icon () is located on the right side of the canvas by the navigation icons.

If you have multiple fragments in the pipeline, the Expand Fragments icon expands all fragments in the pipeline. After being used, the Expand Fragments icon becomes the Collapse Fragments icon (), which you can use to hide the fragment details.

You cannot edit a fragment when designing a pipeline. To navigate away from the pipeline to edit the fragment, click the fragment name in either of the following locations in the Properties panel:

Using Fragment Versions

If a fragment has multiple versions, be sure to select the correct version when you design a pipeline. A pipeline uses the selected fragment version until manually changed.

To avoid invalidating existing pipelines, fragments in pipelines do not automatically update to the latest version. To change the fragment version that a pipeline uses, edit the pipeline and update the fragment version manually.

You can see the pipelines that use a fragment version on the Pipelines tab of the fragment details pane, as shown below:

You can click a pipeline name to quickly navigate to the pipeline.

To view or change the fragment version used in a pipeline, select the fragment, then configure the Fragment Commit/Tag property as needed:

Stream Order

The stream order of a fragment stage in a pipeline follows the order and location of the stages in the fragment - from left to right across the primary branch, then across any remaining upper and lower branches, with all event streams displaying below data streams.

For example, the following fragment stage has four output streams:

The expanded fragment looks like this:

The stream order follows the primary branch, JSON Parser to Hadoop FS, then returns to the remaining Aggregator branch. It also lists the event streams after the data streams. So the resulting stream order for this fragment is as follows:
  • Input stream - JSON Parser input stream.
  • Output stream 1 - Stream Selector output stream 2.
  • Output stream 2 - Aggregator output stream.
  • Output stream 3 - Hadoop FS event stream.
  • Output stream 4 - Aggregator event stream.