Fragments Tips and Best Practices

Use the following tips and best practices when designing and working with pipeline fragments.

Design Best Practices

Use identifiable fragment names and descriptions
Make sure that the pipeline fragment name and description offers enough information so the developer can easily select the fragment to use from a larger list of pipeline fragments.
Use detailed commit messages to differentiate between fragment versions
Use detailed commit messages to help a data engineer determine the version of a fragment to use.
Optionally, use pipeline labels or tags to codify fragment versions
When you have a version that is ready for testing or production, you can use pipeline labels or pipeline tags to clearly define the fragment.
Select the execution engine and execution mode carefully
The execution engine and fragment execution mode determines the stages you can include in the fragment and the pipelines where you can use the fragment.
Once you select an execution engine, Data Collector or Data Collector Edge, you cannot change it. You can configure the fragment execution mode as needed, but changing it can make the latest version of the fragment invalid for any pipelines that use the previous version.
For more information, see Execution Engine and Execution Mode.
Choose an appropriate authoring Data Collector
When configuring a pipeline fragment, choose an authoring Data Collector that is the same version as the Data Collectors that you intend to use to run the pipelines that use the pipeline fragment. Using a different Data Collector version can result in developing a pipeline fragment that is invalid for execution Data Collectors.
You must use a Data Collector version 3.2.0.0 or later.
Evaluate the number of input and output streams to use
Before publishing a pipeline fragment, carefully consider the number of input and output streams that the fragment uses. After you publish a pipeline fragment, you cannot change the number of input or output streams in subsequent fragment versions.
This prevents invalidating pipelines when updating the fragment version that they use. For more information, see Fragment Input and Output.
Use a Dev Identity processor to create additional input or output streams
You can use the Dev Identity processor to create additional input or output streams in the fragment, creating a corresponding input or output stream in the resulting fragment stage. For more information, see Creating Additional Streams.
Use data preview to test fragment logic
You can use data preview to help design and test the processing logic of a fragment. You can use a test origin to provide source data for data preview. This can be especially useful when working with a fragment that does not include an origin. When the fragment contains an origin, you can also use the origin to provide source data for the preview.
Use pipeline explicit validation to validate fragments
At this time, you cannot use explicit validation when designing a fragment. Best practice is to validate the fragment in a test pipeline before using the fragment in a production pipeline. For more information, see Explicit Validation.
After publishing a new version of a fragment, review the pipelines that use previous versions of the fragment
You can view the pipelines that use each version of a fragment in the fragment details pane. After publishing a new version of a fragment, you might want to check the pipelines that use earlier versions to see if they should use the updated fragment version.
Set appropriate prefixes when adding fragments to pipelines
Consider how you will reuse the fragment and whether you want the values of runtime parameters to be the same or different in each instance of the fragment:
  • To use the same values for the runtime parameters in all fragments, enter the same prefix for each fragment.
  • To use unique values for the runtime parameters in each fragment, enter a unique prefix for each fragment.

Usage Tips

Select the correct version of the pipeline fragment
Use the Fragment Commit/Tag stage property to select the version of the fragment to use in the pipeline. For more information, see Using Fragment Versions.
If needed, override runtime parameter values that are defined in the fragment
The pipeline inherits runtime parameters defined in the fragment. You can override the default values for those runtime parameters. You can also delete the parameter if you end up removing the fragment from the pipeline.
For more information about working with runtime parameters in pipeline fragments, see Runtime Parameters.
Use data preview or explicit validation to verify and validate fragment processing
When using a fragment in a pipeline, you can use pipeline validation and data preview to verify that the fragment processes data as expected.