What is a Pipeline?

A pipeline describes how data flows from origin to destination systems and how the data is processed along the way.

You can use the following stages in pipelines:
An origin stage represents an origin system. A pipeline can include a single origin stage.
A processor stage represents a type of data processing that you want to perform. A pipeline can include multiple processor stages.
A destination stage represents a destination system. A pipeline can include multiple destination stages.
An executor stage triggers a task when it receives an event. A pipeline can include multiple executor stages.

When you design a pipeline, you can also include development stages to provide sample data and generate errors to test error handling. You can preview the data to determine how stages alter the data through the pipeline.

When you start a pipeline, StreamSets Cloud runs the pipeline until you stop the pipeline.

While the pipeline runs, you can monitor the pipeline to verify that the pipeline performs as expected. You can also define metric and data rules and alerts to let you know when certain thresholds are reached.