Pipeline Maintenance

Overview

After creating and configuring a pipeline, you run the pipeline to start the flow of data from the origin to destination systems. Each pipeline runs until you stop the pipeline.

You can export a pipeline and then import that pipeline into StreamSets Cloud. You might want to export a pipeline to create a backup or to share the pipeline with another user.

You can delete a pipeline when you no longer need the pipeline.

Running a Pipeline

When you run a pipeline, you start the flow of data from the origin to destination systems. Each pipeline runs until you stop the pipeline.

A running pipeline is read-only. You must stop the pipeline to edit the pipeline.

When you stop a pipeline, most origins maintain the offset to note where they stop reading. When you run the pipeline again, you configure the pipeline run to start from an offset saved with a previous run or to start from the beginning to read all available data.

  1. Open the pipeline in the pipeline canvas.
    1. In the left navigation pane, click the Pipelines icon: .
    2. Click the name of the pipeline that you want to run.
  2. In the toolbar above the pipeline canvas, click the Run icon: .
  3. In the Run Configuration dialog box, select the offset that you want the origin to start reading from.

    For the initial run, you can start from the beginning only.

  4. Click Run Pipeline.

    It takes a few minutes for the pipeline to be deployed and started.

Stopping a Pipeline

Stop a pipeline when you want to stop processing data for the pipeline.

You also must stop a pipeline to edit the pipeline. A running pipeline is read-only.

  1. Open the pipeline in the pipeline canvas.
    1. In the left navigation pane, click the Pipelines icon: .
    2. Click the name of the pipeline that you want to stop.
  2. In the toolbar above the pipeline canvas, click the Stop icon: .
    When the pipeline successfully stops, you can edit the pipeline as needed.

Exporting a Pipeline

Export a pipeline to create a backup or to share the pipeline with another StreamSets Cloud user.

When you export a pipeline, StreamSets Cloud generates a JSON file named after the pipeline, as follows: <pipeline name>.json.

When pipeline stages contain secrets such as user names and passwords, the secret values are not exported.

  1. In the left navigation pane, click the Pipelines icon: .
  2. Hover over the pipeline that you want to export, click , and then click Export Pipeline.

    StreamSets Cloud exports the pipeline to your default downloads directory.

Importing a Pipeline

Import a pipeline to restore a backup file or to use a pipeline shared by another StreamSets Cloud user.

You can import a pipeline JSON file that has been exported from StreamSets Cloud. You cannot import a pipeline that has been exported from another StreamSets product.

When you import a pipeline, StreamSets Cloud uses the name of the JSON file as the pipeline name. If an existing pipeline uses the same name, you cannot import the file unless you rename the file or rename the existing pipeline.

  1. In the left navigation pane, click the Pipelines icon: .
  2. Click Import Pipeline.
  3. Browse and select the pipeline JSON file, and then click Open.
    Note: If the imported pipeline contains secrets such as user names and passwords, you must enter the secret values again. Secret values are not exported.

Deleting a Pipeline

You can delete a pipeline when you no longer need the pipeline.

Deleting a pipeline is permanent. To keep a backup, export the pipeline before you delete it.

  1. In the left navigation pane, click the Pipelines icon: .
  2. Hover over the pipeline that you want to delete, click , and then click Delete.
  3. Click OK to confirm the deletion.