Manage Pipelines on SDC Edge

After designing edge pipelines in Data Collector and then deploying the edge pipelines to SDC Edge, you can manage the pipelines on SDC Edge. Managing edge pipelines includes previewing, validating, starting, stopping, resetting the origin, and monitoring the pipelines.

You can use the following methods to manage edge pipelines on SDC Edge:

Note: When you use Control Hub, you add edge pipelines to jobs and then start, stop, and monitor the jobs running on groups of Edge Data Collectors from the Control Hub UI. For more information, see Meet StreamSets Control Hub.

Using the Data Collector UI

When SDC Edge is accessible by the Data Collector machine and when the deployed edge pipeline has the Data Collector Edge URL property correctly configured, you can use the Data Collector UI to manage edge pipelines.

Managing edge pipelines includes previewing, validating, starting, stopping, resetting the origin, and monitoring the pipelines.
Note: When you preview, validate, or start an edge pipeline from Data Collector that has not yet been deployed to SDC Edge, Data Collector publishes and then starts the pipeline.
  1. Open the edge pipeline in the pipeline canvas.
  2. On the General tab, set the Data Collector Edge URL property to the URL of the SDC Edge instance that you want to manage the pipeline on.
    The default URL is http://localhost:18633.
  3. Use the Data Collector top toolbar to preview, validate, start, stop, and reset the origin for the edge pipeline running on SDC Edge - just as you manage standalone or cluster pipelines running on Data Collector.

    As an edge pipeline runs, the Data Collector UI displays the pipeline in Monitor mode so that you can monitor the edge pipeline status and metrics.

Sending Commands to SDC Edge

When SDC Edge is not accessible by the Data Collector machine, you must use the command line and REST API to manage edge pipelines on Data Collector Edge (SDC Edge).

Managing edge pipelines includes previewing, validating, starting, stopping, resetting the offset, and viewing the pipeline status and metrics.

Commands to Start a Pipeline at Startup

You can start an edge pipeline at the same time that you start SDC Edge.

Important: Be sure to start the receiving pipelines before starting the sending pipelines.

On the edge device, run the following command from the SDC Edge home directory:

bin/edge -start=<pipelineID>
For example, use the following command to start SDC Edge and an edge pipeline with the specified ID:
bin/edge -start=FileTailToHTTP74f418f8-c7d3-4d0c-8788-b6474f7160e0
If the edge pipeline uses parameters, use the following command:
bin/edge -start=<pipelineID> -runtimeParameters='{"<parameter_name1>":"<parameter_value1>",
"<parameter_name2":"<parameter_value2>"}'
For example, use the following command if the edge pipeline uses parameters to define the file path used by the File Tail origin:
bin/edge -start=FileTailToHTTP74f418f8-c7d3-4d0c-8788-b6474f7160e0 -runtimeParameters='{"filePath":"/tmp/sds.log"}'

Commands to Manage Pipelines After Start Up

After SDC Edge is started, use the following command to start another edge pipeline:
curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipeline/<pipelineID>/start
For example, use the following command to start a pipeline on a running SDC Edge:
curl -X POST http://localhost:18633/rest/v1/pipeline/fileTailControl74f418f8-cl2f5-4r7k-8790-b9854h1450e8/start
If the edge pipeline uses parameters, use the following command:
curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipeline/<pipelineID>/start -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"<parameter_name1>":"<parameter_value1>","<parameter_name2":"<parameter_value2>"}'
To list all edge pipelines deployed to SDC Edge, use the following command:
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipelines
To preview edge pipelines, return the status of the preview, return the preview data, and then stop the preview, use the following commands:
curl -X POST http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview/<previewerID>/status
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview/<previewerID>
curl -X DELETE http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview/<previewerID>
To validate edge pipelines, use the following commands:
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/validate
To manage running edge pipelines - including stopping, resetting the offset, and viewing the pipeline status and metrics - use the following commands:
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/status
curl -X POST http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/stop
curl -X POST http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/resetOffset
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/metrics