Design Edge Pipelines

Edge pipelines run in edge execution mode. You design edge pipelines in Data Collector.

After designing edge pipelines, you deploy the edge pipelines to SDC Edge installed on an edge device. You then run the edge pipelines on SDC Edge.

You can design the following types of edge pipelines:

Edge sending pipelines
An edge sending pipeline uses an origin specific to the edge device to read local data residing on the device. The pipeline can perform minimal processing on the data before sending the data to a Data Collector receiving pipeline.
Optionally, you can also design an edge sending pipeline to monitor the data being processed and then send data to an edge receiving pipeline running on the same SDC Edge. The edge receiving pipeline acts on the data to control the edge device.
Edge receiving pipelines
An edge receiving pipeline listens for data sent by another pipeline running on Data Collector or on SDC Edge and then acts on that data to control the edge device.
An edge receiving pipeline includes the corresponding origin to read from the destination in the pipeline that sends the data. For example, if the sending pipeline writes to an HTTP Client destination, then the edge receiving pipeline uses an HTTP Server origin to read the data.

Edge pipelines support a limited number of origins, processors, destinations, error record handling options, and data formats. Edge pipelines do not currently support any executors.

Origins

Edge pipelines support a limited number of origins. Origins function the same way in edge pipelines as they do in other pipelines. However, some origins have limitations in edge pipelines as noted below.

Edge pipelines support the following origins:

Supported Origin Limitations
Dev Random Record Source None
Dev Raw Data Source None
Directory Edge pipelines do not support multithreaded processing.

In an edge pipeline, the Directory origin always creates a single thread to read the files even if you configure it to use multiple threads.

File Tail

In edge pipelines, the File Tail origin can read a single set of files.

If you configure multiple sets of files for the origin, the origin reads only the files configured in the first set.

HTTP Client

In edge pipelines, the HTTP Client origin does not support batch processing mode, pagination, or OAuth2 authorization.

HTTP Server Edge pipelines do not support multithreaded processing.

In an edge pipeline, the HTTP Server origin always creates a single thread to read the files even if you configure it to use multiple threads.

MQTT Subscriber Edge pipelines that use MQTT stages require using an intermediary MQTT broker.

For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the edge receiving pipeline reads the data.

Sensor Reader None
System Metrics None
WebSocket Client None
Windows Event Log None

Processors

Edge pipelines support a limited number of processors. Processors function the same way in edge pipelines as they do in other pipelines. However, some processors have limitations in edge pipelines as noted below.

Edge pipelines support the following processors:

Supported Processor Limitations
Delay None
Dev Identity None
Expression Evaluator None
Field Remover None
JavaScript Evaluator In edge pipelines, the JavaScript Evaluator processor does not support the sdcFunctions scripting object.
Stream Selector None

Destinations

Edge pipelines support a limited number of destinations. Destinations function the same way in edge pipelines as they do in other pipelines. However, some destinations have limitations in edge pipelines as noted below.

Edge pipelines support the following destinations:

Destination Limitations
CoAP Client None
HTTP Client None
Kafka Producer None
MQTT Publisher Edge pipelines that use MQTT stages require using an intermediary MQTT broker.

For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the Data Collector receiving pipeline reads the data.

Trash None
WebSocket Client None

Error Record Handling

You can configure the following error record handling options for an edge pipeline:
Discard
The pipeline discards the record.
Write to File
The pipeline writes error records and related details to a local directory on the edge device. Create another edge pipeline with a Directory origin to process the error records written to the file.
Write to MQTT
The pipeline publishes error records and related details to a topic on an MQTT broker. Create another edge or standalone Data Collector pipeline with an MQTT Subscriber origin to process the error records published to the broker.

Data Formats

Edge pipelines can process the following data formats:
  • JSON
  • Text
Important: Although some supported stages list additional data formats, configure them to use JSON or Text only.

Configure corresponding stages to use the same data format. For example, if the MQTT Publisher destination in an edge sending pipeline uses the JSON data format, then configure the MQTT Subscriber origin in the Data Collector receiving pipeline to also use the JSON data format.

Edge Pipeline Limitations

Edge pipelines run on SDC Edge which is a lightweight agent without a UI. As a result, some features available for standalone pipelines are not available for edge pipelines at this time. We will provide support for some of these features in edge pipelines in a future release.

Please note the following limitations for edge pipelines:
  • Email and webhook notifications cannot be sent by edge pipelines.
  • Rules and alerts cannot be defined for edge pipelines.
  • Edge pipelines support a limited number of record, math, pipeline, and string functions.

  • Edge pipelines do not support dataflow triggers.
  • Edge pipelines do not support multithreaded processing.
  • You cannot capture snapshots for edge pipelines.