Administer SDC Edge

Administering Data Collector Edge (SDC Edge) involves configuring, starting, and shutting down the agent on the edge device.

Configuring SDC Edge

You can customize SDC Edge by editing the SDC Edge configuration file, <SDCEdge_home>/etc/edge.conf.

Use a text editor to edit the configuration file. To enable the changes, shut down and then restart SDC Edge.

The SDC Edge configuration file includes the following log property:
Execution Property Description
log-dir Directory where SDC Edge writes the log file.

Default is <SDCEdge_home>/log/. Be sure to uncomment the property if you change the value.

The SDC Edge configuration file includes the following execution property:
Execution Property Description
max-batch-size Maximum number of records included in a batch when the pipeline runs.

Default is 1000.

The SDC Edge configuration file includes the following process property:
Process Property Description
process-metrics-capture-interval Frequency in milliseconds that metrics about the SDC Edge process, such as memory usage, are refreshed.

Default is -1, which means that the metrics are refreshed when the HTTP REST API is called.

The SDC Edge configuration file includes the following HTTP properties:
HTTP Property Description
enabled Specifies whether a web server runs inside the SDC Edge process. When enabled, you can use the Data Collector UI or REST API to manage edge pipelines deployed to SDC Edge. Managing edge pipelines includes previewing, validating, starting, stopping, and monitoring pipelines as well as resetting the origin for pipelines.

Set to true in a development environment to simplify edge pipeline development.

Set to false in a production environment to secure the edge pipelines deployed to SDC Edge. When false, you manage edge pipelines using Control Hub or by starting pipelines when you start SDC Edge.

Default is true.

bind-address Address that SDC Edge binds to - localhost and the defined port number.

Default port number is 18633.

The SDC Edge configuration file includes the following Control Hub properties that define how SDC Edge works with Control Hub:
Control Hub Property Description
enabled Specifies whether the SDC Edge is enabled to work with Control Hub.

Default is false.

base-url URL to access Control Hub.

Set to https://cloud.streamsets.com.

app-auth-token Authentication token for this SDC Edge instance when enabled to work with Control Hub.
job-labels Labels to assign to this SDC Edge. Use labels to group Edge Data Collectors registered with Control Hub. To assign multiple labels, enter a comma-separated list of labels.

Default is "all", which you can use to run a job on all registered instances of SDC Edge.

events-recipient Name of the internal Control Hub messaging queue to which SDC Edge sends events.

Do not change this value.

ping-frequency Frequency in milliseconds that SDC Edge notifies Control Hub that it is running.

Default is 6,000.

status-events-interval Frequency in milliseconds that SDC Edge informs Control Hub of the following information:
  • Status of all local and published pipelines that are running on this SDC Edge.
  • Performance information for this SDC Edge - including CPU load and memory usage.

Default is 60,000.

Starting SDC Edge

You can start SDC Edge manually or as a service:

Manual start
To manually start SDC Edge, run the following command from the SDC Edge home directory:
bin/edge
Note: In a test or development environment, you can start SDC Edge so that it skips verifying trusted certificates. To do so, run the following command from the SDC Edge home directory to manually start SDC Edge: bin/edge -insecureSkipVerify. StreamSets highly recommends that you configure SDC Edge to verify trusted certificates in a production environment.
To modify the log level when you manually start SDC Edge, see Logs.
Service start
Before you can start SDC Edge as a service, you must register SDC Edge as a service on the edge device.
To start SDC Edge as a service, run the following command as an administrator from the SDC Edge home directory:
bin/edge -service start
For example, you might run the following command on Linux:
sudo opt/local/streamsets-datacollector-edge/bin/edge -service start

Shutting Down SDC Edge

You can shut down and then manually launch SDC Edge to apply changes to the SDC Edge configuration file.

Shut down SDC Edge based on whether SDC Edge is started manually or as a service:

Manual start
To shut down SDC Edge, type Ctrl+C in the command prompt.
Service start
To shut down SDC Edge started as a service, run the following command as an administrator from the SDC Edge home directory:
bin/edge -service stop
For example, you might run the following command on Linux:
sudo opt/local/streamsets-datacollector-edge/bin/edge -service stop

Restarting SDC Edge

When SDC Edge runs as a service, you can restart SDC Edge to apply changes to the SDC Edge configuration file. During the restart process, SDC Edge shuts down and then automatically restarts.

To restart SDC Edge, run the following command as an administrator from the SDC Edge home directory:
bin/edge -service restart
For example, you might run the following command on Linux:
sudo opt/local/streamsets-datacollector-edge/bin/edge -service restart

Logs

By default, SDC Edge writes log messages at the info severity level to the <SDCEdge_home>/log/edge.log file. To view the logs, simply open the edge.log file in a text editor.

To change the default log file directory, modify the SDC Edge configuration file.

You can modify the log level or send the log messages to the console when you manually start SDC Edge. You cannot modify this log information when SDC Edge runs as a service.

To modify logs, use the appropriate command:

  • To set the log level to debug for troubleshooting, run the following command from the SDC Edge home directory:
    bin/edge -debug
  • To send the log messages to the console instead of to the edge.log file, run the following command from the SDC Edge home directory:
    bin/edge -logToConsole