Custom Stage Libraries

If you develop custom stages, store the stage libraries in a local directory external to the Data Collector installation directory. Use an external directory to enable use of the custom stage libraries after Data Collector upgrades.

We have tutorials for creating a custom origin, processor, and destination on our StreamSets tutorials page. To check them out, go to: https://github.com/streamsets/tutorials.

Storing Custom Libraries in an External Directory

To store custom stage libraries in a local directory external to the Data Collector installation directory, perform the following steps:

  1. Create a local directory external to the Data Collector installation directory for the custom stage libraries. Use an external directory to enable use of the libraries after Data Collector upgrades.
    For example, if you installed Data Collector in the following directory:
    /opt/sdc/
    you might create the custom stage library directory at:
    /opt/sdc-user-libs
  2. Copy the custom stage libraries to the directory. Use a directory structure for each custom stage.
    For example to store libraries for a custom stage named customstage1, you would copy them to the following directory:
    /opt/sdc-user-libs/customstage1
  3. In the Data Collector environment configuration file, add the USER_LIBRARIES_DIR environment variable and point it to the custom stage library directory.

    If you start Data Collector as a service, set the environment variable in the $SDC_DIST/libexec/sdcd-env.sh file. If you start Data Collector manually, set the variable in the $SDC_DIST/libexec/sdc-env.sh file.

    Set the environment variable as follows:

    export USER_LIBRARIES_DIR="<custom stage library directory>"

    For example:

    export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"
  4. When using the Java Security Manager, which is enabled by default, update the Data Collector security policy to include the custom stage library directory as follows:
    1. In the Data Collector configuration directory, open the security policy file:
      $SDC_CONF/sdc-security.policy
    2. Add the following lines to the file:
      // custom stage library directory
      grant codebase "file://<custom stage library directory>-" {
         permission java.security.AllPermission;
      };
      For example:
      // custom stage library directory
      grant codebase "file:///opt/sdc-user-libs/-" {
         permission java.security.AllPermission;
      };
  5. Restart Data Collector.
    The Data Collector stage library displays the custom stages.