Custom Stage Libraries

If you develop custom stages, store the stage libraries in a local directory external to the Data Collector installation directory. Use an external directory to enable use of the custom stage libraries after Data Collector upgrades.

Use the required procedure for your installation type.

We have tutorials for creating a custom origin, processor, and destination on our StreamSets tutorials page. To check them out, go to: https://github.com/streamsets/tutorials.

Storing Custom Libraries for RPM and Tarball

To store custom stage libraries for an RPM or tarball installation, perform the following steps:

  1. Create a local directory external to the Data Collector installation directory for the custom stage libraries. Use an external directory to enable use of the libraries after Data Collector upgrades.
    For example, if you installed Data Collector in the following directory:
    /opt/sdc/
    you might create the custom stage library directory at:
    /opt/sdc-user-libs
  2. Copy the custom stage libraries to the directory. Use a directory structure for each custom stage.
    For example to store libraries for a custom stage named customstage1, you would copy them to the following directory:
    /opt/sdc-user-libs/customstage1
  3. Add the USER_LIBRARIES_DIR environment variable to the appropriate file and point it to the custom stage library directory.

    Modify environment variables using the method required by your installation type.

    Set the environment variable as follows:

    export USER_LIBRARIES_DIR="<custom stage library directory>"

    For example:

    export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"
  4. When using the Java Security Manager, which is enabled by default, update the Data Collector security policy to include the custom stage library directory as follows:
    1. In the Data Collector configuration directory, open the security policy file:
      $SDC_CONF/sdc-security.policy
    2. Add the following lines to the file:
      // custom stage library directory
      grant codebase "file://<custom stage library directory>-" {
         permission java.security.AllPermission;
      };
      For example:
      // custom stage library directory
      grant codebase "file:///opt/sdc-user-libs/-" {
         permission java.security.AllPermission;
      };
  5. Restart Data Collector.
    The Data Collector stage library displays the custom stages.

Storing Custom Libraries for Cloudera Manager

To store custom stage libraries for a Cloudera Manager installation, perform the following steps:

  1. In Cloudera Manager, select the StreamSets service and then click Configuration.
  2. On the Configuration page, in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field, add the USER_LIBRARIES_DIR environment variable and point it to the custom stage library directory, as follows:
    export USER_LIBRARIES_DIR="<custom stage library directory>"
    For example:
    export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"
  3. On every node that runs Data Collector, copy the custom stage libraries to the directory defined for the USER_LIBRARIES_DIR environment variable. Use a directory structure for each custom stage.
    For example to store libraries for a custom stage named customstage1, you would copy them to the following directory:
    /opt/sdc-user-libs/customstage1
  4. When using the Java Security Manager, which is enabled by default, update the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-security.policy property to include the custom stage library directory as follows:
    // custom stage library directory
    grant codebase "file://<custom stage library directory>-" {
       permission java.security.AllPermission;
    };
    For example:
    // custom stage library directory
    grant codebase "file:///opt/sdc-user-libs/-" {
       permission java.security.AllPermission;
    };
  5. Restart Data Collector.