External Libraries

You can install a driver or other library as an external library to make it available to a Transformer stage.

Transformer includes the libraries needed to use most Transformer stages out of the box. However, you might install an external library in the following cases:
  • Some stages, such as the Oracle JDBC Table origin and the MySQL JDBC Table origins, require installing a driver as an external library.
  • Some stages, such as the JDBC origin, lookup, and destination, include several drivers, but require installing a driver to access certain databases.
  • Some stages provide the required libraries, but you can install custom libraries to access custom functionality. For example, you might install a custom Java or Scala library for the Scala processor.

When installing an external library, you install it into the stage library that includes the stage. For example, to use a custom Scala library with Scala processors, you install the Scala library as an external library for the Basic stage library.

To use an external library with multiple stages, install the external library into each stage library associated with the stages. For example, if you want to use an Oracle JDBC driver with the Scala processor and the Oracle JDBC Table origin, you install the driver as an external library for the Basic stage library and for the JDBC stage library.

You can install external libraries using Package Manager in the Transformer UI, or you can install them manually.

Package Manager Installation

To install external libraries using the Transformer Package Manager, complete the following general steps:
  1. Set up an external directory to store the libraries.
  2. Use the Transformer Package Manager to install the external libraries.

Step 1. Set Up an External Directory

Before you install external libraries, set up a local directory external to the Transformer installation directory to store the libraries. Use an external directory to enable use of the libraries after Transformer upgrades. Use the procedure for your Transformer installation type.

Setting Up for Tarball and RPM Installations

Before you install external libraries for a tarball or RPM installation, set up an external directory to store the libraries.

  1. Create a local directory external to the Transformer installation directory.
    For example, if you installed Transformer in the following directory:
    /opt/transformer/
    you might create the external directory at:
    /opt/transformer-extras
  2. Grant the user who starts Transformer ownership on the external directory.
    For example, if you use the default system user and group named transformer to run Transformer as a service, use the following command to change the owner of the external directory and all files in the directory to transformer:transformer:
    chown -R transformer:transformer /opt/transformer-extras
  3. Add the STREAMSETS_LIBRARIES_EXTRA_DIR environment variable to the appropriate file and point it to the external directory.

    Modify environment variables using the method required by your installation type.

    Set the environment variable as follows:

    export STREAMSETS_LIBRARIES_EXTRA_DIR="<external directory>"

    For example:

    export STREAMSETS_LIBRARIES_EXTRA_DIR="/opt/transformer-extras/"
  4. When using the Java Security Manager, which is enabled by default, update the Transformer security policy to include the external directory as follows:
    1. In the Transformer configuration directory, open the security policy file, $TRANSFORMER_CONF/transformer-security.policy.
    2. Add the following lines to the file:
      // user-defined external directory
      grant codebase "file://<external directory>-" {
        permission java.security.AllPermission;
      };
      For example:
      // user-defined external directory
      grant codebase "file:///opt/transformer-extras/-" {
        permission java.security.AllPermission;
      };
  5. Restart Transformer.

Step 2. Install External Libraries

After you've set up the external directory, use the Transformer Package Manager to install external libraries.

  1. In Transformer, in the top right toolbar, click the Package Manager icon:

  2. In Package Manager, determine the stage library where you want to install the external library.

    Each stage library lists the stages included in the stage library. For example, the File stage library shown below includes the File and Whole Directory origins, and the File destination:

  3. At the bottom of the navigation panel, click External Libraries.

    Package Manager lists any currently installed external libraries.

  4. Click the Install External Libraries icon, located under the top right toolbar:

  5. In the Install External Libraries dialog box, select the stage library that needs to access the external library.

    For example, to install a custom Java library for the Scala processor, select the Basic stage library.

  6. Browse to select the external library to install and click Open.
  7. To install the external library into the specified stage library, click Upload.

    Package Manager installs the external library and displays a message offering to restart Transformer.

  8. To install additional external libraries, click Cancel, then repeat steps 4 - 7 for every stage library that needs access to the external library.
  9. After installing all of the external libraries that you want, restart Transformer in one of the following ways:
    • If you started Transformer manually, click Restart Transformer in the Install External Libraries window.
    • If you started Transformer as a service, you must use the command line for restart. Click Cancel in the Install External Libraries window, and then run the required command for your operating system:
      • For CentOS 6, Oracle Linux 6, or Red Hat Enterprise Linux 6, use:
        service transformer restart
      • For CentOS 7, Oracle Linux 7, or Red Hat Enterprise Linux 7, use:
        systemctl restart transformer

Manual Installation

To install external libraries manually, use the procedure required for your installation type.

Tarball and RPM Installations

To manually install external libraries for a tarball or RPM Transformer installation, perform the following steps:
  1. Create a local directory external to the Transformer installation directory.
    For example, if you installed Transformer in the following directory:
    /opt/transformer/
    you might create the external directory at:
    /opt/transformer-extras
  2. Create subdirectories for each set of external libraries based on the stage library name as follows:
    /opt/transformer-extras/<stage library name>/lib/
    For example, to install drivers for stages included with the JDBC stage library, create the following subdirectory:
    /opt/transformer-extras/streamsets-spark-jdbc-lib/lib/

    To also install drivers for stages in the Basic stage library, create the following subdirectory:

    /opt/transformer-extras/streamsets-spark-basic-lib/lib/
    Note: You can use Package Manager to easily view the stages in each stage library. You can find stage library names in the $TRANSFORMER_DIST/streamsets-libs directory.
  3. Copy the external libraries to the appropriate subdirectories.
  4. Add the STREAMSETS_LIBRARIES_EXTRA_DIR environment variable to the appropriate file and point it to the external directory.

    Modify environment variables using the method required by your installation type.

    Set the environment variable as follows:

    export STREAMSETS_LIBRARIES_EXTRA_DIR="<external directory>"

    For example:

    export STREAMSETS_LIBRARIES_EXTRA_DIR="/opt/transformer-extras/"
  5. When using the Java Security Manager, which is enabled by default, update the Transformer security policy to include the external directory as follows:
    1. In the Transformer configuration directory, open the security policy file, $TRANSFORMER_CONF/transformer-security.policy.
    2. Add the following lines to the file:
      // user-defined external directory
      grant codebase "file://<external directory>-" {
        permission java.security.AllPermission;
      };
      For example:
      // user-defined external directory
      grant codebase "file:///opt/transformer-extras/-" {
        permission java.security.AllPermission;
      };
  6. Restart Transformer.