Installation

Prerequisites

Prior to installing the StreamSets Test Framework, a number of prerequisites must be satisfied:

Python

The StreamSets Test Framework has been tested against Python 3.6 and this is the version we recommend be used when invoking stf commands. If your machine has a different version of Python 3+ installed, feel free to try it out, but if things seem broken, we recommend using pyenv to quickly and non-destructively switch to a different version of Python.

Warning

Users have reported errors when running stf on machines with Python 3.7 installed. As such, we recommend not using it at this time.

Docker

Any recent version of Docker should work with the StreamSets Test Framework.

Note

If using Docker for Mac, the StreamSets Test Framework requires that you go into your Docker Preferences and deselect the “Securely store Docker logins in macOS keychain” if you want to pull any Docker images stored on private repositories. This is needed in order to allow credentials entered via docker login to be passed into the STF container at runtime.

Activation key

The StreamSets Test Framework depends upon the presence of a StreamSets Toolkit activation key in the user’s home folder, as described in the StreamSets SDK for Python documentation. This key is available to customers as part of the StreamSets Premium and Enterprise subscription tier.

Installation from PyPI

To install the most recent stable release of the project, use your Python 3 installation’s instance of pip:

$ pip3 install streamsets-testframework

To do a simple check to verify that the installation was successful, simply run

$ stf --version
stf 1.0.0

Docker images

The stf command launches most of its sub-commands in a Docker container started from the streamsets/testframework:master image on Docker Hub.

Many STF tests of StreamSets Data Collector support the automatic starting and stopping of SDC containers from Docker images. These tests will also handle orchestration of stage libraries and external libraries, as required by the tests, which are also packaged as Docker images. As part of our automated build process, StreamSets creates and pushes to Docker Hub images of StreamSets Data Collector and its stage libraries for stable releases and nightly builds. Because of license restrictions, though, there are a number of Docker images required to run certain STF tests that we do not distribute. These include images containing the JDBC drivers needed as external libraries by Data Collector. To build these, simply run the stf build extras command:

$ stf build extras
2018-09-09 04:50:57 PM streamsets.testframework.cli INFO     Pulling Docker image streamsets/testframework:master ...
2018-09-09 04:50:58 PM streamsets.testframework.cli INFO     Building STF Docker images ...
...