Dataflow Performance Blog

Getting Started with StreamSets Data Collector on Docker

Docker logoSimplicity is the ultimate sophistication.
– Leonardo da Vinci

As a recent hire on the Engineering Productivity team here at StreamSets, my early days at the company were marked by efforts to dive head-first into StreamSets Data Collector (SDC). As it turns out, the Docker images we publish for SDC were the easiest way to explore its vast set of features and capabilities, which is exactly why I am writing this blog post.

Without further ado, let’s get started.

Start a Docker container with SDC

To start a Docker container with the most recent release of StreamSets Data Collector, just run the following command:

Here are the options we specified (for a full list, check out the image notes on Docker Hub):

-dCreate Docker container in the background in detached mode
--nameName for this container
-pPublish container's port 18630 on host's 18630
--restart on-failureRestart only if the container exits with a non-zero exit status

If all goes well, running docker ps will show output like the following:

Voila! We have successfully created a Docker container with SDC. Pretty simple. Right?
Note the port 18630. This is the host port to which Docker has published the SDC container’s port 18630. We can verify that the service has started by using a web browser pointed to localhost:18630

Login Page

This will present a prompt for username and password. Type the default credentials (admin:admin) and we would see a screen like following:

Working with SDC

Now that we have access to the web UI, we can start playing with all the cool capabilities that SDC has to offer. For someone new, a great place to start would be our tutorials, which walk one through everything from creating and running a pipeline to more advanced operations like data manipulation.

Here are a few tricks I learned along the way which helped a lot.

Exploring the Docker container

After we have created the Docker container, we might want to take a look around (e.g. just to see how files are laid out). One simple way is to run the following command to start a Bash session inside the container:

Once we are inside, we can run whatever commands we need and, when we’re done, can use exit (or CTRL+D) to come back to the host:

Restarting SDC

One common gotcha with running SDC in Docker happens when we need to install additional stage libraries. In the web UI, go ahead and select a library and then click the “Install” icon (see documentation for details). At this point, we would see a dialog like the following:

If we click “Restart Data Collector,” we would discover that the container comes back online automatically.

Looking into logs

While exploring, if we do something that ends up crashing SDC, here is how to see its logs along with some sample output:

If SDC is running, we could also tail these by adding in the -f argument to docker logs.

Removing SDC

To clean up our instance of StreamSets Data Collector and all the resources it is using, just run the following command. Keep in mind that this will remove our SDC instance and we shall not be able to get back any data/logs/resources that were created in the process.

Conclusion

In this blog post, we have learned how to start StreamSets Data Collector in a Docker container, how to use it (along with a few tricks), and finally, how to remove it; a complete cycle of working with SDC and Docker.

What interesting facts have you come across in your journey running SDC in Docker? Did I miss something here? I would love to hear from you in the comments or over in the StreamSets community.

Kirti VelankarGetting Started with StreamSets Data Collector on Docker