Supported Systems and Versions

Data Collector supports working with a wide range of external systems. StreamSets tests to verify that Data Collector performs without issues when working with those systems.

The following tables list the systems that Data Collector supports and tests, and the stages that work with those systems.

Cloud Native

Data Collector supports the cloud native providers listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the listed stages on the tested environment.
Supported Cloud Provider Stages Tested Environment
Amazon Origins:
  • Amazon S3
  • Amazon SQS Consumer
  • Kinesis Consumer
Destinations:
  • Amazon S3
  • Kinesis Firehose
  • Kinesis Producer
Executor:
  • Amazon S3

Credential Store:

  • Amazon Secrets Manager

AWS
Databricks (Runtime 6.3 or later) Databricks Delta Lake destination

Databricks Job Launcher executor

Databricks Query executor

Databricks Delta Lake
Google Cloud Storage Origins:
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Destinations:
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Google Cloud Storage
Microsoft Azure Origins:
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure IoT/Event Hub Consumer
Destinations:
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure Event Hub Producer
  • Azure IoT Hub Producer
  • Azure Synapse SQL
Executors:
  • ADLS Gen1 File Metadata
  • ADLS Gen2 File Metadata
Credential Store:
  • Azure Key Vault
Microsoft Azure
Salesforce Salesforce origin

Salesforce Lookup processor

Einstein Analytics destination

Salesforce destination

Salesforce
Snowflake Snowflake destination Amazon S3

Microsoft Azure

Protocols

Data Collector supports the protocols listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the following protocols unless the implementation proves below standard for the protocol.

Private extensions for the protocols are not supported unless specified in the table.
Supported Protocol Stages Tested Environment
CoAP CoAP Server origin

CoAP Client destination

Eclipse Californium 1.0.4
HTTP Origins:
  • HTTP Client
  • HTTP Server
  • NiFi HTTP Server
Processors:
  • HTTP Client
  • HTTP Router
Destinations:
  • HTTP Client
Apache HTTP from Centos 6.8
JMS JMS Consumer origin

JMS Producer destination

ActiveMq 5.14.3
MQTT MQTT Subscriber origin

MQTT Publisher destination

Mosquitto
OPC UA OPC UA Client origin Full testing not performed at this time
SFTP/ FTP / FTPS SFTP/FTP/FTPS Client origin

SFTP/FTP/FTPS Client destination

SFTP/FTP/FTPS Client executor

vsftpd 3.0
Syslog Syslog destination Full testing not performed at this time
TCP TCP Server origin Java TCP Stack
UDP UDP Multithreaded Source origin

UDP Source origin

Java UDP Stack
Websocket Origins:
  • WebSocket Client
  • WebSocket Server
Destination:
  • WebSocket Client
Java HTTP Stack

Versioned Systems

Versioned systems are external systems with multiple versions. When Data Collector supports multiple versions of an external system, you might need to install a specific stage library to work with a particular version, depending on your Data Collector installation. For details on individual stage libraries and the stages that they include, see Available Stage Libraries.

The following table lists the system versions that are supported and tested for Data Collector.

The supported versions column lists the system versions that customers with an enterprise account can receive help with. The tested versions column lists the subset of the supported versions that have been fully tested.
System Stages Supported Versions Tested Versions
Aerospike Aerospike destination Aerospike 3.15.x Full testing not performed at this time
Cassandra Cassandra destination Cassandra 1.2, 2.x, 3.x Cassandra 3.11
Couchbase Server Couchbase destination Couchbase Server 5.x Couchbase Server 5.1.1
Elasticsearch Elasticsearch origin

Elasticsearch destination

Elasticsearch 5.x - 7.x Elasticsearch 5.20, 6.8.12, 7.9.0
Flume Flume destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
Greenplum GPSS Producer destination Greenplum 5.x Greenplum 5.12.0
Hadoop Distributed File System (HDFS):

Data Collector cluster mode

Origin:
  • Hadoop FS
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • Amazon EMR 5.14.x with Hadoop 2.8.3.
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • HDP 3.1.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • HDP 2.6.4.0, 3.1.0
Hadoop Distributed File System (HDFS):

Data Collector standalone mode

Origin:
  • Hadoop FS Standalone
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • HDP 3.1.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • HDP 2.6.4.0, 3.1.0
Hashicorp Vault Hashicorp Vault credential store General support Full testing not performed at this time
HBase HBase Lookup processor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • HDP 3.1.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • HDP 2.6.4.0, 3.1.0
Hive Hive Metadata processor

Hive Metastore destination

Hive Query executor

  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • HDP 2.6.x distribution of Hive 2.1
  • HDP 2.6.x distribution of Hive 1.x.
  • HDP 3.1.x
  • MapR 6.0.0 with MEP 4.x
  • MapR 6.0.1 with MEP 5.x
  • MapR 6.1.0 with MEP 6.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • HDP 2.6.4.0, 3.1.0
Hive Streaming Hive Streaming destination

Hive Query executor

  • Hive 0.13 and later
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • MapR 6.0.0 with MEP 4.x
  • MapR 6.0.1 with MEP 5.x
  • MapR 6.1.0 with MEP 6.x
Full testing not performed at this time
InfluxDB InfluxDB destination InfluxDB 0.9 or greater InfluxDB 0.13, 1.7.10
Java Keystore Java Keystore credential store Java Virtual Machine Java Virtual Machine
Kafka:

Data Collector cluster mode

Kafka Consumer origin
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x with:
    • CDS powered by Spark 2.2 release 1
    • CDS powered by Spark 2.3 release 2, 3, 4
  • HDP 3.1.0
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDH 5.10 with CDH Kafka 2.1 and CDS 2.1

  • CDH 5.15 with CDH Kafka 3.1 and CDS 2.3 release 3

  • CDH 5.16 with CDH Kafka 4.1 and Spark 2.1

Kafka:

Data Collector standalone mode

Origins:
  • Kafka Consumer
  • Kafka Multitopic Consumer
Destination:
  • Kafka Producer
  • Apache Kafka 1.0.x, 1.1.x
  • Apache Kafka 2.0.x - 2.6.x
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x
  • HDP 3.1.0
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDH Kafka 2.1.0, 3.0.0, 3.1.0, 4.1.0
  • HDP 3.1.0

  • Apache Kafka 0.10, 0.11, 0.9, 1.0, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6
KineticaDB Kinetica destination
  • KineticaDB 6.0.x - 6.2.x
  • KineticaDB 7.0.x
Full testing not performed at this time
Kudu Kudu Lookup processor

Kudu destination

  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
MapR DB Origin:
  • MapR DB
Destinations:
  • MapR DB
  • MapR DB JSON
  • MapR 6.0.0 with optional MEP 4.x
  • MapR 6.0.1 with optional MEP 5.x
  • MapR 6.1.0 with optional MEP 6.x
  • MapR 6.0.0 with MEP 4
  • MapR 6.0.1 with MEP 5
  • MapR 6.1.0 with MEP 6
MapR FS:

Data Collector cluster mode

MapR FS origin

MapR FS destination

  • MapR 6.0.0 with MEP 4.x
  • MapR 6.0.1 with MEP 5.x
  • MapR 6.1.0 with MEP 6.x
  • MapR 6.0.0 with MEP 4
  • MapR 6.0.1 with MEP 5
  • MapR 6.1.0 with MEP 6
MapR FS:

Data Collector standalone mode

MapR FS Standalone origin

MapR FS destination

MapReduce executor

  • MapR 6.0.0 with optional MEP 4.x
  • MapR 6.0.1 with optional MEP 5.x
  • MapR 6.1.0 with optional MEP 6.x
  • MapR 6.0.0 with MEP 4
  • MapR 6.0.1 with MEP 5
  • MapR 6.1.0 with MEP 6
MapR Streams Origins:
  • MapR Multitopic Streams Consumer
  • MapR Streams Consumer
  • MapR DB CDC
Destination:
  • MapR Streams Producer
MapR 6.1.0 with optional MEP 6.x MapR 6.1.0 with MEP 6
MemSQL MemSQL Fast Loader destination MemSQL 6.8 and later MemSQL 6.8.15 with the MySQL Connector/J 8.0.12 driver
Microsoft SQL Server SQL Server 2019 BDC origin SQL Server 2019 Big Data Cluster SQL Server 2019 Big Data Cluster
SQL Server CDC Client origin

SQL Server Change Tracking origin

  • SQL Server 2017
  • SQL Server 2019
  • SQL Server 2017
  • SQL Server 2019
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
SQL Server 2017 and later
  • SQL Server 2017
  • SQL Server 2019
MongoDB MongoDB origin

MongoDB Lookup processor

MongoDB destination

MongoDB 3.x, 4.x MongoDB 3.6, 4.0
MongoDB Oplog origin MongoDB 3.x, 4.x MongoDB 3.6, 4.0
MySQL Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
MySQL 5.7 and later
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
MySQL Binary Log MySQL 5.7, 5.8
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
NiFi NiFi HTTP Server origin General support Full testing not performed at this time
Omniture Omniture origin General support Full testing not performed at this time
Oracle Oracle Bulkload origin
  • Oracle 11g, 12c, 18c, 19c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c

Hosted systems, such as AWS RDS, and derived systems, such as Oracle Exadata, are not supported unless listed above.

  • Oracle 11g, 19c with the Oracle 19.3.0 JDBC driver
  • Oracle RAC 12c, 19c with the Oracle 19.3.0 JDBC driver
Oracle CDC Client origin
  • Oracle 11g, 12c, 18c, 19c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c

Hosted systems, such as AWS RDS, and derived systems, such as Oracle Exadata, are not supported unless listed above.

  • Oracle 11g, 19c with the Oracle 19.3.0 JDBC driver
  • Oracle RAC 12c, 19c with the Oracle 19.3.0 JDBC driver
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
  • Oracle 11g, 12c, 18c, 19c, and later
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, and later
Also supported:
  • Hosted systems, such as AWS RDS
  • Derived systems, such as Oracle Exadata
  • Oracle 11g, 19c with the Oracle 19.3.0 JDBC driver
  • Oracle RAC 12c, 19c with the Oracle 19.3.0 JDBC driver
PMML PMML Evaluator processor General support Full testing not performed at this time
PostgreSQL Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
PostgreSQL 9.x and later
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
PostgreSQL CDC Client origin
  • PostgreSQL 9.4 or later 9.x
  • PostgreSQL 10.x -13.x
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
Pulsar Pulsar Consumer origin

Pulsar Producer destination

Pulsar 2.x
  • Pulsar 2.1.0

  • Pulsar 2.2.1

  • Pulsar 2.3.2

  • Pulsar 2.4.2

  • Pulsar 2.5.1

  • Pulsar 2.6.2

RabbitMQ RabbitMQ Consumer origin

RabbitMQ Producer destination

RabbitMQ 3.5.x and later RabbitMQ 3.5.6, 3.8.0
Redis Redis Consumer origin

Redis destination

Redis 2.x - 4.x Redis 4.0.1
SAP HANA SAP HANA Query Consumer origin SAP HANA 2.4.x SAP HANA 2.0 with the SAP HANA JDBC driver version 2.4.76
Solr Solr destination
  • Apache Solr 6.x
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • HDP 2.6.4.0, 3.1.0
Spark

Spark Evaluator processor

Spark executor

  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH Spark 2.1.x Release 1
  • HDP 3.1.x
  • CDH 5.10.0, 5.11.1, 5.12.0, 5.13.0, 5.14.0, 5.15.0, 5.16.1
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • HDP 2.6.4.0, 3.1.0
Splunk Splunk destination General support Full testing not performed at this time
TensorFlow TensorFlow Evaluator processor TensorFlow 1.x Full testing not performed at this time
Teradata Teradata Consumer origin Teradata 16.x and later Teradata Database release 16.20 with the Teradata JDBC driver version 16.20.00.08
Thycotic Secret Server Thycotic Secret Server credential store Full testing not performed at this time Full testing not performed at this time