Data Formats by Stage

Data Format Support

This appendix lists the data formats supported by origin, processor, and destination stages.

Origins

The following table lists the data formats supported by each origin.

Origin Avro Binary Datagram Delimited Excel JSON Log Protobuf SDC Record Text Whole File XML
Amazon S3    
Amazon SQS Consumer      
Azure IoT/Event Hub Consumer                
CoAP Server      
Directory    
Elasticsearch * * * Not Applicable * * *
File Tail                  
Google BigQuery * * * Not Applicable * * *
Google Cloud Storage        
Google Pub/Sub Subscriber        
gRPC Client                  
Hadoop FS                  
Hadoop FS Standalone      
HTTP Client          
HTTP Server        
HTTP to Kafka * * * Not Applicable * * *
JDBC Multitable Consumer * * * Not Applicable * * *
JDBC Query Consumer * * * Not Applicable * * *
JMS Consumer      
Kafka Consumer    
Kafka Multitopic Consumer    
Kinesis Consumer      
MapR DB CDC * * * Not Applicable * * *
MapR DB JSON * * * Not Applicable * * *
MapR FS                  
MapR FS Standalone      
MapR Multitopic Streams Consumer      
MapR Streams Consumer        
MongoDB * * * Not Applicable * * *
MongoDB Oplog * * * Not Applicable * * *
MQTT Subscriber      
MySQL Binary Log * * * Not Applicable * * *
Omniture * * * Not Applicable * * *
OPC UA Client * * * Not Applicable * * *
Oracle Bulkload * * * Not Applicable * * *
Oracle CDC Client * * * Not Applicable * * *
PostgreSQL CDC Client * * * Not Applicable * * *
Pulsar Consumer      
RabbitMQ Consumer      
Redis Consumer      
REST Service        
Salesforce * * * Not Applicable * * *
SDC RPC                      
SDC RPC to Kafka                      
SFTP/FTP Client    
SQL Server CDC Client * * * Not Applicable * * *
SQL Server Change Tracking * * * Not Applicable * * *
System Metrics * * * Not Applicable * * *
TCP Server      
Teradata Consumer * * * Not Applicable * * *
UDP Multithreaded Source * * * Not Applicable * * *
UDP Source * * * Not Applicable * * *
UDP to Kafka * * * Not Applicable * * *
WebSocket Client      
WebSocket Server      
Windows Event Log * * * Not Applicable * * *

Processors

The following table lists the processors that read data of the listed format:
Processor Avro Binary Datagram Delimited JSON Log Netflow Protobuf SDC Record Syslog Text XML
Data Parser        
HTTP Client    
JSON Parser                      
Log Parser                      
XML Parser                      

The following table lists the processors that write data of the specified data format to a field:

Processor Avro Binary Delimited JSON Log Protobuf SDC Record Text XML
Data Generator  
JSON Generator                

Destinations

The following table lists the data formats supported by each destination.

Destination Avro Binary Delimited JSON Protobuf SDC Record Text Whole File XML
Aerospike * * * Not Applicable * * *
Amazon S3  
Azure Data Lake Storage    
Azure Event Hub Producer          
Azure IoT Hub Producer          
Cassandra * * * Not Applicable * * *
CoAP Client          
Couchbase * * * Not Applicable * * *
Elasticsearch * * * Not Applicable * * *
Einstein Analytics * * * Not Applicable * * *
Flume    
Google BigQuery * * * Not Applicable * * *
Google Bigtable * * * Not Applicable * * *
Google Cloud Storage    
Google Pub/Sub Publisher  
Hadoop FS    
HBase * * * Not Applicable * * *
Hive Metastore                
Hive Streaming * * * Not Applicable * * *
HTTP Client    
InfluxDB * * * Not Applicable * * *
JDBC Producer * * * Not Applicable * * *
JMS Producer  
Kafka Producer  
Kinesis Firehose              
Kinesis Producer    
KineticaDB * * * Not Applicable * * *
Kudu * * * Not Applicable * * *
Local FS    
MapR DB * * * Not Applicable * * *
MapR DB JSON * * * Not Applicable * * *
MapR FS    
MapR Streams Producer    
MemSQL Fast Loader * * * Not Applicable * * *
MongoDB * * * Not Applicable * * *
MQTT Publisher          
Named Pipe          
Pulsar Producer    
RabbitMQ Producer    
Redis    
Salesforce * * * Not Applicable * * *
Send Response to Origin                
SDC RPC                
Snowflake * * * Not Applicable * * *
Solr * * * Not Applicable * * *
Splunk * * * Not Applicable * * *
Syslog  
To Error * * * Not Applicable * * *
Trash * * * Not Applicable * * *
WebSocket Client