Overview

Data formats - such as Avro, JSON, and log - are methods to encode data that adhere to generally accepted specifications.

The way that stages process data can be similar based on the stage type and the type of data being processed.

This chapter discusses how stages process data formats generally. For the details of how a stage processes different data formats, see the "Data Formats" section of the stage documentation.

File Compression Formats

Origins and processors that read files can read uncompressed files, compressed files, archives, and compressed archives.

The following table lists the supported file types by extension:
Compression Format Description
Uncompressed Processes uncompressed files of the configured data format.
Compressed Processes files compressed by the following compression formats:
  • gzip
  • bzip2
  • xz
  • lzma
  • Pack200
  • DEFLATE
  • Z
Archive Processes files archived by the following archive formats:
  • 7z
  • ar
  • arj
  • cpio
  • dump
  • tar
  • zip
Compressed Archive Processes files in compressed archives created by supported compression and archive formats.