A processor stage represents a type of data processing that you want to perform. You can use as many processors in a pipeline as you need.

You can use the following processors in a pipeline:
  • Delay - Delays passing a batch to the rest of the pipeline.
  • Expression Evaluator - Performs calculations on data. Can also add or modify record header attributes.
  • Field Flattener - Flattens nested fields.
  • Field Merger - Merges fields in complex lists or maps.
  • Field Order - Orders fields in a map or list_map root field type and outputs the fields into a list_map or list root field type.
  • Field Pivoter - Pivots data in a list, map, or list_map field and creates a record for each item in the field.
  • Field Remover - Removes fields from a record.
  • Field Renamer - Renames fields in a record.
  • Field Replacer - Replaces field values.
  • Field Splitter - Splits the string values in a field into different fields.
  • Field Type Converter - Converts the data types of fields.
  • Field Zip - Merges list data from two fields.
  • HTTP Client - Sends requests to an HTTP resource URL and writes the results to a field.
  • JSON Generator - Serializes data from a field to a JSON-encoded string.
  • JSON Parser - Parses a JSON object embedded in a string field.
  • Record Deduplicator - Removes duplicate records.
  • Salesforce Lookup - Performs lookups in Salesforce to enrich records with data.
  • Stream Selector - Routes data to different streams based on conditions.
To help create or test pipelines, you can use the following development processors:
  • Dev Identity
  • Dev Random Error
  • Dev Record Creator

For more information, see Development Stages.