Processing Changed Data

Certain stages enable you to easily process change capture data (CDC) or transactional data in a pipeline.

CDC-enabled origins can read change capture data. Some exclusively read change capture data, others can be configured to read it. When reading changed data, they determine the CRUD operation associated with the data and include CRUD operations - such as insert, update, upsert, or delete - in the sdc.operation.type record header attribute.

CRUD-enabled processors and destinations can use the CRUD operation type in the sdc.operation.type header attribute when writing records, enabling the external system to perform the appropriate operation.

Using a CDC-enabled origin and CRUD-enabled stages in a pipeline allows you to easily write changed data from one system into another. You can also use a CDC-enabled origin to write to non-CRUD destinations, and non-CDC origins to write to CRUD-enabled stages.

CRUD Operation Header Attribute

CDC-enabled origins read include the sdc.operation.type record header attribute in all records when reading changed data.

CRUD-enabled processors and destinations can use the CRUD operation type in the sdc.operation.type header attribute when writing records, enabling the external system to perform the appropriate operation.

The sdc.operation.type record header attribute uses the following integers to represent CRUD operations:
  • 1 for INSERT records
  • 2 for DELETE records
  • 3 for UPDATE records
  • 4 for UPSERT records
  • 5 for unsupported operations or codes
  • 6 for UNDELETE records
  • 7 for REPLACE records
  • 8 for MERGE records
Note: Some origins use only a subset of the operations, based on the operations supported by the origin system. Similarly, destinations recognize only the subset of the operations that the destination systems support. See the origin and destination documentation for details about supported operations.

CDC-Enabled Stages

CDC-enabled stages provide the CRUD operation type in the sdc.operation.type record header attribute. Some origins provide alternate and additional header attributes.

The following stages provide CRUD record header attributes:
CDC-Enabled Stage CRUD Record Header Attributes
MySQL Binary Log Includes the CRUD operation type in the sdc.operation.type record header attribute.

Includes additional CDC information in record fields.

For more information, see Generated Records.

Oracle CDC Client Includes the CRUD operation type in the sdc.operation.type record header attribute.

For more information, see CRUD Operation Header Attributes.

Includes additional CDC information in record header attributes with the oracle.cdc prefix, such as oracle.cdc.table.

Salesforce Includes the CRUD operation type in the sdc.operation.type record header attribute.

For more information, see CRUD Operation Header Attribute.

CRUD-Enabled Stages

The following stages recognize CRUD operations stored in record header attributes and can perform writes based on those values. Some stages also provide CRUD-related properties.
CRUD-Enabled Stage Supported Operations Stage Processing
MySQL Producer destination
  • INSERT
  • UPDATE
  • DELETE
Determines the operation to use based on:
  • sdc.operation.type record header attribute
  • Default Operation and Unsupported Operation Handling properties in the stage

For more information, see Define the CRUD Operation.

Oracle Producer destination
  • INSERT
  • UPDATE
  • DELETE
Determines the operation to use based on:
  • sdc.operation.type record header attribute
  • Default Operation and Unsupported Operation Handling properties in the stage

For more information, see Define the CRUD Operation.

PostgreSQL Producer destination
  • INSERT
  • UPDATE
  • DELETE
Determines the operation to use based on:
  • sdc.operation.type record header attribute
  • Default Operation and Unsupported Operation Handling properties in the stage

For more information, see Define the CRUD Operation.

Salesforce destination
  • INSERT
  • UPDATE
  • UPSERT
  • DELETE
  • UNDELETE
Determines the operation to use based on:
  • sdc.operation.type record header attribute
  • Default Operation and Unsupported Operation Handling properties in the stage

For more information, see Define the CRUD Operation.

Snowflake destination
  • INSERT
  • UPDATE
  • DELETE
Determines the operation to use based on:
  • sdc.operation.type record header attribute

For more information, see Define the CRUD Operation.

SQL Server Producer destination
  • INSERT
  • UPDATE
  • DELETE
Determines the operation to use based on:
  • sdc.operation.type record header attribute
  • Default Operation and Unsupported Operation Handling properties in the stage

For more information, see Define the CRUD Operation.