Logical Pairings

You can use events in any way that works for your needs.

The following tables outline some logical pairings of event generation with executors and destinations.

Pipeline Events

Pipeline Event Type Event Consumer
Pipeline Start

Any single executor, except Pipeline Finisher.

Pipeline Stop

Any single executor, except Pipeline Finisher.

Origin Events

Event Generating Origin Event Consumer
Amazon S3
  • Pipeline Finisher executor to stop the pipeline after processing queried data from all objects.
  • Any destination for event storage.
Azure Data Lake Storage Gen1
  • Pipeline Finisher executor to stop the pipeline after processing all available files.
  • Any destination for event storage.
Azure Data Lake Storage Gen2
  • Pipeline Finisher executor to stop the pipeline after processing all available files.
  • Any destination for event storage.
Google BigQuery Any destination for event storage.
Google Cloud Storage
  • Pipeline Finisher executor to stop the pipeline after processing queried data from all objects.
  • Any destination for event storage.
MySQL Multitable Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data from all tables.
  • Any destination for event storage.
MySQL Query Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data.
  • Any destination for event storage.
Oracle CDC Client Any destination for event storage.
Oracle Multitable Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data from all tables.
  • Any destination for event storage.
Oracle Query Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data.
  • Any destination for event storage.
PostgreSQL Multitable Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data from all tables.
  • Any destination for event storage.
PostgreSQL Query Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data.
  • Any destination for event storage.
Salesforce
  • Pipeline Finisher executor to stop the pipeline after processing queried data.
  • Any destination for event storage.
SQL Server Multitable Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data from all tables.
  • Any destination for event storage.
SQL Server Query Consumer
  • Pipeline Finisher executor to stop the pipeline after processing queried data.
  • Any destination for event storage.

Destination Events

Event Generating Destination Event Consumer
Amazon S3
  • Amazon S3 executor to create or copy objects or to add tags to closed objects.
  • Databricks Delta Lake executor to copy or merge data from Amazon S3 to a Delta Lake table.
  • Any destination for event storage.
Azure Data Lake Storage Gen1
  • ADLS Gen1 executor to change file metadata, create an empty file, or remove a file or directory after closing a file.
  • Any destination for event storage.
Azure Data Lake Storage Gen2
  • ADLS Gen2 executor to change file metadata, create an empty file, or remove a file or directory after closing a file.
  • Databricks Delta Lake executor to copy or merge data from Azure Data Lake Storage Gen2 to a Delta Lake table.
  • Any destination for event storage.
Google Cloud Storage Any destination for event storage.

Executor Events

Event Generating Executor Event Consumer
ADLS Gen1 File Metadata executor Any destination for event storage.
ADLS Gen2 File Metadata executor Any destination for event storage.
Amazon S3 executor Any destination for event storage.
Databricks Delta Lake executor Any destination for event storage.