Event Generation

The event framework generates pipeline events for Data Collector standalone pipelines when the pipeline starts and stops.

Not available in Data Collector Edge pipelines.

In a Data Collector standalone pipeline, you can pass pipeline events to an executor or to another pipeline for additional processing. By default, these events are discarded. For more information about pipeline events, see Pipeline Event Generation.

For general information about the event framework, see Dataflow Triggers Overview.

Pipeline Event Records

Pipeline event records have the following event-related record header attributes. Record header attributes are stored as String values:

Record Header Attribute Description
sdc.event.type Event type. Uses one of the following types:
  • pipeline-start - Generated as the pipeline starts.
  • pipeline-stop - Generated as the pipeline stops.
sdc.event.version Integer that indicates the version of the event record type.
sdc.event.creation_timestamp Epoch timestamp when the stage created the event.
The event framework generates the following types of pipeline events:
pipeline-start
The event framework generates start events as the pipeline initializes, immediately after it starts and before individual stages are initialized.
The start event record has sdc.event.type set to pipeline-start, and the following fields:
Pipeline Start Event Field Description
pipelineId The ID of the pipeline that started.
pipelineTitle The user-defined name of the pipeline that started.
user The user who started the pipeline.
parameters Any parameters used when starting the pipeline.
pipeline-stop
The event framework generates stop events as the pipeline stops, either manually, programmatically, or due to a failure. The stop event is generated after all stages have completed processing and cleaning up temporary resources, such as removing temporary files.
The stop event record has sdc.event.type set to pipeline-stop, and the following fields:
Pipeline Stop Event Field Description
pipelineId The ID of the pipeline that stopped.
pipelineTitle The user-defined name of the pipeline that stopped.
reason The reason why the pipeline stopped. Can be set to the following reasons:
  • Error - An error occurred as the pipeline was running.
  • Finished - The pipeline finished all expected processing.
  • User - A user stopped the pipeline.
user The user who stopped the pipeline, when relevant.