Event Generation

The event framework generates events for standalone pipelines when the pipeline starts and stops. You can pass the event to an executor or to another pipeline for additional processing. By default, these events are discarded. For more information about pipeline events, see Pipeline Event Generation.

For general information about the event framework, see Dataflow Triggers Overview.

Pipeline Event Records

Pipeline event records have the following event-related record header attributes. Record header attributes are stored as String values:

Record Header Attribute Description
sdc.event.type Event type. Uses one of the following types:
  • start - Generated as the pipeline starts.
  • stop - Generated as the pipeline stops.
sdc.event.version An integer that indicates the version of the event record type.
sdc.event.creation_timestamp Epoch timestamp when the stage created the event.
The event framework generates the following types of pipeline events:
Start
The event framework generates start events as the pipeline initializes, immediately after it starts and before individual stages are initialized.
The start event record has the sdc.event.type set to "start", and the following fields:
Pipeline Start Event Field Description
pipelineId The ID of the pipeline that started.
pipelineName The user-defined name of the pipeline that started.
user The user who started the pipeline.
parameters Any parameters used when starting the pipeline.
Stop
The event framework generates stop events as the pipeline stops, either manually, programmatically, or due to a failure. The stop event is generated after all stages have completed processing and cleaning up temporary resources, such as removing temporary files.
The stop event record has the sdc.event.type set to "stop", and the following fields:
Pipeline Stop Event Field Description
pipelineId The ID of the pipeline that stopped.
pipelineName The user-defined name of the pipeline that stopped.
reason The reason why the pipeline stopped. Can be set to the following reasons:
  • Error - An error occurred as the pipeline was running.
  • Finished - The pipeline finished all expected processing.
  • User - A user stopped the pipeline.
user The user who stopped the pipeline, when relevant.