Summary

Here are the key points about dataflow triggers and the event framework:
  1. You can use the event framework in any pipeline where the logic suits your needs.
  2. The event framework generates pipeline-related events and stage-related events.
  3. Pipeline events are generated when the pipeline starts and stops. For details, see Pipeline Event Generation.
  4. You can configure each pipeline event type to pass to a single executor for more complex processing.
  5. Stage events are generated based on the processing logic of the stage. For a list of event-generating stages, see Stage Event Generation.
  6. Events generate event records to pass relevant information regarding the event, such as the path to the file that was closed.

    Stage-generated event records differ from stage to stage. For a description of stage events, see "Event Record" in the documentation for the event-generating stage. For a description of pipeline events, see Pipeline Event Records.

  7. In the simplest use case, you can route stage event records to a destination to save event information.
  8. You can route stage event records to an executor stage so it can perform a task upon receiving an event.

    For a list of logical event generation and executor pairings, see Logical Pairings.

  9. You can add processors to event streams for stage events or to consuming pipelines for pipeline events.

    For example, you might add an Expression Evaluator to add the event generation time to an event record before writing it to a destination. Or, you might use a Stream Selector to route different types of event records to different executors.

  10. When working with stage events, you cannot merge event streams with data streams.
  11. You can use the Dev Data Generator and To Event development stages to generate events for pipeline development and testing.
  12. In preview and in Monitor mode, stage-generated event records display separately in the event-generating stage. Afterwards, they are treated like any standard record.
  13. You can configure preview to generate and execute pipeline events.

For examples of how you might use the event framework, see the case studies earlier in this chapter.