Development Stages

The Data Collector provides several development stages that you can use to help develop or test pipelines.
Note: Do not use development stages in production pipelines.
You can use the following development stages when developing or testing pipelines:
Dev Data Generator origin
Generates records with the configured number of fields with the selected data types. You can use a Map or List-Map root field type.
The origin can generate events to test event handling functionality. To generate event records, select the Produce Events property.
When generating events, the origin uses the configured fields as the body of the event record and adds event record header attributes. You can also specify the event type with the Event Name property. For example, to create a no-more-data event, enter "no-more-data" for the event name. For more information about the event framework and event record header attributes, see Dataflow Triggers Overview.
The origin can also generate multiple threads for testing for a multithreaded pipeline. To generate multiple threads, enter a number larger than 1 for the Number of Threads property. For more information about multithreaded pipelines, see Multithreaded Pipeline Overview.
The On Record Error property has no effect in this stage.
Dev Random Source origin
Generates records with the configured number of Long fields. You can define a delay between batches and a maximum number of records to generate.
The On Record Error property has no effect in this stage.
Dev Raw Data Source origin
Generates records based on user-supplied data. You can enter raw data, select the data format of the data, and then configure any format-related configuration options.
For example, you can enter a set of log data, select the log data format, and then define the log format and other log properties for the data.
In data preview, this stage displays the raw source data as well as the data generated by the stage.
Dev SDC RPC with Buffering origin
Receives records from an SDC RPC destination, temporarily buffering the records to disk before passing the records to the next stage in the pipeline. Use as the origin in an SDC RPC destination pipeline.
Note: After a deliberate or unexpected stop of the pipeline, buffered records are lost.
Dev Identity processor
Passes all records through to the next stage. Use as a placeholder in the pipeline. You can define required fields and preconditions, and configure stage error handling.
Dev Random Error processor
Generates error records so you can test pipeline error handling. You can configure the stage to discard records, define required fields and preconditions, and configure stage error handling.
Dev Record Creator processor
Generates two records for each record that enters the stage. You can define required fields and preconditions, and configure stage error handling.
To Event destination
Generates events for testing event handling functionality. To generate events, select the Produce Events property.
The destination generates an event record for each incoming record. It uses the incoming record as the body of an event record and adds event record header attributes. Note that any record header attributes in the incoming record might be lost or replaced.
For more information about the event framework and event record header attributes, see Dataflow Triggers Overview.