Data Preview

Data Preview Overview

You can preview data to help build or fine-tune a pipeline. When using Control Hub, you can also use data preview when developing pipeline fragments.

You can use data preview with complete or incomplete pipelines and fragments. And you can choose from several options to provide source data for the preview.

When you preview data, source data passes through the pipeline or fragment, allowing you to review how the data passes and changes through each stage.

Data Preview Availability

You can preview complete and incomplete pipelines and Control Hub pipeline fragments. The Data Preview icon becomes active when data preview is available.

You can preview data under the following conditions:
  • The authoring Data Collector is an available registered Data Collector.
  • All stages in the pipeline are connected
  • All required properties are defined
Tip: Stage configuration does not have to be accurate or complete to preview data. After you connect all stages, you can enable data preview by entering any valid value for required properties.

Source Data for Data Preview

You can use the following types of data for a data preview:
  • Data from the origin - Use available data from the origin.
  • Data from the test origin - Use data from the test origin configured in the pipeline or fragment properties.
  • Data from a snapshot - Use snapshot data from the same pipeline or another pipeline. Available for pipelines only.

Writing to Destinations

As a tool for development, data preview does not write data to destinations by default.

If you like, you can configure the preview to write data to destinations. We advise against writing preview data to production destinations.

Notes

Keep the following notes in mind when previewing your data:
  • Date, datetime, and time data - Data preview displays date, datetime, and time data using the default format of the browser locale. For example, if the browser uses the en_US locale, preview displays dates using the following format: MMM d, y h:mm:ss a.
  • Oracle CDC Client pipelines - When previewing a pipeline that uses the Oracle CDC Client origin, data preview might time out before connecting to the origin system. When this occurs, try increasing the timeout to 120,000 milliseconds to allow the origin time to connect.
  • Whole file data format - When previewing a pipeline that processes whole file data, data preview displays only one record.

Previewing a Single Stage

You can preview data for a single stage. In the Preview panel, you can review the values for each record to determine if the stage transforms data as expected.

  1. Above the pipeline canvas, click the Preview icon: .
    If the Preview icon is disabled, check the Issues list for unconnected stages and required properties that are not defined.
  2. In the Preview Configuration dialog box, configure the following properties, then click Run Preview.
    Preview Property Description
    Preview Source Source data for the preview:
    • Configured Source - Provides data from the origin system.
    • Test Origin - Provides data from the test origin configured for the pipeline.
    • Snapshot Data - Uses available snapshot data. Available for pipelines only.
    Preview Batch Size Number of records to use in the preview. Honors values up to the Data Collector preview batch size.

    Default is 10. The Data Collector default is 10.

    Preview Timeout Milliseconds to wait for preview data. Use to limit the time data preview waits for data to arrive at the origin. Relevant for transient origins only.
    Write to Destinations and Executors Determines whether the preview passes data to destinations or executors.

    By default, does not pass data to destination or executor stages.

    Execute Pipeline Lifecycle Events Triggers the generation of any appropriate pipeline events, typically the Start event. If the event is configured to be used, event consumption is also triggered.
    Show Record/Field Header Displays record header attributes and field attributes when in List view. Attributes do not display in Table view.
    Show Field Type Displays the data type for fields in List view. Field types do not display in Table view.
    Snapshot Data When using a snapshot for source data, select the snapshot to use. Available for pipelines only.
    Remember the Configuration Stores the current preview configuration for use every time you request a preview for this pipeline.

    After you run data preview, you can change this option in the Preview panel by selecting the Preview Configuration icon () and clearing the option. The change takes effect the next time you run data preview.

    The Preview panel highlights the origin stage. Since this is the origin of the pipeline, no input data displays.
  3. To view data for the next stage, select the stage in the pipeline canvas.
  4. To exit data preview, click the Close Preview icon: .