Data Preview
Data Preview Overview
You can preview data to help build or fine-tune a pipeline. When using Control Hub, you can also use data preview when developing pipeline fragments.
You can use data preview with complete or incomplete pipelines and fragments. And you can choose from several options to provide source data for the preview.
When you preview data, source data passes through the pipeline or fragment, allowing you to review how the data passes and changes through each stage. You can edit stage properties and run the preview again to see how your changes affect the data. You can also edit preview data to test and tune the pipeline logic.
You can preview data for one stage at a time or for a group of stages. You can also view the data in list or table view, and refresh the preview data.
Data Preview Availability
You can preview complete and incomplete pipelines and Control Hub pipeline fragments. The Data Preview icon becomes active when data preview is available.
- All stages in the pipeline are connected
- All required properties are defined
Source Data for Data Preview
- Data from the origin - Use available data from the origin.
- Data from the test origin - Use data from the test origin configured in the pipeline or fragment properties.
- Data from a snapshot - Use snapshot data from the same pipeline or another pipeline. Available for pipelines only.
Writing to Destinations and Executors
Since data preview is a tool for development, by default, it does not write data to destination systems or pass data to executors connected to destination stages.
Data preview also does not display the data that is written by destinations in the pipeline. You can, however, view the data that is passed to a destination stage, which is typically similar to what is written to destination systems.
If you like, you can configure the preview to write data to destination systems and to trigger executors connected to destination stages. For example, you might enable writing to an executor to verify that it performs the configured task as expected.
To write to destination systems and connected executors, in the Preview Configuration dialog box, select Write to Destinations and Executors.
Notes
- Date, datetime, and time data - Data preview displays date, datetime, and time data using the default format of the browser locale. For example, if the browser uses the en_US locale, preview displays dates using the following format: MMM d, y h:mm:ss a.
- Oracle CDC Client pipelines - When previewing a pipeline that uses the Oracle CDC Client origin, data preview might time out before connecting to the origin system. When this occurs, try increasing the timeout to 120,000 milliseconds to allow the origin time to connect.
- Whole file data format - When previewing a pipeline that processes whole file data, data preview displays only one record.
Data Collector UI - Preview Mode
You can use Data Collector to view how data passes through the pipeline.
The following image shows Data Collector in Preview mode:

Area/Icon | Name | Description |
---|---|---|
1 | Pipeline canvas | Displays the pipeline. |
2 | Preview panel |
Displays the data that enters and exits the selected stage or group of stages. It can also display stage properties and preview configuration. |
![]() |
StreamSets Control Hub icon | Provides information about StreamSets Control Hub and lets you register this Data Collector with Control Hub. |
![]() |
Home icon | Displays a home page with a list of pipelines and their statuses, allowing you to perform pipeline maintenance and navigate to individual pipelines. |
![]() |
Package Manager icon | Displays the Package Manager which allows you to install additional stage libraries for a core or common Data Collector installation. |
![]() |
Notifications icon | Displays notifications. |
![]() |
Administration icon | Provides access to Data Collector configuration properties, directories, and log. Also allows you to shut down Data Collector. |
![]() |
User icon | Displays the active user and the roles assigned to the user. Also allows you to log out of Data Collector. |
![]() |
Help icon | Provides context-sensitive help based on the information in the panel. Allows you
to configure display settings and to specify whether to use a local or hosted version of
the help. Provides access to the REST API and the Data Collector version. |
![]() |
Link to a pipeline list | Link to a pipeline list on the Home page. Use to view a list of available pipelines, perform pipeline maintenance like starting or sharing a pipeline, and navigate to individual pipelines. |
![]() |
Records icon | Displays data preview records. |
![]() |
Stage Configuration icon | Displays stage properties. |
![]() |
Preview Configuration icon | Displays data preview properties. |
![]() |
Single button | Displays input and output data for a single stage. |
![]() |
Multiple button | Displays input and output data for a group of stages. |
![]() |
List View icon | Displays preview data in a list. |
![]() |
Table View icon | Displays preview data in a table. |
![]() |
Previous Stage icon | Moves the preview to the previous stage. |
![]() |
Next Stage icon | Moves the preview to the next stage. |
![]() |
Refresh Preview icon | Provides a fresh set of data from the origin. |
![]() |
Run Preview with Changes icon | Runs the data preview using changed data. Use to see how edited data is processed by the pipeline. |
![]() |
Revert Changes icon | Reverts all changes to preview data and returns the preview to the origin. |
![]() |
Close Preview icon | Closes the data preview. |
For information about maintaining pipelines on the Home page, see Data Collector UI - Pipelines on the Home Page.
For information about configuring pipelines, see Data Collector UI - Edit Mode.
For information about pipeline monitoring options, see Data Collector UI - Monitor Mode.
Preview Codes
Data preview displays different colors for different types of data. Preview also uses other codes and formatting to highlight changed fields.
Preview Code | Description |
---|---|
Black values | Date data |
Blue values | Numeric data |
Green values | String data |
Red values | Boolean data |
Asterisk | Records that include edited field values |
Red italic field labels | Fields that contain edited data |
Light red background | Fields removed by a stage |
Italic values | Edited data |
Green stage | First stage in a multiple-stage preview |
Red stage | Last stage in a multiple-stage preview |
Previewing a Single Stage
You can preview data for a single stage. In the Preview panel, you can review the values for each record to determine if the stage transforms data as expected.
Previewing Multiple Stages
You can preview data for a group of linked stages within a pipeline.
When you preview multiple stages, you select the first stage and the last stage in the group. The Preview panel then displays the output data of the first stage in the group and the input data of the last stage in the group.
In the Preview panel, you can review the values for each record to determine if the group of stages transforms data as expected.
Editing Preview Data
You can edit preview data to view how a stage or group of stages processes the changed data. Edit preview data to test for data conditions that might not appear in the preview data set.
For example, when the stage filters integer data based on an expression, you might change the input data to test positive and negative integer values, as well as zero.
- The output data column for an origin.
- The input data column for processors.
When you edit preview data, you can pass the changed data through the pipeline, or you can revert your changes to return to the original data.
Editing Properties
In data preview, you can edit stage properties to see how the changes affect preview data. For example, you might edit the expression in an Expression Evaluator to see how the expression alters data.
When you edit properties, you can test the change with the existing preview data or you can refresh the preview data.