Delimited Data Root Field Type

When reading delimited data, Data Collector can create records that use the list or list-map root field type.

When Data Collector creates records for delimited data, it creates a single root field of the specified type and writes the delimited data within the root field.

Use the default list-map root field type to easily process delimited data.

List-Map
Provides easy use of field names or column positions in expressions. Recommended for all new pipelines.
A list-map root field type results in a structure that preserves the order of data, as follows:
/<first header>:<value>
/<second header>:<value>
/<third header>:<value>
...

For example, with the list-map root field type, the following delimited rows:

TransactionID,Type,UserID
0003420303,04,362
0003420304,08,1008
are converted to records as follows:
/TransactionID: 0003420303
/Type: 04
/UserID: 362

/TransactionID: 0003420304
/Type: 08
/UserID: 1008
If data does not include a header or if you choose to ignore a header, list-map records use the column position as a header as follows:
0: <value>
1: <value>
2: <value>
For example, when you ignore the header for the same data, you get the following records:
0: 0003420303
1: 04
2: 362

0: 0003420304
1: 08
2: 1008
In an expression, you can use the field name or the column position with a standard record function to call a field. For example, you can use either of the following record:value() expressions to return data in the TransactionID field:
${record:value('/TransactionID')}
${record:value('[0]'}
Note: When writing scripts for scripting processors, such as the Jython Evaluator or JavaScript Evaluator, you should treat list-map records as maps.
For more information about standard record functions, see Record Functions.
List
Provides continued support for pipelines created before version 1.1.0. Not recommended for new pipelines.
A list root field type results in list with an index for the header position and a map with each header and associated value, as follows:
0
   /header = <first header>
   /value = <value for first header>
1
   /header = <second header>
   /value = <value for second header>
2  
   /header = <third header>
   /value = <value for third header>
...

For example, the same delimited rows described above are converted to records as follows:

0
   /header = TransactionID
   /value = 0003420303
1
   /header = Type
   /value = 04
2
   /header = UserID
   /value = 362

0
   /header = TransactionID
   /value = 0003420304
1
   /header = Type
   /value = 08
2
   /header = UserID
   /value = 1008
If the data does not include a header or if you choose to ignore a header, the list records omit the header from the map as follows:
0
   /value = <value>
1
   /value = <value>
2
   /value = <value>
...
For example, when you ignore the header for the same sample data, you get the following records:
0
   /value = 0003420303
1
   /value = 04
2
   /value = 362

0
   /value = 0003420304
1
   /value = 08
2
   /value = 1008
For data in the list records, you should either use the delimited data functions or include the full field path in standard record functions. For example, you can use the record:dValue() delimited data function to return the value associated with the specified header.
Tip: You can use the record:dToMap() function to convert a list record to a map, and then use standard functions for record processing.

For more information about record:dToMap and full list of delimited data record functions and their syntax, see Delimited Data Record Functions.