Field Flattener

The Field Flattener flattens list and map fields. The processor can flatten the entire record to produce a record with no nested fields. Or it can flatten specific list or map fields.

Use the Field Flattener when you have nested fields that you need to flatten. For example, the Drift Synchronization Solution for Hive cannot process records with nested fields, so you can use the Field Flattener to flatten records before passing them to the Hive Metadata processor.

When you configure a Field Flattener, you configure whether to flatten the entire record or specific fields in the record. You also configure the name separator to use for flattened field names.

Flatten the Entire Record

When the Field Flattener flattens the entire record, it flattens all nested structures in the record until the record is flat.

For example, say you have the following record with nested map fields:
{
  "store": {
     "id": "10342",
     "location": {
         "street": "34 2nd St",
         "city": "Wilma",
         "state": "OH",
         "zipcode": "33333"
      },
     "ip": "234.56.7890"
  }
}
If you configure Field Flattener to flatten the entire record and to use a period as the name separator, the processor generates the following record:
store.id store.location.street store.location.city store.location.state store.location.zipcode store.ip
10342 34 2nd St Wilma OH 33333 234.56.7890

Flatten Specific Fields

The Field Flattener can flatten specific list and map fields that contain additional nested list or map fields. When the processor flattens a list or map field, it flattens all nested structures in the field until the field is flat.

For example, say you have the following record with nested map fields:

{
  "contact": {
     "name": "Jane Smith",
     "id": "557",
     "address": {
       "home": {
         "street": "101 3rd St",
         "city": "Huntsville",
         "state": "NC",
         "zipcode": "27023"
          },
       "work": {
         "street": "15 Main St",
         "city": "Jonestown",
         "state": "NC",
         "zipcode": "27011"
       }
      }
  }
}

If you configure Field Flattener to flatten the /contact/address map field and to use a period as the name separator, the processor generates the following record:

{
  "contact": {
     "name": "Jane Smith",
     "id": "10342",
     "address": {
         "home.street": "34 2nd St",
         "home.city": "Wilma",
         "home.state": "OH",
         "home.zipcode": "33333",
         "work.street": "15 Main St",
         "work.city": "Jonestown",
         "work.state": "OH",
         "work.zipcode": "55555"
      }
  }
}

Configuring a Field Flattener

Configure a Field Flattener to flatten fields.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the pipeline.

    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions.

    Records that do not meet all preconditions are processed based on the error handling configured for the stage.

    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline. Not valid for cluster pipelines.
  2. On the Flatten tab, configure the following properties:
    Field Flattener Property Description
    Flatten Select whether to flatten the entire record or to flatten specific list or map fields.
    Fields List or map fields to flatten. You can flatten list and map fields that contain additional nested list or map fields. Enter the path to the field, for example: /contact/address.

    Click Add to specify additional fields to flatten.

    Name Separator One or more characters to use between nested field names to create a flattened field name.

    For example, if you use the underscore character, a location field nested in a store field is flattened as follows: store_location.