Couchbase Lookup

Supported pipeline types:
  • Data Collector

The Couchbase Lookup processor looks up documents in Couchbase Server and returns values to fields in the record. Use the Couchbase Lookup processor to enrich records with additional data.

For example, suppose Couchbase Server has multiple department documents, each listing the employees in a department. You can configure a pipeline to store the Couchbase document key, which uniquely identifies each department, in a record header attribute. Then, include a Couchbase Lookup processor to find the matching document and return the values to a new department_employees field in the record.

The Couchbase Lookup processor can look up documents using the document key or the Couchbase Server query language, N1QL. For key lookups, the processor can return the data from the entire document to a specified map field. Alternatively, for both key and N1QL lookups, the processor can return data from sub-documents to specified map fields. For N1QL lookups, when a lookup results in multiple matched documents, the Couchbase Lookup processor can return values from the first matching document or return values from all matching documents in separate records.

When you configure the Couchbase Lookup processor, you enter connection information, such as the nodes and bucket to connect to, as well as timeout properties for the connection. Optionally, you can enable TLS for the connection. You also enter information to authenticate with Couchbase Server.

Record Header Attributes

For key/value lookups, the Couchbase Lookup processor creates a record header attribute, couchbase.cas, that stores a value that represents the state of the looked-up document.

When configured to use CAS (compare and swap), the Couchbase destination uses this attribute value to prevent conflicts with other processes.

Configuring a Couchbase Lookup Processor

Configure a Couchbase Lookup processor to look up data in Couchbase Server.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the pipeline.

    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions.

    Records that do not meet all preconditions are processed based on the error handling configured for the stage.

    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline. Not valid for cluster pipelines.
  2. On the Couchbase tab, configure the following properties:
    Couchbase Property Description
    Node List One or more nodes in a Couchbase cluster, separated by commas.
    Bucket Name of an existing Couchbase bucket to connect to.
    Key-Value Timeout (ms) Maximum number of milliseconds allowed to execute each key-value operation.
    Connect Timeout (ms) Maximum number of milliseconds allowed to connect to Couchbase Server.
    Disconnect Timeout (ms) Maximum number of milliseconds allowed to gracefully close a connection.
    Advanced Environment Settings Client settings for connections with Couchbase Server. For available settings, see the Couchbase Java SDK documentation.
    Use TLS Enables the use of TLS.
    Keystore File Path to the keystore file. Enter an absolute path to the file or a path relative to the Data Collector resources directory: $SDC_RESOURCES.

    For more information about environment variables, see Data Collector Environment Configuration.

    By default, no keystore is used.

    Keystore Type Type of keystore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).

    Keystore Password Password to the keystore file. A password is optional, but recommended.
    Tip: To secure sensitive information such as passwords, you can use runtime resources or credential stores.
    Keystore Key Algorithm The algorithm used to manage the keystore.

    Default is SunX509.

    Truststore File The path to the truststore file. Enter an absolute path to the file or a path relative to the Data Collector resources directory: $SDC_RESOURCES.

    For more information about environment variables, see Data Collector Environment Configuration.

    By default, no truststore is used.

    Truststore Type Type of truststore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).

    Truststore Password Password to the truststore file. A password is optional, but recommended.
    Tip: To secure sensitive information such as passwords, you can use runtime resources or credential stores.
    Truststore Trust Algorithm The algorithm used to manage the truststore.

    Default is SunX509.

  3. On the Credentials tab, configure the following properties:
    Credentials Property Description
    Authentication Mode Method to authenticate with Couchbase Server:
    • Bucket Authentication - Authenticates using the bucket password. Use for Couchbase Server 4.x and earlier.
    • User Authentication - Authenticates using a Couchbase user name and password. Use for Couchbase Server 5.0 and later.
    Bucket Password Password to access the bucket if the bucket is secured in the Couchbase database.
    Tip: To secure sensitive information such as user names and passwords, you can use runtime resources or credential stores.

    Available for bucket authentication.

    User Name Couchbase user name.

    Available for user authentication.

    Password Couchbase password.
    Tip: To secure sensitive information such as user names and passwords, you can use runtime resources or credential stores.

    Available for user authentication.

  4. On the Lookup tab, configure the following properties:
    Lookup Property Description
    Lookup Type Method to specify lookup:
    • Key/Value - Uses the document key to find a document.
    • N1QL - Uses the Couchbase Server query language to find a document.
    Document Key Unique ID or key for the document that the processor looks up. For example, you might specify an expression that resolves to the document key.

    Available for key/value lookups.

    N1QL Query Query that returns a document. Specify in N1QL, the Couchbase Server query language. For more information, see the Couchbase documentation.

    Available for N1QL lookups.

    Return Properties Returns specific sub-documents rather than the full document.

    Available for key/value lookups.

    Property Mappings List that maps returned sub-documents to fields in the record.
    Enter the following:
    • Property Name - Sub-document returned. Use dot notation syntax to separate components in the document hierarchy. For more information, see the Couchbase documentation.
    • SDC Field - Name of the map field in the record where the processor returns the sub-document.

    Available for key/value lookups when returning sub-documents and for N1QL lookups.

    SDC Field Name of the map field in the record where the processor returns the data from the lookup. You can specify an existing field or a new field. If the field does not exist, the Couchbase Lookup processor creates the field.

    Available for key/value lookups when not returning sub-documents.

    Submit as Prepared Statement Submits the query to Couchbase as a prepared statement.

    Available for N1QL lookups.

    Query Timeout (ms) Maximum number of milliseconds allowed for Couchbase Server to complete the query.

    Available for N1QL lookups.

    Multiple Value Behavior Action to take when the lookup finds multiple documents:
    • First value only - Returns values from the first document.
    • Split into multiple records - Returns values from each document to a separate record.

    Available for N1QL lookups.

    Missing Value Behavior Action to take when the lookup returns no document:
    • Send to error - Sends the record to error.
    • Pass the record along the pipeline unchanged - Passes the record without a lookup return value.