SFTP/FTP/FTPS Client

Supported pipeline types:
  • Data Collector

The SFTP/FTP/FTPS Client destination writes whole files to a URL using the Secure File Transfer Protocol (SFTP), File Transfer Protocol (FTP), or FTP Secure (FTPS) protocol.

When you configure the SFTP/FTP/FTPS Client destination, you specify the URL where the destination writes files on the remote server. The destination can create paths that do not exist. You also specify an expression for the file name to write and the action to take when the file already exists on the server.

If the server requires authentication, configure the credentials for the protocol you are using. For the SFTP protocol, the destination can require that the server be listed in a known hosts file. For the FTPS protocol, the destination can authenticate with the server using a client certificate and can authenticate the certificate from the FTPS server.

The destination can generate events for an event stream. For more information about the event framework, see Dataflow Triggers Overview.

Note: StreamSets has tested the stage with vsftpd 3.0.

Credentials

The SFTP/FTP/FTPS Client destination can use several methods to authenticate with the remote server. From the Credentials tab, configure the authentication required by the remote server.

Authentication options differ for each protocol:
  • For all protocols, select an authentication method to log in to the remote server. Choose the method based on the protocol and remote server requirements:
    • None - The stage does not authenticate with the server.
    • Password - The stage authenticates with the server using a user name and password. You must specify the user name and password.
    • Private key - The stage authenticates using a private key. Use only with the SFTP protocol. You must specify the private key, either in a local file or in plain text.
  • For the SFTP protocol, the stage can require that the server be listed in a known hosts file. You must specify the path to the known hosts file that contains the host keys for the approved SFTP servers.
  • For the FTPS protocol, the stage can use certificates to authenticate with the server. You must specify the keystore file and password. You can also configure the stage to authenticate the server by specifying a truststore provider. For more information about keystores and truststores, see Keystore and Truststore Configuration.

Event Generation

The SFTP/FTP/FTPS Client destination can generate events that you can use in an event stream. When you enable event generation, the destination generates event records each time the destination closes a file or completes streaming a whole file.

You can use events generated by the SFTP/FTP/FTPS Client destination in any logical way. For example:

For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.

Event Records

Event records generated by the SFTP/FTP/FTPS Client destination include the following event-related record header attributes. Record header attributes are stored as String values:

Record Header Attribute Description
sdc.event.type Event type. Uses one of the following types:
  • file-closed - Generated when the destination closes a file.
  • wholeFileProcessed - Generated when the destination completes streaming a whole file.
sdc.event.version Integer that indicates the version of the event record type.
sdc.event.creation_timestamp Epoch timestamp when the stage created the event.
The destination can generate the following types of event records:
File closure
The destination generates a file closure event record when it closes an output file.
File closure event records have the sdc.event.type record header attribute set to file-closed and include the following fields:
Field Description
filepath Absolute path to the closed file.
filename File name of the closed file.
length Size of the closed file in bytes.
Whole file processed
The destination generates an event record when it completes streaming a whole file. Whole file event records have the sdc.event.type record header attribute set to wholeFileProcessed and have the following fields:
Field Description
sourceFileInfo A map of attributes about the original whole file that was processed. The attributes include:
  • size - Size of the whole file in bytes.

Additional attributes depend on the information provided by the origin system.

targetFileInfo A map of attributes about the whole file written to the destination. The attributes include:
  • path - An absolute path to the processed whole file.

Data Format

The SFTP/FTP/FTPS Client destination writes data in the following data format:
Whole File
Streams whole files to the destination system. The destination writes the data to the file and location defined in the stage. If a file of the same name already exists, you can configure the destination to overwrite the existing file or send the current file to error.
By default, written files use the default access permissions for the destination system. You can specify an expression that defines access permissions.
For more information about the whole file data format, see Whole File Data Format.

Configuring an SFTP/FTP/FTPS Client Destination

Configure an SFTP/FTP/FTPS Client destination to send data to a URL using SFTP, FTP, or FTPS.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Produce Events Generates event records when events occur. Use for event handling.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the pipeline.

    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions.

    Records that do not meet all preconditions are processed based on the error handling configured for the stage.

    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline.
  2. On the SFTP/FTP/FTPS tab, configure the following properties:
    SFTP/FTP/FTPS Property Description
    Resource URL URL where the destination sends data on the remote server. Use the appropriate format:
    • SFTP protocol:
      sftp://<host name>:<port number>/<path>
    • FTP protocol:
      ftp://<host name>:<port number>/<path> 
    • FTPS protocol:
      ftps://<host name>:<port number>/<path> 

    You can omit the port number from the URL if the server uses the standard port number: 22 for SFTP, or 21 for FTP or FTPS.

    You can optionally include the user name to log in to the SFTP, FTP, or FTPS server in the URL. For example, for the FTP protocol, you can use the following format:
    ftp://<user name>:<password>@<host name>/<path>

    You can enter an email address as a user name.

    Note: If you enter a user name in the resource URL and configure password or private key authentication on the Credentials tab, the value entered in the URL takes precedence.
    Path Relative to User Home Directory Interprets the path entered in the resource URL as relative to the home directory of the user that logs in to the remote server.

    You specify the user name in the URL or when you configure password or private key authentication on the Credentials tab.

    Create Path Creates the specified path on the remote server when the path does not exist.
    FTPS Mode Encryption negotiation mode to use for the FTPS protocol:
    • Implicit - Uses encryption immediately.
    • Explicit - Uses plain FTP to connect to the server and then negotiates encryption with the server.
    FTPS Data Channel Protection Level Protection level to use for the FTPS data channel:
    • Clear - Encrypts only communication with the server, not data sent to the server.
    • Private - Encrypts both communication with the server and data sent to the server.
  3. On the Credentials tab, configure the following properties:
    Credentials Property Description
    Authentication Authentication method to use to log in to the remote server:
    • None - Does not authenticate with remote server.
    • Password - Authenticates with the remote server using a user name and password.
    • Private key - Authenticates with an SFTP server using a private key.

    Default is None.

    Username User name to log in to the remote server.

    Available for password and private key authentication.

    Password Password to log in to the remote server.

    Available for password authentication.

    Tip: To secure sensitive information such as user names and passwords, you can use runtime resources or credential stores.
    Private Key Provider Source that provides the private key:
    • File - Reads the private key from a local file.
    • Plain-Text - Reads the private key from a plain-text field.

    Available when using private key authentication.

    Private Key File Full path to the local private key file used to log in to the remote server.

    Available for private key authentication when the provider is a file.

    Private Key Private key used to log in to the remote server.

    Available for private key authentication when the provider is plain text.

    Private Key Passphrase Passphrase used to open the private key.

    Available for private key authentication if the private key is protected with a passphrase.

    Strict Host Checking Requires that the SFTP server is listed in the known hosts file. When enabled, the destination connects to the server only if the server is listed in the known hosts file.

    Requires the known hosts file to include an RSA key.

    Only used for the SFTP protocol.

    Known Hosts File Full path to the local known hosts file. Required if strict host checking is selected.

    Available when using strict host checking.

    Use Client Certificate for FTPS Authenticates with the FTPS server using a client certificate.

    Select this option when the FTPS server requires mutual authentication. You must provide a keystore file that contains the client certificate.

    Only used for the FTPS protocol.

    FTPS Client Certificate Keystore File Full path to the keystore file that contains the client certificate.

    Available when using a client certificate for FTPS.

    FTPS Client Certificate Keystore Type Type of keystore file that contains the client certificate.

    Available when using a client certificate for FTPS.

    FTPS Client Certificate Keystore Password Password to the keystore file that contains the client certificate. A password is optional, but recommended.
    Tip: To secure sensitive information such as user names and passwords, you can use runtime resources or credential stores.

    Available when using a client certificate for FTPS.

    FTPS Truststore Provider Method that the destination uses to authenticate the certificate from the FTPS server:
    • Allow All - Allows any certificate, skipping authentication.
    • File - Authenticates certificate with a specified truststore file.
    • JVM Default - Authenticates certificate with the JVM default truststore.

    Only used for the FTPS protocol.

    FTPS Truststore File Full path to the truststore file that contains the server certificate.

    Available when using a file as the FTPS truststore provider.

    FTPS Truststore Type Type of truststore:
    • Java Keystore file (JKS)
    • PKCS-12 (p12 file)

    Available when using a file as the FTPS truststore provider.

    FTPS Truststore Password Password to the truststore file. A password is optional, but recommended.
    Tip: To secure sensitive information such as user names and passwords, you can use runtime resources or credential stores.

    Available when using a file as the FTPS truststore provider.

  4. On the Data Format tab, configure the following properties:
    Whole File Property Description
    Data Format Format of data to be written. The destination uses the whole file data format.
    File Name Expression

    Expression to use for the file names.

    For tips on how to name files based on input file names, see Writing Whole Files.

    File Exists Action to take when a file of the same name already exists in the output directory. Use one of the following options:
    • Send to Error - Handles the record based on stage error record handling.
    • Overwrite - Overwrites the existing file.