WebSocket Server
The WebSocket Server origin is a multithreaded origin that listens on a WebSocket endpoint and processes the contents of all authorized WebSocket client requests.
The WebSocket Server origin can use multiple threads to enable parallel processing of data from multiple WebSocket clients. The origin can also send responses back to the origin system when used in a microservice pipeline.
Before you configure the origin, perform additional steps to configure the WebSocket clients.
When you configure the WebSocket Server origin, you specify the maximum number of concurrent requests to determine how many threads to use. You define the listening port, application ID, and the maximum message size. You can also configure SSL/TLS properties, including default transport protocols and cipher suites.
When you want the origin to send responses back to the origin system as part of a microservice pipeline, you configure the data format and character set for the responses.
Prerequisites
Before you run a pipeline with the WebSocket Server origin, complete the following prerequisites to configure the WebSocket clients.
Send Data to the Listening Port
Configure the WebSocket clients to send data to the WebSocket Server listening port.
When you configure the WebSocket Server origin, you define a listening port number where the origin listens for data. To pass data to the pipeline, configure each WebSocket client to send data to a URL that includes the listening port number.
<ws | wss>://<sdc_hostname>:<listening_port>/
The URL includes the following components:
- <ws | wss> - Use wss for secure WebSocket connections over HTTPS.
- <sdc_hostname> - The Data Collector host name.
- <listening_port> - The port number where the origin listens for data.
For example: wss://localhost:8080/
Include the Application ID in Requests
Configure the WebSocket clients to include the WebSocket Server application ID in each request.
When you configure the WebSocket Server origin, you define an application ID that is used to pass requests to the origin. All messages sent to the origin must include the application ID.
Include the application ID for each client request in one of the following ways:
- In request headers
- Add the following information to the request header for all WebSocket requests
that you want the origin to
process:
X-SDC-APPLICATION-ID: <application_ID>
- In a query parameter in the URL
- If you cannot configure the client request headers - for example if the requests are generated by another system - then configure each WebSocket client to send data to a URL that includes the application ID in a query parameter.
Multithreaded Processing
The WebSocket Server origin performs parallel processing and enables the creation of a multithreaded pipeline.
The WebSocket Server origin uses multiple concurrent threads based on the Max Concurrent Requests property. Each thread connects to the origin system, creates a batch of data, and passes the batch to an available pipeline runner.
A pipeline runner is a sourceless pipeline instance - an instance of the pipeline that includes all of the processors and destinations in the pipeline and represents all pipeline processing after the origin. Each pipeline runner processes one batch at a time, just like a pipeline that runs on a single thread. When the flow of data slows, the pipeline runners wait idly until they are needed, generating an empty batch at regular intervals. You can configure the Runner Idle Time pipeline property to specify the interval or to opt out of empty batch generation.
Multithreaded pipelines preserve the order of records within each batch, just like a single-threaded pipeline. But since batches are processed by different pipeline instances, the order that batches are written to destinations is not ensured.
For example, say you set the Max Concurrent Requests property to 5. When you start the pipeline, the origin creates five threads, and Data Collector creates a matching number of pipeline runners. Upon receiving data, the origin passes a batch to each of the pipeline runners for processing. In the batch, WebSocket Server includes only the WebSocket requests with the specified application ID.
At any given moment, the five pipeline runners can each process a batch, so this multithreaded pipeline processes up to five batches at a time. When incoming data slows, the pipeline runners sit idle, available for use as soon as the data flow increases.
For more information about multithreaded pipelines, see Multithreaded Pipeline Overview.
Generated Microservice Responses
Unlike standard origins, the WebSocket Server origin can receive records from microservice destinations in a microservice pipeline. When the pipeline is configured to use the Send Response to Origin pipeline error handling, the origin also receives all pipeline error records. The origin generates a response with all of the records that it receives from the pipeline and sends the response back to the originating WebSocket client.
Just as each request that the WebSocket Server origin receives is processed as a batch, the records from the batch that are passed to the origin from the pipeline are incorporated into a single response.
In the generated response, records that were passed by
microservice destinations in the pipeline are included as a list in a
data
attribute. Error records passed by the Send
Response to Origin pipeline error handling are included as a list in an
error
attribute.
{
"httpStatusCode":<status code>,
"data":[<list of success records>],
"error":[<list of error records>],
"errorMessage": <error message, if any>
}
Response Attribute | Value |
---|---|
httpStatusCode | The status code associated with the records in the
response. If the records in the generated response share the same status code, the code is written to the httpStatusCode attribute. If the records have different status codes, the httpStatusCode is set to 207 for multiple statuses. |
data | The records passed to the origin by the microservice destinations used in the pipeline. |
error | The pipeline error records passed to the origin by the Send Response to Origin pipeline error handling. |
errorMessage | The error message associated with the first error
record in the response. Used only when the response includes error records. |
Sample Responses
{
"httpStatusCode":200,
"data":[{"ID":"103","NAME":"Jack","AGE":"37","STATE":"MD"}],
"error":[],
"errorMessage":null
}
The
following response includes several records, data and error. Since the records have
different status codes associated with them, the httpStatusCode is 207 for multiple
statuses. The errorMessage attribute includes the error associated with the first record
which has a missing ID:
{
"httpStatusCode":207,
"data":[{"ID":"248","NAME":"Pina","AGE":"24","STATE":"RI"}],
"error":[{"NAME":"Liz","AGE":"37","STATE":"DE"}, {"ID":"302","NAME":"Roco","AGE":"","STATE":"CA"}],
"errorMessage":"COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:exists('/ID')}'"
}
Data Formats
The WebSocket Server origin processes data differently based on the data format that you select. The origin processes the following types of data:
- Binary
- Generates a record with a single byte array field at the root of the record.
- Datagram
- Generates a record for every message. The origin can process collectd messages, NetFlow 5 and NetFlow 9 messages, and the following types of syslog messages:
- Delimited
- Generates a record for each delimited line. You can use the
following delimited format types:
- Default CSV - File that includes comma-separated values. Ignores empty lines in the file.
- RFC4180 CSV - Comma-separated file that strictly follows RFC4180 guidelines.
- MS Excel CSV - Microsoft Excel comma-separated file.
- MySQL CSV - MySQL comma-separated file.
- PostgreSQL CSV - PostgreSQL comma-separated file.
- PostgreSQL Text - PostgreSQL text file.
- Tab-Separated Values - File that includes tab-separated values.
- Custom - File that uses user-defined delimiter, escape, and quote characters.
- JSON
- Generates a record for each JSON object. You can process JSON files that include multiple JSON objects or a single JSON array.
- Log
- Generates a record for every log line.
- Protobuf
- Generates a record for every protobuf message. By default, the origin assumes messages contain multiple protobuf messages.
- SDC Record
- Generates a record for every record. Use to process records generated by a Data Collector pipeline using the SDC Record data format.
- Text
- Generates a record for each line of text or for each section of text based on a custom delimiter.
- XML
- Generates records based on a user-defined delimiter element. Use an XML element directly under the root element or define a simplified XPath expression. If you do not define a delimiter element, the origin treats the XML file as a single record.
Configuring a WebSocket Server Origin
Configure a WebSocket Server origin to generate multiple threads for parallel processing of WebSocket client requests.