Understanding Protection Policies

Protection policies alter and protect data. When you configure a job, you specify a read policy to protect data when reading from an origin system. You also specify a write policy to protect data when writing to destination systems.

When you do not specify a read or write policy, default policies are used to ensure the protection of sensitive data. You can designate one default read policy and one default write policy for the organization.

Protection policies use classification rules to identify the data to be altered and protected. Protection policies can protect data based on both StreamSets classification rules and custom classification rules. When necessary, policies can also protect unclassified data in known field paths.

When you configure a protection policy, you specify the enactment type, read or write, then select the sampling to perform. You can configure the policy to catch unprotected records. When catching unprotected records, the policy routes records with classified but unprotected fields to a security violation destination. Classified unprotected fields might exist in records because the policy deliberately leaves them visible for use, or they might also exist because a policy fails to properly protect the classified data.

When you write records with classified unprotected fields to a security violation destination, you can then review the problematic records and update protection policies to address outstanding issues. This helps to ensure that unprotected sensitive data does not become accessible to unauthorized users.

After you create a protection policy, you must configure the procedures for the policy to perform. Procedures define the sensitive data to protect, and how to protect that data.

For example, you have sensitive data that includes company IDs, social security numbers, and salaries, and you want that data obfuscated when written to less-secure systems. To do this, you create a write policy to be used for jobs that write to those systems. In the write policy, you configure procedures to alter and protect all sensitive data, including one for company IDs, one for social security numbers, and one for salaries. Those procedures are based on custom classification rules that identify company IDs and salaries, and the StreamSets classification rule for social security numbers.

It's important to test all policies and classification rules to verify that they work as expected before using them in production jobs. For more information about testing, see Testing Rules and Policies.

For more information about how protection policies function as part of Data Protector, see Data Protection in Control Hub.

Working with Protection Policies

The Protection Policies view displays a list of all protection policies in the organization. When viewing policy details, you can also view information about the procedures in the policy.

You can complete the following tasks in the Protection Policies view:

  • Create protection policies - Create new policies to alter and protect sensitive data.
  • View protection policy details - Display policy details such as the sampling type and security violation destination. Also displays all procedures included in the policy.
  • Edit protection policies - Edit policies to update policy configuration. To edit a policy, click the policy name to view policy details, then click Edit.
  • Set default protection policies - The organization uses one default read and default write policy. To set a policy as the default, click the policy name to view policy details, then click Set Default.
  • Delete protection policies - Delete policies that are no longer needed. To delete a policy, click the policy name to view policy details, then click Delete.
    Note: The policy and its procedures are immediately deleted and cannot be recovered.
  • Work with procedures - Procedures specify how data is protected by a policy. When you view policy details, you can view all existing procedures and perform the following tasks:
    • Configure a procedure - Create or edit procedures as needed.
    • View procedure details - To view the details of a specific procedure, click Edit for the procedure you want to view.
    • Delete a procedure - Delete a procedure when it is no longer needed. To delete a procedure, click Delete for the procedure you want to remove.
      Note: The procedure is immediately deleted and cannot be recovered.

The following image shows a list of policies in the Protection Policies view. Each policy is listed with the policy name, enactment and sampling types, and the last modified time. When you click the policy name, additional details about the policy display, including the procedures in the policy: