Upgrade an Installation with Cloudera Manager

When you upgrade an installation with Cloudera Manager, the new version uses the same configuration, data, log, and resource directories. As a result, the new version has access to the files created in the previous version.

Note: If you installed external libraries or developed custom stages, verify that those libraries are stored in a local directory external to the Data Collector runtime directory before you upgrade. That way, Data Collector can still use the libraries after the upgrade.
To upgrade Data Collector through Cloudera Manager, perform the following steps:

Step 1. Stop All Pipelines

Step 2. Install the StreamSets Custom Service Descriptor

Step 3. Manually Install the Parcel and Checksum Files (Optional)

Step 4. Distribute and Activate the New StreamSets Parcel

Warning: You must perform the steps in this order, or Data Collector will fail to start.

Step 1. Stop All Pipelines

In Data Collector, stop all running pipelines.

  1. From the Home page, select all running pipelines in the list and then click the Stop icon.
    The Stop Pipeline Confirmation dialog box appears.
  2. Click Yes to stop the pipelines.

Step 2. Install the StreamSets Custom Service Descriptor

Install the new StreamSets custom service descriptor file (CSD), and then restart Cloudera Manager.

  1. Use the following URL to download the CSD from the StreamSets website: https://streamsets.com/opensource.
  2. Remove the previous StreamSets CSD file from Cloudera Manager.
    For example:
    rm -f /opt/cloudera/csd/STREAMSETS*.jar
  3. Copy the Data Collector CSD file to the Local Descriptor Repository Path. By default, the path is /opt/cloudera/csd.
    To verify the path to use, in Cloudera Manager, click Administration > Settings. In the navigation panel, select the Custom Service Descriptors category. Place the CSD file in the path configured for Local Descriptor Repository Path.
  4. Set the file ownership to cloudera-scm:cloudera-scm with permission 644.
    For example:
    chown cloudera-scm:cloudera-scm /opt/cloudera/csd/STREAMSETS*.jar
    chmod 644 /opt/cloudera/csd/STREAMSETS*.jar
  5. Use one of the following commands to restart Cloudera Manager Server:
    For Ubuntu 14.04, CentOS 6, or RedHat Enterprise Linux 6:
    service cloudera-scm-server restart
    For Ubuntu 16.04, CentOS 7, or RedHat Enterprise Linux 7:
    systemctl restart cloudera-scm-server
  6. In Cloudera Manager, to restart the Cloudera Management Service, click Home > Status. To the right of Cloudera Management Service, click the Menu icon and select Restart.

Step 3. Manually Install the Parcel and Checksum Files (Optional)

You can manually install the StreamSets parcel and related checksum files. Manually install the files when the Cloudera Manager Server does not have internet access.

When working with multiple clusters, perform the following steps for each cluster.

  1. Download the StreamSets parcel and related checksum file for the Cloudera Manager Server operating system from the following location:
  2. Copy the StreamSets parcel and checksum files to the Cloudera Manager Local Parcel Repository Path.
    By default, the path is /opt/cloudera/parcel-repo.
    To verify the path to use, click Administration > Settings. In the navigation panel, select the Parcels category. Place the StreamSets parcel file in the path configured for Local Parcel Repository Path.

Step 4. Distribute and Activate the New StreamSets Parcel

After you add the StreamSets repository to Cloudera Manager, you can download and distribute the new StreamSets parcel across the cluster. Stop the StreamSets service and deactivate the previous parcel before you activate the new parcel.

  1. To view the list of available parcels, in the menu bar, click the Parcels icon.

    The new StreamSets parcel displays in the list of available parcels. If it doesn't display, click Check for New Parcels.

  2. To download the new StreamSets parcel to the local repository, click Download.

    After the parcel is downloaded, the Download button becomes the Distribute button.

  3. To distribute the new StreamSets parcel to the cluster, click Distribute.
  4. To stop the StreamSets service, click Clusters > StreamSets and then click Actions > Stop.
  5. Click the Parcels icon to return to the Parcels page.
  6. To deactivate the previous StreamSets parcel, choose the appropriate cluster in the Location selector, and then click Deactivate for the parcel.
  7. To activate the new StreamSets parcel, choose the appropriate cluster in the Location selector, and then click Activate for the parcel.
  8. To restart the StreamSets service, click Clusters > StreamSets and then click Actions > Start.
    When you restart the StreamSets service, Cloudera Manager updates the Data Collector configuration properties for you. Cloudera Manager retains any customized values that you added in the previous Data Collector version. It also adds any new properties included in the new Data Collector version.