Cloudera Cluster Upgrade Planning

Plan your Cloudera Cluster Upgrade effectively

Introduction

Every Hadoop Administrator does cluster upgrades as part of their maintenance activity. A well-documented, top-level approach is hard to come across, and hence below is my attempt at blogging one down. Here, I will break down the entire upgrade into a modular approach. A modular approach gives much more control over the entire activity and helps validate the upgrade at each step.

Almost every upgrade activity is majorly broken down into the following steps:

1. Pre-Upgrade

2. Upgrade

3. Post-Upgrade

Pre-Upgrade

1. Check Compatibility with a new release

   One of the biggest challenges while upgrading to a new major software platform release is its compatibility. You should go through official current and targeted release documentation to get compatibility details.

2. Major changes Introduced

   This is very much needed as new features need code changes in jobs. Some of the cluster monitoring metrics are not available.

3. How the existing and properly running jobs will behave

   Official documentation gives a pretty detailed understanding of what new features are introduced and which features need to be deprecated/removed. Have a discussion with the development team so that they are aware of the new changes.
   If possible, create some sample jobs which you can test after the upgrade and before handing over a cluster to end-users to run their jobs.

4. Review earlier design assumptions

   Considering you are doing a rolling upgrade but the cluster services HA is not properly tested. This can prove to be a nightmare. HA validation from infrastructure-related services like Kerberos KDC, Nagios setup, etc., is necessary.

5. Pre-Upgrade Steps for CM and CDH

   We cannot yet hit the upgrade button yet without making sure that there is no need to run some pre-upgrade steps. Again, the official documentation will help here. This step is crucial during a major upgrade. e.g., Upgrading from CDH5 to CDH6. Typically during minor upgrades (i.e., CDH 5.14 → CDH 5.16), major changes across existing services and configuration do not happen. But during a major upgrade, some services may be deprecated, sometimes even removed/replaced by other services/components. So in-depth research about such changes is much needed.

6. Backup

   It is the most important task and saves a lot of time if something goes unplanned.

   Some of the files that need a backup are configuration files, database/tables, etc.,

7. Incorporate all the above inputs, create a detailed upgrade plan, and get it reviewed by one of your senior peers. Revisit the document and make necessary changes as per the peer-review feedback.

8. Once the plan is internally reviewed and considering you have a valid subscription to Cloudera, open a pro-active case for upgrade activity and get Cloudera support opinion on compatibility and necessary pre-upgrade steps. If there are any changes, update the upgrade plan document and freeze it.
   
   

Upgrade

1. Operating systems (OS)

   This may or may not be needed depending on what version you are currently using and the targeted version.
   Make sure certain packages are not updated (e.g., python and java versions depend on many Hadoop services) depending upon compatibility.

2. Cloudera Manager (CM)
   This may or may not be needed depending on what version you are currently using and the targeted version. Consider upgrading or keeping a database that can be used for CM.

3. Cloudera Distribution Hadoop (CDH)

   Upgrade components like CDSW, Kafka, Spark2 (there could be a few others as well), etc., as they are available through separate parcels in CDH5. Once the upgrading activity is completed, stop the entire cluster and start it to ensure that services are coming up without any errors/warnings. In case of any errors/warnings, review and fix them.

Post-Upgrade

1. Make sure that everything works as smoothly as the pre-upgrade cluster state.

1. Smoke testing each and every cluster service
   This makes sure that the services are coming up and running after the upgrade. Perform a few basic operations so that it gives you confidence that it's working as expected.

2. Smoke test applications/jobs
   This makes sure that the cluster’s end-users are happy and are lauding this challenging upgrade activity that you just finished. If it’s possible to run actual jobs, please do so.

As stated, this is more of a top-level approach on how to kick start the cluster upgrade planning. This blog does not provide low-level details about what to do and how to perform each step. But it’s more of an approach or a thinking process guideline to help you plan the cluster upgrade, so there are very few Oh God! moments. Furthermore, learn how to upgrade the Cloudera CDH cluster to CDP Private Cloud Base with our blog here.

To get the best data engineering solutions for your business, reach out to us at Clairvoyant.