Upgrading Cloudera Distribution Hadoop and Cloudera Manager from 5.7 to 5.16 in a Docker container
Pre-requisites
1. Docker Installed 2. Cloudera Image container running
Follow this link if the pre-requisites are not met.
To use Spark2 in Cloudera, we require CDH 5.9 and above, and for CDH 5.9 you will require a compatible version of Cloudera Manager.
-
Upgrading Cloudera Manager (CM) version from 5.7 → 5.16.x.
-
Upgrading CDH (Cloudera Distribution Hadoop) version from 5.7 TO 5.16.x.
Upgrading Cloudera Manager (CM) version
Let’s first check the current version of Cloudera Manager. Go to Home → Support → About.
Open up a Docker Bash terminal and go to the directory /etc/yum.repo.d/.
cd /etc/yum.repo.d/
List of configuration files available in the /etc/yum.repo.d/ Directory.
ls -al
Open the “cloudera-manager.repo” file using the vi editor. To check the baseurl source available for Cloudera Manager, we can use an archive link as well as a local copy of the Cloudera Manager.
In this case, we have to use the source which is available at “archive.cloudera.com”.
http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
We have used given base URL
Cloudera — CM5 version.
Operating System — Redhat 6 version.
vi cloudera-manager.repo
These are the details available in the cloudera-manager.repo.
Clean all repositories available in the Cloudera
yum clean all yum makecache
Now, let’s stop all running Cloudera Services.
Stopping Cluster Services
Stopping Cluster Services Completed
Now, we are going to stop Cloudera Manager Services
Stopping Cloudera Manager Services
Stopping Cloudera Manager Services Completed
Now, let’s stop Cloudera Manager Services: Cloudera-scm-server and Cloduera-scm-agent.
service cloudera-scm-server stop service cloudera-scm-agent stop
Let’s list the Cloudera services.
yum list all | grep cloudera
Let’s update Cloudera-manager-agent, Cloudera-manager-deamons and cloudera-manager-server.
yum -y upgrade cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server
Check the status of Cloudera Manager Services, and start them.
service cloudera-scm-agent status service cloudera-scm-agent start service cloudera-scm-server status service cloudera-scm-server start
Now go to your browser, and openhttp://localhost:7180/cmf/login
1) Admin Credentials: admin/admin
2) Other Credentials: cloudera/cloudera
Cloudera Manager Login Page
Since we update repositories, after login we will be landing at Upgrade Cloudera Manager. Click on Run Host Inspector.
Running Upgrade Cloudera Manager
Completed Upgrade Cloudera Manager
Click Start Cloudera Management Service
Review & Start Cloudera Management Services
Click Continue
Review & Start Cloudera Management Services
Click Continue
Start Cloudera Manager Services
Click Finish
Now, our Cloudera Manager Services are running using CM 5 version.
Upgrade Cloudera Manager Screen
Click on Home Page.
You might see some services are in the stale configuration and require a restart.
Cloudera Manager Home Page after Upgrade
(Optional) If you want, you could deploy new configurations. Click on the stale configuration button placed near each service and follow the steps.
Deploying New Client Configurations
Validate Cloudera Manager (CM) Upgrade
Go to Home → Support → About.
Cloudera Manager Home Page Support Dropdown
Check, we are running Cloudera Express 5.16.2.
About Dialog Box
Start all Cloudera Hadoop services: Click Drop Down Icon → Start
Starting Hadoop Services
Cloudera Manager Home Page All Cluster Services Running
Upgrading CDH (Cloudera Distribution Hadoop)
Go to Home → Hosts → Parcels
Cloudera Manager Hosts Dropdown
List of parcels that are added by default in the configuration using URL. Cloudera CDH URL is added to the configuration. That’s why we are seeing CDH 5 in Parcel Name Column.
Note : These are not installed, you can instal them as per your requirement.
Available Parcels
Click the Configuration button on top right
Parcel Repos
Now, let’s replace that “http://archive.cloudera.com/cdh5/parcels/5.7.0/” with “http://archive.cloudera.com/cdh5/parcels/5/”, as we want to upgrade CDH with the latest version of 5.x.x.
Parcel Repos with new entry added
You will see this CDH 5 with version 5.16.x. It will take a few seconds if you can’t find the correct version in the Parcels list. Click on Check for New Parcels.
New Parcel Available
Click on Download.
Note : It will take some time to download based on your internet speed.
Once it has downloaded, click on Distribute.
Parcel Downloaded
Now, click on Upgrade to enable a new CDH version.
Parcel Distributed
Upgrading from 5.7.0 to 5.16.2 (Latest version)
Follow Checkpoints carefully.
This process takes a while…
Now, our CDH 5 is activated through the parcel.
Now, let’s go to the home page. On the left sidebar, near the cluster name (Cloudera Quickstart..), you can find the currently running version.
Congratulations!! We have successfully upgraded the Cloudera Manager (CM) and Cloudera Distribution Hadoop (CDH).
Also, check out our blog about how to install Spark2 in Cloudera Cluster through Docker here. To get the best data engineering solutions for your business, reach out to us at Clairvoyant.