Cluster Upgradation from HDP to CDP Private Cloud Base

The upgrade procedure from HDP 2.6.5 to CDP private Cloud Base 7.1.6

Cloudera announced a new version of CDP that allows customers to upgrade from their legacy HDP2.6 clusters and drive even more value from their data by gaining access to new features and functionality to enhance their big data experience.

We, at Clairvoyant, go through efficient and strategic planning before starting the upgrade process. This helps us to get the upgrade completed smoothly and with reduced downtime. Below, we have listed a step-by-step process for the upgrade. For all your customized requests, reach out to us here.

This upgrade requires several significant steps, including upgrading to an interim version of Ambari. After the upgrade, our cluster will be managed by Cloudera Manager, and the components will be upgraded to Cloudera Runtime 7.1.6.

Source Version: Ambari 2.6.2.2 and HDP 2.6.5
Target Version: CM 7.3.1 and CDP 7.1.6

The high-level step of the upgrade process is to transition from the Ambari to Cloudera Manager 7.1.6 using the AM2CM tool. It includes the following steps:

1. HDP cluster upgrade from version 2.6.5 to 7.1.6.

2. HDP cluster blueprint is exported from Ambari and converted to a Cloudera Manager deployment template using the AM2CM tool.

3. Deploy the template to Cloudera Manager 7.1.6 and activate the Cloudera Runtime 7.1.6 parcels.

4. Start using the CDP Private Cloud Base cluster with Cloudera Manager 7.1.6.

5. Uninstall Ambari and the HDP stack.

Architecture diagram for the upgrade and the transition

Below are the steps to upgrade the Hadoop cluster from HDP 2.6.5 to CDP 7.1.6.

1. Environment requirement and support versions

One of the significant challenges while upgrading to a major release is to ensure that the environment has the supported Hardware, Operating System, Java version, and Database requirements. The steps in the link will help you to get compatibility details.

2. Pre-Upgrade steps

• Ensure all services in the cluster are running. Run each Service Check (found under the Service Actions menu) and confirm they executed successfully.

• Follow the link to take a backup of Ambari.

• We must configure and review Kerberos as per the link.

Encrypt Database and LDAP Passwords

• Remove the components that are unsupported in CDP Private Cloud Base. For more information, see the supported component.

• Hive prerequisites to handle missing table or partition locations and correct invalid SERDE definitions.

• Stop SmartSense and Log Search service from the cluster.

• We must not keep any services in “Maintenance mode” except for Ambari Metrics System, SmartSense, and Log Search.

• Turn off the “Auto Start Services.”

Auto Start service disabled

• Download, Extract and Run the Ambari Metrics pre-upgrade script. Login to Metrics Collector host and run below command,

The output of the ams pre-upgrade script

3. Upgrading Ambari from 2.6.2.2 to 7.1.6

• Stop the Ambari Server - On the host running Ambari Server.

• Stop the Ambari Agents - On each host in your cluster running an Ambari Agent.

• Download the Ambari Repo ambaridc.repo from https://archive.cloudera.com/p/ambaridc/7.x/7.1.6.0/ and copy it inside /etc/yum.repos.d/ambaridc.repo .

• Open the ambaridc.repo file and update the username and password of baseurl and gpgkey.

• Check the current and available Ambari Server version with the command

  # yum info ambari-server

Installed and Available Ambari Server package

• Once we see the desired version in Available Packages, we can upgrade Ambari Server.

  # yum upgrade ambari-server

  # yum info ambari-server

Ambari Server with 7.1.6 version

• Now copy the ambaridc.repo to all the Cluster hosts under /etc/yum.repos.d/and upgrade Ambari Agent.

  # yum upgrade ambari-agent

  # yum info ambari-agent

Ambari Agent with 7.1.6 version

• Ambari Server database schema Upgrade - Login to Ambari server node and run below command.

  # ambari-server upgrade

Ambari server database upgrade done successfully

• Command to get the JDBC driver jar file in place,

  # ambari-server setup — jdbc-db=mysql — jdbc-driver=/usr/share/java/mysql-connectorjava.jar

• Now start the Ambari server and Ambari agents.

• Log in to the Ambari UI with admin credentials.

Ambari interim UI

Ambari successfully upgraded to the 7.1.6 version

Till now, we have successfully upgraded Ambari to version 7.1.6. Do not manually restart any HDP services unless future steps in the upgrade guide prompt us to do so. Manually restarting these services may significantly disrupt our upgrade. Ambari will restart each service automatically during the HDP upgrade.

4. Upgrading HDP from 2.6.5 to 7.1.6

• Ensure all services in the cluster are running. Run each Service Check and confirm they execute successfully.

• Follow the link to take a backup of the HDP cluster.

• We can see in “Stack and Versions” only HDP 2.6.5.

• Register software repositories for HDP 7.1.6 by going to the “MANAGE VERSIONS” and then “REGISTER VERSION” with HDP 7.1.6.0

Register new HDP version 7.1.6.0

• Click on “INSTALL PACKAGES”. This will install all HDP 7.1.6 packages.

Install the HDP 7.1.6 package

• The "Upgrade" button replaces the "Install Packages" button when the installation completes.  

  Upgrade button under HDP 7.1.6 is visible

  Before going ahead with the Upgrade, there are some pre-upgrade steps to be performed.

• Follow the link for preparing Hive for an upgrade. This needs to be followed carefully as we are moving to Hive 3.

• Follow the link for upgrading the Ambari Metrics system and SmartSense. Do not start the Ambari Metrics System service. It will start automatically during the HDP upgrade process.

• Let's complete the pre-upgrade check before going for an HDP 7 upgrade. This will check if we missed any requirements. If any, we need to get it cleared.

Click Pre-Upgrade Check to review the requirement

• Now click the “UPGRADE” button under HDP 7.1.6 and go ahead with the Express upgrade.

Click Express Upgrade

• We can see the upgrade process has started. Some of the stages require that we complete an action during normal operation. If any stage fails, the upgrade stops and prompts us for action.

• Once the upgrade is completed, we can Finalize the upgrade or to Finalize Later. Finalizing later gives you a chance to perform more validation on the cluster. Click Finalize to complete the express upgrade process.

Upgrade process finished

• Now that the HDP Upgrade to the 7.1.6 version is finished, we need to follow the link for the post HDP upgrade task.

• We have successfully completed Ambari and HDP upgrade to interim version 7.1.6. Now let us plan for transitioning to Cloudera Manager.

5. Setup of new Cloudera Manager 7.3.1

• Follow the link to download the cloudera-manager.repo. Copy the repo to all the hosts.

• Install Cloudera Manager and Agent on CM host.

  # yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

• Once Cloudera packages are installed, follow the link to Install and Configure MySQL for Cloudera Software.

• We should configure the Cloudera Manager to point to the MySQL database with below command,

  #sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm -v

• Now let us start the Cloudera Manager server.

  # service cloudera-scm-server restart

• Login to the CM UI HTTP://<CM_HOST>:7180 and we can see the below screen. Upload the License.

CM UI after the first login

• From the UI, install Cloudera Agent and then add Cloudera Management Services.

  NOTE: Do not set up a cluster using the Wizard.

• Set up Kerberos - If we have a Kerberos cluster, then we must add the KDC details in the Administration>Security>Kerberos Credentials>Setup KDC for the Cloudera Manager page using Cloudera Manager.

6. Transition from HDP interim version 7.1.6 to Cloudera Manager using the AM2CM tool

• Login to Ambari Server.

• In the new tab, enter the URL and save the blueprint in JSON format:

  http://<ambari_ip>:<port>/api/v1/clusters/<cluster-name>?
  format=blueprint_with_hosts

• Download the AM2CM tool and extract the tar.

  # wget https://archive.cloudera.com/am2cm/1.x/am2cm-1.1.1.0-1.tar.gz

  # tar -zxvf am2cm-1.1.1.0–1.tar.gz

• Navigate to the am2cm-1.1.1.0–1/conf/user-settings.ini file and update Parcels, Cluster name, passwords, and JDBC URL information.

• Generate the Cloudera Manager Deployment template from the HDP blueprint.

  # cd am2cm-1.1.1.0–1

  # chmod +x ./am2cm.sh

  # ./am2cm-1.1.1.0–1/am2cm.sh -bp

  HDP_blueprint_with_hosts_20210519142803.json
  -dt cm_deployment_template.json

Output after successfully generating the CM deployment templete

• Stop HDP services from Ambari UI

Stop All of the HDP services

• Once all the services are stopped successfully, import the Cloudera Manager deployment template using the API through CLI.

  # curl - -user admin:admin -k -X PUT -H “Content-Type: application/json” -d @cm_deployment_template.json ‘http://<CM_HOST>:7180/api/v41/cm/deployment?deleteCurrentDeployment=false'

• We can see a screen as below in CM UI. But many other errors and warnings need to be fixed.

CM UI after importing the CM template

• In the Cloudera Manager Parcel screen, download Cloudera Runtime and distribute the parcel.

Download and distribute the CR 7.1.6 parcel to all hosts

• After the parcels are deployed on Cloudera Manager, activate the Cloudera Runtime 7.1.x parcels.

Activate the CR 7.1.6 parcel

• We need to generate all the keytabs in Cloudera Manager under the Security section ->Kerberos Credentials -> Generate Missing Credentials.

Generating Credentials for all the services

• There are some major post-transition steps that need to be followed which are mentioned in the link.

• After the post-transition and troubleshooting of the errors, we get the services are in the green state.

Cluster with all services green

• Time for running a smoke test on all services to verify all looks good.

Finally, we have successfully upgraded HDP to CDP private base. To learn more about upgraading Cloudera CDH cluster to CDP Private Cloud Base, read our blog post.

For all your Cloud based services requirements, reach out to us at Clairvoyant.