Guide to migrating a Single CDH Cluster from a Multi-CDH Cloudera manager to a new Cloudera Manager environment
Migration is an integral part of data strategy in most companies these days. There are multiple forms of migration, but migrating a single CDH cluster from Cloudera Manager to another remains one of many companies' most complex requirements. There might not be enough answers on migrating just a cluster and not the entire CM.
Clairvoyant specializes in maintaining and creating data infrastructures. We, at Clairvoyant, have developed a unique methodology to migrate the cluster from one CM to another and sometimes even a different version of CM altogether. Below, we have listed a step-by-step process for this migration. But for all your customized requests, reach out to us here.
Let's say we have n number of clusters under a single CM and wish to migrate only one cluster to a new CM. We have tested migrating a CDH cluster from a CM of the same version and also migrating a CDH cluster 5.x from a CM of 6.x to a CM of 5.x.
Refer to the below link for Cloudera Manager and CDH Compatibility before migration:
Here we have an environment of CM 6.3.2 with two clusters: One is with CDH 6.3.2, and another is CDH 5.16.2.
Below are the steps to migrate the CDH 5.16.2 cluster under a new CM5.16.2.
1. Ready the Destination CM
Setup the CM server on the destination environment with version 5.16.2.
NOTE: Do not set up any Cluster or CMS.
Please follow this link to set up the CM-
https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/install_cm_cdh.html
Destination CM 5.16.2 without clusters and CMS
2. Obtain the Cloudera Manager Configuration
We use the Cloudera Manager REST API to obtain all of its configuration data. The API exports a JSON document that contains configuration data for the Cloudera Manager instance. Run the below command in the source cluster CM server and get the JSON file cm-deployment.json.
curl -u <USERNAME>:<PASSWORD> "http://<SOURCE_CM_HOST>: <CM_PORT>/api/<VERSION>/cm/deployment" > cm-deployment.json
Where,
USERNAME is a username with either the Full Administrator or Cluster Administrator role
PASSWORD is the password of the USERNAME
SOURCE_CM_HOST is the hostname of the Cloudera Manager server
CM_PORT is the port of the Cloudera Manager server
VERSION is the version of the Cloudera Manager API. Use v19 for 5.x and v33 for 6.x
3. Copy JSON file cm-deployment.json to the destination cluster CM server
4. Importing Cloudera Manager Configuration
We import that configuration to the destination cluster by using a previously saved JSON document containing the Source Cloudera Manager configuration data.
Run the below command in the destination cluster CM server.
curl -H "Content-Type: application/json" — upload-file cm- deployment.json -u <USERNAME>:<PASSWORD> http://<DESTINATION_CM_HOST>:<CM_PORT>/api/<VERSION>/cm/deployment
Where,
USERNAME is a username with either the Full Administrator or Cluster Administrator role.
PASSWORD is the password of the USERNAME
DESTINATION_CM_HOST is the hostname of the Cloudera Manager server
CM_PORT is the port of the Cloudera Manager server
VERSION is the version of the Cloudera Manager API. Use v19 for 5.x and v33 for 6.x
NOTE: If you are using both the source and destination CMs with supported CDH versions, you can directly import the JSON. Here, in my source, one of the clusters is with CDH6.3.2. Cloudera Manager 5.16.2 cannot manage CDH 6.3.2 because the destination Cloudera Manager version, 5, is less than the CDH version 6. So before going forward, in cm-deployment.json, for other clusters with incompatible CDH version, replace the value of versions from “CDH6” to “CDH5” and the fullVersion from “6.3.2” to “5.16.1” as we won’t be using that cluster after importing.
cm-deployment.json before modifying
cm-deployment.json after modifying
Once imported, we will see all the CDH clusters in the destination CM, same as the source cluster, but in a stopped state.
Initial template ready in the destination cluster
5. Delete the CDH clusters (other than the ones which we need to migrate) and CMS from the destination environment.
Cloudera Manager → Cluster 1 → Action → Delete Cloudera Manager → Cloudera Management Service → Actions → Delete
Final template ready in the destination cluster
6. Stop cluster on the source CM
Make sure we have stopped all the services and the CM agent on the “Cluster 2” hosts.
Cloudera Manager → Cluster 2 → Actions → Stop
service cloudera-scm-agent stop service cloudera-scm-agent status
Source CM with two clusters. “Cluster 1” with CDH 6.3.2 and “Cluster 2” with CDH 5.16.2
7. Downgrading CM agents
Optional: If you use a different CM version in the destination compared to the source,
copy the CM repository file to the CM agent nodes. Use the same repository as used in Step 1. Then, downgrade the CM agent version.
8. Pointing CM agent to destination Cluster server
In the destination cluster, login to all the CM agent hosts and change the value of property server_host to point to the destination CM server.
cat /etc/cloudera-scm-agent/config.ini | grep server_host server_host=<DESTINATION_CM_HOST>
9. Starting CM agent
There would be a mismatch in the cm_guid file after reinstalling it. Run the below command and start the CM agent.
rm -rf /var/lib/cloudera-scm-agent/cm_guid service cloudera-scm-agent start
10. Reactivating the parcel
In the destination cluster, deactivate and activate the parcel. Cloudera Manager → Parcels → CDH 5 (5.16.2) → Deactivate → Activate
11. Setting up a Cloudera Management Service
Cloudera Manager → Add Cloudera Management Service
Add Cloudera Management Service
CMS is installed successfully
12. Restarting the Cluster services
Cloudera Manager → Cluster → Actions → Restart
Restarting the cluster services
All services are stable after the restart
13. Confirm that there are no inconsistencies in the data once you’ve migrated a cluster.
14. Deleting the cluster from source CM
As we have successfully migrated the cluster to the destination CM, we will delete the cluster “Cluster-2” from the source CM.
Deleted cluster from source CM
We have now successfully migrated a CDH cluster from one CM to another!
Learn how to upgrade Cloudera CDH cluster to CDP private cloud base with our blog post here. For the best data engineering solutions for your business, reach out to us at Clairvoyant.