<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=2877026&amp;fmt=gif">

Upgrading to Java8 on the Cloudera Quickstart VM

By Robert Sanders - August 27, 2021

Steps to Upgrade Java on the Cloudera Quickstart VM

The purpose of this blog is to describe how to set Java8 as the version of Java to use in the Cloudera Quickstart VM and as the version of Java to use in Hadoop. You can also read more use cases about how to upgrade to Java 8 here. The reason you might want to do this is so that you can run Spark jobs using the Java8 libraries and features (like lambda operations, etc).

High-Level Steps

  1. Stop the Hadoop Services
  2. Upgrade to Java8
  3. Update configurations to use Java8
  4. Restart services

Upgrade Steps

1. Stop the Hadoop Services

Without Cloudera Manager

  • SSH into the machine
  • Login as root
sudo su
  • Execute the stop service commands
service hadoop-hdfs-datanode stop
service hadoop-hdfs-journalnode stop
service hadoop-hdfs-namenode stop
service hadoop-hdfs-secondarynamenode stop
service hadoop-httpfs stop
service hadoop-mapreduce-historyserver stop
service hadoop-yarn-nodemanager stop
service hadoop-yarn-proxyserver stop
service hadoop-yarn-resourcemanager stop
service hbase-master stop
service hbase-regionserver stop
service hbase-rest stop
service hbase-solr-indexer stop
service hbase-thrift stop
service hive-metastore stop
service hive-server2 stop
service impala-catalog stop
service impala-server stop
service impala-state-store stop
service oozie stop
service solr-server stop
service spark-history-server stop
service sqoop2-server stop
service sqoop-metastore stop
service zookeeper-server stop

Using Cloudera Manager

  • Log in to the Cloudera Manager
  • Stop to Cloudera Management Services
  • Stop the Cluster
  • SSH into the machine
  • Log in as root
sudo su
  • Stop the Cloudera SCM Services from the command line
service cloudera-scm-agent stop
service cloudera-scm-server stop

2. Install and Configure JDK 1.8

  • SSH into the machine
  • Log in as root
sudo su
  • Install JDK
yum install java-1.8.0-openjdk -y
  • Edit the /etc/bashrc file and set the JAVA_HOME to java8 so the file resembles:
export JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk.x86_64"

export JRE_HOME="${JAVA_HOME}"

export JDK_HOME="${JAVA_HOME}"

export ANT_HOME="/usr/local/apache-ant/apache-ant-1.9.2"

export M2_HOME="/usr/local/apache-maven/apache-maven-3.0.4"

export
PATH="/usr/local/firefox:/sbin:$JAVA_HOME/bin:$ANT_HOME/bin:$M2_HOME/
bin:$PATH"
  • Apply the changes to the /etc/bashrc
source ~/.bashrc
  • Validate the correct version of Java
java -version
  • Add the line `export JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk.x86_64"` to the bottom of each of the following files:
nano /etc/default/cloudera-scm-server

nano /etc/default/hadoop

nano /etc/default/hadoop-0.20-mapreduce

nano /etc/default/hadoop-hdfs-datanode

nano /etc/default/hadoop-hdfs-journalnode

nano /etc/default/hadoop-hdfs-namenode

nano /etc/default/hadoop-hdfs-secondarynamenode

nano /etc/default/hadoop-yarn-nodemanager

nano /etc/default/spark

nano /etc/default/impala

nano /etc/default/zookeeper

nano /etc/default/solr

3. Restart the Hadoop Services

Without Cloudera Manager

A fast way to Restart the Hadoop Services is to just restart the Virtual Machine. The services are set up to startup when the Virtual Machine starts.

Using Cloudera Manager

  • SSH into the machine
  • Log in as root
sudo su
  • Start the Cloudera SCM Services from the command line
service cloudera-scm-agent start
service cloudera-scm-server start
  • Log in to Cloudera Manager
  • Start the Cluster
  • Start the Cloudera Management Services

4. Validate all the Services are using Java8

  • Print the Hadoop processes
ps -eaf | grep hadoop
  • You should see all the lines resemble the following:

hdfs 5288 1 27 21:53 ? 00:00:04 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/…

Author
Robert Sanders

Director of Big Data and Cloud Engineering for Clairvoyant LLC | Marathon Runner | Triathlete | Endurance Athlete

Tags: Big Data Cloudera Java Virtual Machine Data Engineering Upgrade