Upgrading to Java8 on the Cloudera Quickstart VM

The purpose of this blog is to describe how to set Java8 as the version of Java to use in the Cloudera Quickstart VM and as the version of Java to use in Hadoop. You can also read more use cases about how to upgrade to Java 8 here. The reason you might want to do this is so that you can run Spark jobs using the Java8 libraries and features (like lambda operations, etc).

High-Level Steps

1. Stop the Hadoop Services
2. Upgrade to Java8
3. Update configurations to use Java8
4. Restart services

Upgrade Steps

1. Stop the Hadoop Services

Without Cloudera Manager

• SSH into the machine
• Login as root

sudo su

• Execute the stop service commands

service hadoop-hdfs-datanode stop
service hadoop-hdfs-journalnode stop
service hadoop-hdfs-namenode stop
service hadoop-hdfs-secondarynamenode stop
service hadoop-httpfs stop
service hadoop-mapreduce-historyserver stop
service hadoop-yarn-nodemanager stop
service hadoop-yarn-proxyserver stop
service hadoop-yarn-resourcemanager stop
service hbase-master stop
service hbase-regionserver stop
service hbase-rest stop
service hbase-solr-indexer stop
service hbase-thrift stop
service hive-metastore stop
service hive-server2 stop
service impala-catalog stop
service impala-server stop
service impala-state-store stop
service oozie stop
service solr-server stop
service spark-history-server stop
service sqoop2-server stop
service sqoop-metastore stop
service zookeeper-server stop

Using Cloudera Manager

• Log in to the Cloudera Manager
• Stop to Cloudera Management Services
• Stop the Cluster
• SSH into the machine
• Log in as root

sudo su

• Stop the Cloudera SCM Services from the command line

service cloudera-scm-agent stop
service cloudera-scm-server stop

2. Install and Configure JDK 1.8

• SSH into the machine
• Log in as root

sudo su

• Install JDK

yum install java-1.8.0-openjdk -y

• Edit the /etc/bashrc file and set the JAVA_HOME to java8 so the file resembles:

export JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk.x86_64"

export JRE_HOME="${JAVA_HOME}"

export JDK_HOME="${JAVA_HOME}"

export ANT_HOME="/usr/local/apache-ant/apache-ant-1.9.2"

export M2_HOME="/usr/local/apache-maven/apache-maven-3.0.4"

export
PATH="/usr/local/firefox:/sbin:$JAVA_HOME/bin:$ANT_HOME/bin:$M2_HOME/
bin:$PATH"

• Apply the changes to the /etc/bashrc

source ~/.bashrc

• Validate the correct version of Java

java -version

• Add the line `export JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk.x86_64"` to the bottom of each of the following files:

nano /etc/default/cloudera-scm-server

nano /etc/default/hadoop

nano /etc/default/hadoop-0.20-mapreduce

nano /etc/default/hadoop-hdfs-datanode

nano /etc/default/hadoop-hdfs-journalnode

nano /etc/default/hadoop-hdfs-namenode

nano /etc/default/hadoop-hdfs-secondarynamenode

nano /etc/default/hadoop-yarn-nodemanager

nano /etc/default/spark

nano /etc/default/impala

nano /etc/default/zookeeper

nano /etc/default/solr

3. Restart the Hadoop Services

Without Cloudera Manager

A fast way to Restart the Hadoop Services is to just restart the Virtual Machine. The services are set up to startup when the Virtual Machine starts.

Using Cloudera Manager

• SSH into the machine
• Log in as root

sudo su

• Start the Cloudera SCM Services from the command line

service cloudera-scm-agent start
service cloudera-scm-server start

• Log in to Cloudera Manager
• Start the Cluster
• Start the Cloudera Management Services

4. Validate all the Services are using Java8

• Print the Hadoop processes

ps -eaf | grep hadoop

• You should see all the lines resemble the following:

hdfs 5288 1 27 21:53 ? 00:00:04 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/…

Hope you found this post interesting and informative! Don’t forget to check out our Data Engineering solutions for all your business requirements.