TUT - Cloudera on Docker Archives | 800+ Java & Big Data Interview Q&As

Blog Archives

1 2 »

11: Docker Tutorial: Hadoop Big Data CLIs on Cloudera quickstart

There are a number of CLIs that you can run from the edge node, which is the gateway node to the Hadoop cluster consisting of master & slave (aka worker) nodes. Let’s look at the different CLIs (i.e Command Line Interfaces)

Most of the CLIs listed below are in /usr/bin

[root@quickstart /]# ls -ltr  /usr/bin | grep 'hdfs\|impala-shell\|beeline\|hive\|spark-shell\|pyspark\|spark-submit'
-rwxr-xr-x   1 root root      8144 Sep 22  2015 db_archive
-rwxr-xr-x   1 root root       328 Mar 23  2016 hiveserver2
-rwxr-xr-x   1 root root       321 Mar 23  2016 hive
-rwxr-xr-x   1 root root       324 Mar 23  2016 beeline
-rwxr-xr-x   1 root root       504 Mar 23  2016 sqoop-create-hive-table
-rwxr-xr-x   1 root root   1138056 Mar 23  2016 git-upload-archive
-rwxr-xr-x   1 root root       141 Mar 23  2016 spark-submit
-rwxr-xr-x   1 root root       140 Mar 23  2016 spark-shell
-rwxr-xr-x   1 root root       185 Mar 23  2016 pyspark
-rwxr-xr-x   1 root root       192 Mar 23  2016 hdfs
-rwxr-xr-x   1 root root      1856 Mar 23  2016 impala-shell

-rwxr-xr-x 1 root root 8144 Sep 22 2015 db_archive

-rwxr-xr-x 1 root root 328 Mar 23 2016 hiveserver2

-rwxr-xr-x 1 root root 321 Mar 23 2016 hive

-rwxr-xr-x 1 root root 324 Mar 23 2016 beeline

-rwxr-xr-x 1 root root 504 Mar 23 2016 sqoop-create-hive-table

-rwxr-xr-x 1 root root 1138056 Mar 23 2016 git-upload-archive

-rwxr-xr-x 1 root root 141 Mar 23 2016 spark-submit

-rwxr-xr-x 1 root root 140 Mar 23 2016 spark-shell

-rwxr-xr-x 1 root root 185 Mar 23 2016 pyspark

-rwxr-xr-x 1 root root 192 Mar 23 2016 hdfs

-rwxr-xr-x 1 root root 1856 Mar 23 2016 impala-shell

hdfs CLI

The following command will give you the commands you can use with “hdfs”

…

12: Docker Tutorial: Hadoop Big Data configuration files on Cloudera quickstart

The “etc” (i.e etcetera) folder is mainly for configuration files. It’s purpose is to host various configuration files. For instance, to add a new hard drive to your system and have Linux auto-mount it on boot, you’d have to edit /etc/fstab. Key “Hadoop Cluster Configuration” files are:

[root@quickstart /]# ls -ltr /etc/hadoop/conf/
total 40
-rwxr-xr-x 1 root root  2375 Feb 23  2016 yarn-site.xml
-rwxr-xr-x 1 root root  1104 Feb 23  2016 README
-rwxr-xr-x 1 root root  2890 Feb 23  2016 hadoop-metrics.properties
-rwxr-xr-x 1 root root  1366 Feb 23  2016 hadoop-env.sh
-rwxr-xr-x 1 root root 11291 Mar 23  2016 log4j.properties
-rw-rw-r-- 1 root root  1546 Apr  5  2016 mapred-site.xml
-rw-rw-r-- 1 root root  3739 Apr  5  2016 hdfs-site.xml
-rw-rw-r-- 1 root root  1915 Apr  5  2016 core-site.xml

[root@quickstart /]# ls -ltr /etc/hadoop/conf/

total 40

-rwxr-xr-x 1 root root 2375 Feb 23 2016 yarn-site.xml

-rwxr-xr-x 1 root root 1104 Feb 23 2016 README

-rwxr-xr-x 1 root root 2890 Feb 23 2016 hadoop-metrics.properties

-rwxr-xr-x 1 root root 1366 Feb 23 2016 hadoop-env.sh

-rwxr-xr-x 1 root root 11291 Mar 23 2016 log4j.properties

-rw-rw-r-- 1 root root 1546 Apr 5 2016 mapred-site.xml

-rw-rw-r-- 1 root root 3739 Apr 5 2016 hdfs-site.xml

-rw-rw-r-- 1 root root 1915 Apr 5 2016 core-site.xml

hadoop-env.sh

This file specifies environment variables that affect the JDK used by Hadoop Daemon (i.e.

…

13: Docker Tutorial: Apache Spark (spark-shell & pyspark) on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \
cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \

cloudera/quickstart /usr/bin/docker-quickstart

Step 2: Create a text file “simple.txt” in the local file system.

[root@quickstart /]# touch simple.txt
[root@quickstart /]# vi simple.txt

[root@quickstart /]# touch simple.txt

[root@quickstart /]# vi simple.txt

A big brown fox jumped over a big brown fence.

1	A big brown fox jumped over a big brown fence.

Step 3: Copy this file onto HDFS file system. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

14: Docker Tutorial: Hive (via beeline) on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Step 2: Create a csv file “employee.csv” in the local file system.

[root@quickstart /]# touch employee.csv
[root@quickstart /]# vi employee.csv

[root@quickstart /]# touch employee.csv

[root@quickstart /]# vi employee.csv

John, Samuel, IT
Peter, Smith, Finance
Sean, Mendis, Marketing

John, Samuel, IT

Peter, Smith, Finance

Sean, Mendis, Marketing

Step 3: Copy this file onto HDFS file system. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15: Docker Tutorial: Hive & parquet-tools – csv to parquet on Cloudera quickstart

CSV is a row based storage, and Parquet is columnar in nature, and it is designed from the ground up for efficient storage, compression and encoding, which gives better performance. Run the cloudera/quickstart This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: Docker Tutorial: Apache Spark (spark-shell) & parquet-tools – csv to parquet on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Get parquet-tools Step 2: Install wget. The “uname -a” gets you the info of the kernel. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

17: Docker Tutorial: sqoop import – on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Connect to MySQL DB Step 2: Hive & Impala store the metadata in a database like MySQL. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

18: Docker Tutorial: sqoop export – on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Connect to MySQL DB Step 2: Hive & Impala store the metadata in a database like MySQL. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

19: Docker Tutorial: Apache Spark SQL – on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker.

Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Create a .csv in HDFS

Step 2: Create a csv file “employee.csv” in the local file system.

…

20: Docker Tutorial: Apache Spark (spark-submit) in Java on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Step 2: Java is already installed as part of the “cloudera/quickstart” docker image.

[root@quickstart /]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

[root@quickstart /]# java -version

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

update-alternatives If you have multiple versions or installations of Java, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

21: Docker Tutorial: Apache Spark (spark-submit) in Scala on Cloudera quickstart

Extends 20: Docker Tutorial: Apache Spark-submit in Java – on Cloudera quickstart, and Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Install Java 8 Step 2: Install Java 8. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

22: Docker Tutorial: Apache Spark (spark-submit) in Python 2.6 on Cloudera quickstart

Extends 20: Docker Tutorial: Apache Spark-submit in Java – on Cloudera quickstart, and Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Python version Step 2: The cloudera/quickstart comes with Python 2.6. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

23: Docker Tutorial: Apache Spark (spark-submit) in Python 3 with virtual env on Cloudera quickstart

Cloudera quickstart Docker with Python 3

Prerequisite: Docker is installed on your Windows or Mac, and you have a basic understanding of Docker. Docker tutorials step by step | Hadoop, Hive, Impala & Spark on Cloudera quickstart on Docker tutorials

Step 1: Pull the modified image “gdancik/cloudera”

…

1 2 »

Blog Archives

11: Docker Tutorial: Hadoop Big Data CLIs on Cloudera quickstart

12: Docker Tutorial: Hadoop Big Data configuration files on Cloudera quickstart

13: Docker Tutorial: Apache Spark (spark-shell & pyspark) on Cloudera quickstart

14: Docker Tutorial: Hive (via beeline) on Cloudera quickstart

15: Docker Tutorial: Hive & parquet-tools – csv to parquet on Cloudera quickstart

16: Docker Tutorial: Apache Spark (spark-shell) & parquet-tools – csv to parquet on Cloudera quickstart

17: Docker Tutorial: sqoop import – on Cloudera quickstart

18: Docker Tutorial: sqoop export – on Cloudera quickstart

19: Docker Tutorial: Apache Spark SQL – on Cloudera quickstart

20: Docker Tutorial: Apache Spark (spark-submit) in Java on Cloudera quickstart

21: Docker Tutorial: Apache Spark (spark-submit) in Scala on Cloudera quickstart

22: Docker Tutorial: Apache Spark (spark-submit) in Python 2.6 on Cloudera quickstart

23: Docker Tutorial: Apache Spark (spark-submit) in Python 3 with virtual env on Cloudera quickstart

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer