Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Important: It is not a best practice to mutate values or to use RDD directly as opposed to using Dataframes....
800+ Java & Big Data Interview Q&As
130+ job offers to choose from as a self-taught Java & Big Data contractor for 22+ years. It pays to prepare.
Blog Archives
09. Getting started with Spark & Hadoop – client mode on local & cluster mode on yarn
This extends Setting up & getting started with Spark local mode with Sbt & Scala and Setting up & getting started with Hadoop on Mac.
The Spark job written in Scala will be reading the data from HDFS.
1. CSV data in HDFS
|
1 2 3 |
Maths, 85 English, 94 Science, 75 |
2.…
1. Apache Pig Getting started
Input Data
scores.data in folder:/Users/arulk/projects representing marks of 4 students in 3 subjects:
|
1 2 3 4 5 |
Science, 80, 75, 89, 90 Maths, 90, 87, 78, 92 English, 78, 88, 65, 99 |
Calculate the max mark for each subject.
Step 1: Download Apache Pig from http://apache.mirror.digitalpacific.com.au/pig/ and extract the tar file.
…
1. Hadoop MapReduce Basic Tutorial
Input Data & How Hadoop reads the Data
scores.data in folder:/Users/arulk/projects
|
1 2 3 4 5 6 |
Science, 80, 75, 89, 90 Maths, 90, 87, 78, 92 English, 78, 88, 65, 99 |
Mapper Input
The Hadoop “org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat” class read the input as key/value pairs. The default delimiter is tab.
…
1. HBase Tutorial Getting Started
In standalone mode, HBase does not use HDFS — it uses the local filesystem instead — and it runs all HBase daemons and a local ZooKeeper all up in the same JVM. Zookeeper binds to a well known port so clients may talk to HBase.
Step 1: Download HBase from “http://hbase.apache.org/”.
…
10: Databricks – Spark ML – Linear Regression
Prerequisite: Extends Databricks getting started – Spark, Shell, SQL. You can try these tutorials in Scala using Databricks Notebook. There are Scala tutorials covered in Spark using Scala on Zeppelin Notebook. Problem statement: Predict the land prices by land area in square feet based on a given set of known...
This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only.
Register
80+ Free Java FAQs
80+ Free Big Data FAQs
Already a member? Log in here
10: Solving AlreadyBeingCreatedException & LeaseExpiredException thrown from your Spark jobs
What is wrong with the following Spark code snippet?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
resultsRdd.foreachPartition(results -> { final Configuration hc = new Configuration(); final FileSystem hdfs = FileSystem.get(hc); String prefix = UUID.randomUUID().toString(); long nowInMillis = DateTime.now().toInstant().getMillis(); String strOutFile = "/path/to/output/file.csv"; while (results.hasNext()) { Tuple2<String, String> x = results.next(); if (x._2 != null) { String decodedPayload = x._2; final Path outFilePath = new Path(strOutFile); FSDataOutputStream ouputOs = null; if (hdfs.exists(outFilePath)) { ouputOs = hdfs.append(outFilePath); } else { ouputOs = hdfs.create(outFilePath); } ouputOs.writeBytes(decodedPayload); ouputOs.close(); } } }); |
You are likely to get AlreadyBeingCreatedException & LeaseExpiredException thrown as multiple executors try to either create or append to the same file in HDFS in parallel. HDFS allows only one writer. … Read more ›...
This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only.
Register
80+ Free Java FAQs
80+ Free Big Data FAQs
Already a member? Log in here
10: Spark on Zeppelin – union, udf and explode
Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Step 1: Pull this from the docker hub, … Read more ›...
This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only.
Register
80+ Free Java FAQs
80+ Free Big Data FAQs
Already a member? Log in here
10: Spark RDDs to HBase & HBase to Spark RDDs
Step 1: pom.xml with library dependencies. It is important to note that 1) “https://repository.cloudera.com/artifactory/cloudera-repos/” is added as the “Cloudera Maven Repository” and 2) hbase-spark dependency is used for writing to HBase from Spark RDDs & … Read more ›...
This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only.
Register
80+ Free Java FAQs
80+ Free Big Data FAQs
Already a member? Log in here
10. Setting up & getting started with Hive
Hive is popular SQL based tool to process Big Data. It stores & retrieves data to/from HDFS.
Prerequisite: Xcode & Hadoop are installed as outlined in Setting up & getting started with Hadoop on Mac
1. Install Hive
Let’s install hive-2.3.4 with hadoop 2.7.7.
…
11: Databricks – Spark ML – Multivariate Linear Regression
Prerequisite: Extends Databricks – Spark ML – Linear Regression. Problem statement: Predict the house prices by land area in square feet, no of bedrooms, and how old the house is?, which is the age. … Read more ›...
This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only.
Register
80+ Free Java FAQs
80+ Free Big Data FAQs
Already a member? Log in here
11: Docker Tutorial: Hadoop Big Data CLIs on Cloudera quickstart
There are a number of CLIs that you can run from the edge node, which is the gateway node to the Hadoop cluster consisting of master & slave (aka worker) nodes. Let’s look at the different CLIs (i.e Command Line Interfaces)
Most of the CLIs listed below are in /usr/bin
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@quickstart /]# ls -ltr /usr/bin | grep 'hdfs\|impala-shell\|beeline\|hive\|spark-shell\|pyspark\|spark-submit' -rwxr-xr-x 1 root root 8144 Sep 22 2015 db_archive -rwxr-xr-x 1 root root 328 Mar 23 2016 hiveserver2 -rwxr-xr-x 1 root root 321 Mar 23 2016 hive -rwxr-xr-x 1 root root 324 Mar 23 2016 beeline -rwxr-xr-x 1 root root 504 Mar 23 2016 sqoop-create-hive-table -rwxr-xr-x 1 root root 1138056 Mar 23 2016 git-upload-archive -rwxr-xr-x 1 root root 141 Mar 23 2016 spark-submit -rwxr-xr-x 1 root root 140 Mar 23 2016 spark-shell -rwxr-xr-x 1 root root 185 Mar 23 2016 pyspark -rwxr-xr-x 1 root root 192 Mar 23 2016 hdfs -rwxr-xr-x 1 root root 1856 Mar 23 2016 impala-shell |
hdfs CLI
The following command will give you the commands you can use with “hdfs”
…
11: Spark on Zeppelin – Dataframe groupBy, collect_list, explode & window
Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Step 1: Pull this from the docker hub, … Read more ›...
This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only.
Register
80+ Free Java FAQs
80+ Free Big Data FAQs
Already a member? Log in here
300+ Java Interview Questions - FAQs
10+ Companion Techs Interview Q&As
300+ Big Data Interview Questions - FAQs
16+ Java Tech Key Areas Interview Q&As
Your Career Tips to go places
Tutorials on Java & Big Data
300+ Java Interview Questions Answered
“
I have a good news, I got job offer. I like to share with you from my bottom of my heart feeling thankful to you.
Jithendra
via LinkedIn
Disclaimer
The contents in this Java-Success are copyrighted and from EmpoweringTech pty ltd. The EmpoweringTech pty ltd has the right to correct or enhance the current content without any prior notice. These are general advice only, and one needs to take his/her own circumstances into consideration. The EmpoweringTech pty ltd will not be held liable for any damages caused or alleged to be caused either directly or indirectly by these materials and resources. Any trademarked names or labels used in this blog remain the property of their respective trademark owners. Links to external sites do not imply endorsement of the linked-to sites. Privacy Policy
Responsive WordPress Theme
powered by
CyberChimps
Top
Scroll Up