Tutorials Archives | Page 18 of 30 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 14 15 16 17 18 19 20 21 22 … 30 »

12: Databricks – Spark ML – Categorical Features

Prerequisite: Extends Databricks – Spark ML – Linear Regression. Problem statement: Predict the house prices by land area in square feet, house condition as in “Bad”, “Average”, and “Good”, and house locality as in “Sydney”, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: Docker Tutorial: Hadoop Big Data configuration files on Cloudera quickstart

The “etc” (i.e etcetera) folder is mainly for configuration files. It’s purpose is to host various configuration files. For instance, to add a new hard drive to your system and have Linux auto-mount it on boot, you’d have to edit /etc/fstab. Key “Hadoop Cluster Configuration” files are:

[root@quickstart /]# ls -ltr /etc/hadoop/conf/
total 40
-rwxr-xr-x 1 root root  2375 Feb 23  2016 yarn-site.xml
-rwxr-xr-x 1 root root  1104 Feb 23  2016 README
-rwxr-xr-x 1 root root  2890 Feb 23  2016 hadoop-metrics.properties
-rwxr-xr-x 1 root root  1366 Feb 23  2016 hadoop-env.sh
-rwxr-xr-x 1 root root 11291 Mar 23  2016 log4j.properties
-rw-rw-r-- 1 root root  1546 Apr  5  2016 mapred-site.xml
-rw-rw-r-- 1 root root  3739 Apr  5  2016 hdfs-site.xml
-rw-rw-r-- 1 root root  1915 Apr  5  2016 core-site.xml

[root@quickstart /]# ls -ltr /etc/hadoop/conf/

total 40

-rwxr-xr-x 1 root root 2375 Feb 23 2016 yarn-site.xml

-rwxr-xr-x 1 root root 1104 Feb 23 2016 README

-rwxr-xr-x 1 root root 2890 Feb 23 2016 hadoop-metrics.properties

-rwxr-xr-x 1 root root 1366 Feb 23 2016 hadoop-env.sh

-rwxr-xr-x 1 root root 11291 Mar 23 2016 log4j.properties

-rw-rw-r-- 1 root root 1546 Apr 5 2016 mapred-site.xml

-rw-rw-r-- 1 root root 3739 Apr 5 2016 hdfs-site.xml

-rw-rw-r-- 1 root root 1915 Apr 5 2016 core-site.xml

hadoop-env.sh

This file specifies environment variables that affect the JDK used by Hadoop Daemon (i.e.

…

12: Spark on Zeppelin – Dataframe pivot

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Step 1: Pull this from the docker hub, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: Spark streaming with “fileStream” and “PortableDataStream” simple tutorial

This extends the Spark streaming with “textFileStream” simple tutorial to use fileStream(…) and PortableDataStream. The pom.xml file is same as the previous Spark streaming tutorial. Step 1: Using “fileStream(…)”.

package com.mytutorial;

import java.io.DataInputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.List;

import org.apache.hadoop.fs.Path;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.input.PortableDataStream;
import org.apache.spark.input.StreamInputFormat;
import org.apache.spark.streaming.Seconds;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;

import scala.Tuple2;

public class SparkSimpleStreaming {

    public static void main(String[] args) {

        SparkConf sparkConf = new SparkConf().setAppName("Streaming HDFS path").setMaster("local[2]");

        sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
    
        JavaStreamingContext jstreamCtxt = new JavaStreamingContext(sparkConf, Seconds.apply(5)); //every 5 seconds
        
        JavaPairInputDStream<String, PortableDataStream> fileStream = jstreamCtxt.fileStream(
                "file:///home/user/stream", String.class, PortableDataStream.class, StreamInputFormat.class);

        fileStream.foreachRDD(x -> {
            List<Tuple2<String, PortableDataStream>> dataWithFileNames = x.collect();

            for (Tuple2<String, PortableDataStream> dataAndFileNm : dataWithFileNames) {

                String filePath = dataAndFileNm._1.split(":")[1];
                PortableDataStream data = dataAndFileNm._2;

                DataInputStream dis = data.open();
                byte[] bs = new byte[dis.available()];

                dis.readFully(bs);

                System.out.println("###### file " + filePath);

                System.out.println("###### data " + new String(bs));

                Files.deleteIfExists(Paths.get(filePath));
            }

        });

        jstreamCtxt.start();
        jstreamCtxt.awaitTermination();

        jstreamCtxt.close();

    }

}

package com.mytutorial;

import java.io.DataInputStream;

import java.nio.file.Files;

import java.nio.file.Paths;

import java.util.List;

import org.apache.hadoop.fs.Path;

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.function.Function;

import org.apache.spark.input.PortableDataStream;

import org.apache.spark.input.StreamInputFormat;

import org.apache.spark.streaming.Seconds;

import org.apache.spark.streaming.api.java.JavaPairInputDStream;

import org.apache.spark.streaming.api.java.JavaStreamingContext;

import scala.Tuple2;

public class SparkSimpleStreaming {

public static void main(String[] args) {

SparkConf sparkConf = new SparkConf().setAppName("Streaming HDFS path").setMaster("local[2]");

sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");

JavaStreamingContext jstreamCtxt = new JavaStreamingContext(sparkConf, Seconds.apply(5)); //every 5 seconds

JavaPairInputDStream<String, PortableDataStream> fileStream = jstreamCtxt.fileStream(

"file:///home/user/stream", String.class, PortableDataStream.class, StreamInputFormat.class);

fileStream.foreachRDD(x -> {

List<Tuple2<String, PortableDataStream>> dataWithFileNames = x.collect();

for (Tuple2<String, PortableDataStream> dataAndFileNm : dataWithFileNames) {

String filePath = dataAndFileNm._1.split(":")[1];

PortableDataStream data = dataAndFileNm._2;

DataInputStream dis = data.open();

byte[] bs = new byte[dis.available()];

dis.readFully(bs);

System.out.println("###### file " + filePath);

System.out.println("###### data " + new String(bs));

Files.deleteIfExists(Paths.get(filePath));

}

});

jstreamCtxt.start();

jstreamCtxt.awaitTermination();

jstreamCtxt.close();

}

What if you want to process the files already in the folder when the streaming job started?… Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: XML Processing in Spark with XmlInputFormat

Step 1: Read the XML snippet in between the tags “<Record>”. Upload this file to HDFS “/user/cloudera/xml/orders.xml”.


<?xml version="1.0" encoding="UTF-8"?>
<Message xmlns="http://com.mytut">

   <Header>
       <MsgId>01-01-2017T22:00:30</MsgId>
       <MsgTimestamp>01-01-2017T22:00:30</MsgTimestamp>
   </Header>

   <Record>
       <OrderId>123</OrderId>
       <OrderName></OrderName>
   </Record>

</Message>

<?xml version="1.0" encoding="UTF-8"?>

</Header>

</Record>

</Message>

Step 2: You need the XmlInputFormat class as shown below. You can find this in the Mahout library. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Databricks – Spark ML – Dummy Variables

Prerequisite: Extends Databricks – Spark ML – Categorical Features – Linear Regression. Problem statement: Predict the house prices by land area in square feet, house color as in “White”, “Grey”, and “Cream”, and house locality as in “Eastwood”, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Docker Tutorial: Apache Spark (spark-shell & pyspark) on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \
cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \

cloudera/quickstart /usr/bin/docker-quickstart

Step 2: Create a text file “simple.txt” in the local file system.

[root@quickstart /]# touch simple.txt
[root@quickstart /]# vi simple.txt

[root@quickstart /]# touch simple.txt

[root@quickstart /]# vi simple.txt

A big brown fox jumped over a big brown fence.

1	A big brown fox jumped over a big brown fence.

Step 3: Copy this file onto HDFS file system. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Q98 – Q104 Apache Hive Basics Interview questions & answers

Q98. What is Hive? A98. Hive is used for accessing and analyzing data in Hadoop using SQL syntax. It is known as the HiveQL. Q99. What is the difference between Hive internal tables & external tables? A99. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Spark inner & outer joins in Java with JavaPairRDDs

RDD inner join via JavaPairRDD Here is an inner join displaying all the orders with line items.

package com.mytutorial;

import java.util.Arrays;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;

import scala.Tuple2;

public class SparkRddJoins {

    public static void main(String[] args) {

        SparkConf conf = new SparkConf().setAppName("Spark Rdd joins")
                .setMaster("local[3]");
        
        List<Order> orders = Arrays.asList(new Order("123", "John"), new Order("456", "Smith"), new Order("789", "Samuel"));
        List<LineItem> items = Arrays.asList(new LineItem("123", "Pen"), new LineItem("456", "Pencil"));
        
        try (JavaSparkContext sc = new JavaSparkContext(conf);) {
            JavaRDD<Order> rddOrders = sc.parallelize(orders);
            JavaRDD<LineItem> rddLineItems = sc.parallelize(items);
            
            JavaPairRDD<String, Order> pairRddOrders = rddOrders.mapToPair(x -> {
                return new Tuple2<String, Order>(x.getOrderId(), x);
            });
            
            JavaPairRDD<String, LineItem> pairRddLineItems = rddLineItems.mapToPair(x -> {
                return new Tuple2<String, LineItem>(x.getOrderId(), x); 
            });
            
            JavaPairRDD<String, Tuple2<Order, LineItem>> joinedRdd = pairRddOrders.join(pairRddLineItems);
                                                                                           
            
            joinedRdd.foreach(x -> {
                Tuple2<Order, LineItem> orderAndLineItems = x._2();
                System.out.println("Order= " + orderAndLineItems._1() );
                System.out.println("LineItems= " + orderAndLineItems._2() );
                
            });
        }
    }
}

package com.mytutorial;

import java.util.Arrays;

import java.util.List;

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.JavaPairRDD;

import org.apache.spark.api.java.JavaRDD;

import org.apache.spark.api.java.JavaSparkContext;

import scala.Tuple2;

public class SparkRddJoins {

public static void main(String[] args) {

SparkConf conf = new SparkConf().setAppName("Spark Rdd joins")

.setMaster("local[3]");

List<Order> orders = Arrays.asList(new Order("123", "John"), new Order("456", "Smith"), new Order("789", "Samuel"));

List<LineItem> items = Arrays.asList(new LineItem("123", "Pen"), new LineItem("456", "Pencil"));

try (JavaSparkContext sc = new JavaSparkContext(conf);) {

JavaRDD<Order> rddOrders = sc.parallelize(orders);

JavaRDD<LineItem> rddLineItems = sc.parallelize(items);

JavaPairRDD<String, Order> pairRddOrders = rddOrders.mapToPair(x -> {

return new Tuple2<String, Order>(x.getOrderId(), x);

});

JavaPairRDD<String, LineItem> pairRddLineItems = rddLineItems.mapToPair(x -> {

return new Tuple2<String, LineItem>(x.getOrderId(), x);

});

JavaPairRDD<String, Tuple2<Order, LineItem>> joinedRdd = pairRddOrders.join(pairRddLineItems);

joinedRdd.foreach(x -> {

Tuple2<Order, LineItem> orderAndLineItems = x._2();

System.out.println("Order= " + orderAndLineItems._1() );

System.out.println("LineItems= " + orderAndLineItems._2() );

});

}

Outputs:


Order= Order [orderId=456, orderName=Smith]
LineItems= LineItem [orderId=456, productName=Pencil]
Order= Order [orderId=123, orderName=John]
LineItems= LineItem [orderId=123, productName=Pen]

Order= Order [orderId=456, orderName=Smith]

LineItems= LineItem [orderId=456, productName=Pencil]

Order= Order [orderId=123, orderName=John]

LineItems= LineItem [orderId=123, productName=Pen]

RDD left outer join with filtering via JavaPairRDD Here is a left outer join with filtering to display all the orders without any line items. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Spark on Zeppelin – Dataframe date & timestamp

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

14: Databricks – Spark ML – StringIndexer & OneHotEncoder

Prerequisite: Extends Databricks – Spark ML – Linear Regression Categorical Features. Problem statement: Predict the house prices by land area in square feet, house condition as in “Bad”, “Average”, and “Good”, and house locality as in “Sydney”, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

14: Docker Tutorial: Hive (via beeline) on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart

Step 2: Create a csv file “employee.csv” in the local file system.

[root@quickstart /]# touch employee.csv
[root@quickstart /]# vi employee.csv

[root@quickstart /]# touch employee.csv

[root@quickstart /]# vi employee.csv

John, Samuel, IT
Peter, Smith, Finance
Sean, Mendis, Marketing

John, Samuel, IT

Peter, Smith, Finance

Sean, Mendis, Marketing

Step 3: Copy this file onto HDFS file system. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

14: Spark joins with SQLContext & JavaPairRDD

This extends the last tutorial where Spark inner & outer joins in Java with JavaPairRDDs. In this tutorial let’s read the orders via a Hive table using SQLContext & Dataframe. RDD left outer join with filtering via JavaPairRDD Here is a left outer join with filtering to display all the...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 14 15 16 17 18 19 20 21 22 … 30 »

Blog Archives

12: Databricks – Spark ML – Categorical Features

12: Docker Tutorial: Hadoop Big Data configuration files on Cloudera quickstart

12: Spark on Zeppelin – Dataframe pivot

12: Spark streaming with “fileStream” and “PortableDataStream” simple tutorial

12: XML Processing in Spark with XmlInputFormat

13: Databricks – Spark ML – Dummy Variables

13: Docker Tutorial: Apache Spark (spark-shell & pyspark) on Cloudera quickstart

13: Q98 – Q104 Apache Hive Basics Interview questions & answers

13: Spark inner & outer joins in Java with JavaPairRDDs

13: Spark on Zeppelin – Dataframe date & timestamp

14: Databricks – Spark ML – StringIndexer & OneHotEncoder

14: Docker Tutorial: Hive (via beeline) on Cloudera quickstart

14: Spark joins with SQLContext & JavaPairRDD

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer