Tutorials - Big Data Archives | Page 9 of 13 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 5 6 7 8 9 10 11 12 13 »

11: Spark streaming with “textFileStream” simple tutorial

Using Spark streaming data can be ingested from many sources like Kafka, Flume, HDFS, Unix/Windows File system, etc. In this example, let’s run the Spark in a local mode to ingest data from a Unix file system. Step 1: The pom.xml file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-spark</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-spark</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <hadoop.version>2.7.2</hadoop.version>
        <avro.version>1.8.1</avro.version>
        <spark.version>1.6.0-cdh5.8.3</spark.version>
        <scala.version>2.10.4</scala.version>
        <scala.binary.version>2.10</scala.binary.version>
    </properties>

    <repositories>
        <repository>
            <id>central</id>
            <name>Maven Central</name>
            <url>http://repo1.maven.org/maven2/</url>
        </repository>
        <repository>
            <id>cloudera</id>
            <url>http://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- Spark libraries -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
    
    <build>
        <plugins>
            <!-- build an uber jar with hadoop dependencies included -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-spark</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-spark</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

<hadoop.version>2.7.2</hadoop.version>

<avro.version>1.8.1</avro.version>

<spark.version>1.6.0-cdh5.8.3</spark.version>

<scala.version>2.10.4</scala.version>

<scala.binary.version>2.10</scala.binary.version>

</properties>

<id>central</id>

<name>Maven Central</name>

<url>http://repo1.maven.org/maven2/</url>

</repository>

<id>cloudera</id>

<url>http://repository.cloudera.com/artifactory/cloudera-repos/</url>

</repository>

</repositories>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

</dependency>

<groupId>org.scala-lang</groupId>

<artifactId>scala-library</artifactId>

<version>${scala.version}</version>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-sql_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

</dependencies>

<build>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>

Using textFileStream(..) textFileStream watches a directory for...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11. What are part- files in Hadoop & 6 ways to merge them

What are the part-xxxx files generated by Hadoop? When you invoke rdd.saveAsTextFile(…) or rdd.saveAsNewAPIHadoopFile(…) from Spark you will get part- files. When you do “INSERT INTO” command in Hive, the execution results in multiple part files in HDFS. You will have one part-xxxx file per partition in the RDD you...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11A: Databricks – Spark ML – Pandas Dataframe & Matplotlib

Prerequisite: Extends 11: Databricks – Spark ML – Multivariate Linear Regression. How do you convert Pyspark dataframe to Pndas Dataframe? df.toPandas() converts Pyspark Dataframe to Pandas Dataframe.

from pyspark.sql.types import *

input_data = (
 (400,3,5,480000.00),
 (500,3,20,400000.00,), 
 (600,5,None,850000.00),
 (700,4,10,770000.00),
 (800,4,15,800000.00),
 (1000,6,20,1100000.00),
 (1200,7,2,1600000.00) 
)
 
data_schema = [StructField('land_area', IntegerType(), False), StructField('bedrooms', IntegerType(), True), StructField('age', IntegerType(), True), StructField('house_price',DoubleType(),False)]
final_struc = StructType(fields=data_schema)
 
df = spark.sparkContext.parallelize(input_data)
 
df_input= spark.createDataFrame(df, final_struc)
 
pandas_df = df_input.toPandas()

print(pandas_df)

from pyspark.sql.types import *

input_data = (

(400,3,5,480000.00),

(500,3,20,400000.00,),

(600,5,None,850000.00),

(700,4,10,770000.00),

(800,4,15,800000.00),

(1000,6,20,1100000.00),

(1200,7,2,1600000.00)

)

data_schema = [StructField('land_area', IntegerType(), False), StructField('bedrooms', IntegerType(), True), StructField('age', IntegerType(), True), StructField('house_price',DoubleType(),False)]

final_struc = StructType(fields=data_schema)

df = spark.sparkContext.parallelize(input_data)

df_input= spark.createDataFrame(df, final_struc)

pandas_df = df_input.toPandas()

print(pandas_df)

Output:

df_input:pyspark.sql.dataframe.DataFrame
land_area:integer
bedrooms:integer
age:integer
house_price:double

   land_area  bedrooms   age  house_price
0        400         3   5.0     480000.0
1        500         3  20.0     400000.0
2        600         5   NaN     850000.0
3        700         4  10.0     770000.0
4        800         4  15.0     800000.0
5       1000         6  20.0    1100000.0
6       1200         7   2.0    1600000.0

df_input:pyspark.sql.dataframe.DataFrame

land_area:integer

bedrooms:integer

age:integer

house_price:double

land_area bedrooms age house_price

0 400 3 5.0 480000.0

1 500 3 20.0 400000.0

2 600 5 NaN 850000.0

3 700 4 10.0 770000.0

4 800 4 15.0 800000.0

5 1000 6 20.0 1100000.0

6 1200 7 2.0 1600000.0

How to perform statistical data exploration?… Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: Databricks – Spark ML – Categorical Features

Prerequisite: Extends Databricks – Spark ML – Linear Regression. Problem statement: Predict the house prices by land area in square feet, house condition as in “Bad”, “Average”, and “Good”, and house locality as in “Sydney”, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: Docker Tutorial: Hadoop Big Data configuration files on Cloudera quickstart

The “etc” (i.e etcetera) folder is mainly for configuration files. It’s purpose is to host various configuration files. For instance, to add a new hard drive to your system and have Linux auto-mount it on boot, you’d have to edit /etc/fstab. Key “Hadoop Cluster Configuration” files are:

[root@quickstart /]# ls -ltr /etc/hadoop/conf/
total 40
-rwxr-xr-x 1 root root  2375 Feb 23  2016 yarn-site.xml
-rwxr-xr-x 1 root root  1104 Feb 23  2016 README
-rwxr-xr-x 1 root root  2890 Feb 23  2016 hadoop-metrics.properties
-rwxr-xr-x 1 root root  1366 Feb 23  2016 hadoop-env.sh
-rwxr-xr-x 1 root root 11291 Mar 23  2016 log4j.properties
-rw-rw-r-- 1 root root  1546 Apr  5  2016 mapred-site.xml
-rw-rw-r-- 1 root root  3739 Apr  5  2016 hdfs-site.xml
-rw-rw-r-- 1 root root  1915 Apr  5  2016 core-site.xml

[root@quickstart /]# ls -ltr /etc/hadoop/conf/

total 40

-rwxr-xr-x 1 root root 2375 Feb 23 2016 yarn-site.xml

-rwxr-xr-x 1 root root 1104 Feb 23 2016 README

-rwxr-xr-x 1 root root 2890 Feb 23 2016 hadoop-metrics.properties

-rwxr-xr-x 1 root root 1366 Feb 23 2016 hadoop-env.sh

-rwxr-xr-x 1 root root 11291 Mar 23 2016 log4j.properties

-rw-rw-r-- 1 root root 1546 Apr 5 2016 mapred-site.xml

-rw-rw-r-- 1 root root 3739 Apr 5 2016 hdfs-site.xml

-rw-rw-r-- 1 root root 1915 Apr 5 2016 core-site.xml

hadoop-env.sh

This file specifies environment variables that affect the JDK used by Hadoop Daemon (i.e.

…

12: Spark on Zeppelin – Dataframe pivot

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Step 1: Pull this from the docker hub, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: Spark streaming with “fileStream” and “PortableDataStream” simple tutorial

This extends the Spark streaming with “textFileStream” simple tutorial to use fileStream(…) and PortableDataStream. The pom.xml file is same as the previous Spark streaming tutorial. Step 1: Using “fileStream(…)”.

package com.mytutorial;

import java.io.DataInputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.List;

import org.apache.hadoop.fs.Path;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.input.PortableDataStream;
import org.apache.spark.input.StreamInputFormat;
import org.apache.spark.streaming.Seconds;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;

import scala.Tuple2;

public class SparkSimpleStreaming {

    public static void main(String[] args) {

        SparkConf sparkConf = new SparkConf().setAppName("Streaming HDFS path").setMaster("local[2]");

        sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
    
        JavaStreamingContext jstreamCtxt = new JavaStreamingContext(sparkConf, Seconds.apply(5)); //every 5 seconds
        
        JavaPairInputDStream<String, PortableDataStream> fileStream = jstreamCtxt.fileStream(
                "file:///home/user/stream", String.class, PortableDataStream.class, StreamInputFormat.class);

        fileStream.foreachRDD(x -> {
            List<Tuple2<String, PortableDataStream>> dataWithFileNames = x.collect();

            for (Tuple2<String, PortableDataStream> dataAndFileNm : dataWithFileNames) {

                String filePath = dataAndFileNm._1.split(":")[1];
                PortableDataStream data = dataAndFileNm._2;

                DataInputStream dis = data.open();
                byte[] bs = new byte[dis.available()];

                dis.readFully(bs);

                System.out.println("###### file " + filePath);

                System.out.println("###### data " + new String(bs));

                Files.deleteIfExists(Paths.get(filePath));
            }

        });

        jstreamCtxt.start();
        jstreamCtxt.awaitTermination();

        jstreamCtxt.close();

    }

}

package com.mytutorial;

import java.io.DataInputStream;

import java.nio.file.Files;

import java.nio.file.Paths;

import java.util.List;

import org.apache.hadoop.fs.Path;

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.function.Function;

import org.apache.spark.input.PortableDataStream;

import org.apache.spark.input.StreamInputFormat;

import org.apache.spark.streaming.Seconds;

import org.apache.spark.streaming.api.java.JavaPairInputDStream;

import org.apache.spark.streaming.api.java.JavaStreamingContext;

import scala.Tuple2;

public class SparkSimpleStreaming {

public static void main(String[] args) {

SparkConf sparkConf = new SparkConf().setAppName("Streaming HDFS path").setMaster("local[2]");

sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");

JavaStreamingContext jstreamCtxt = new JavaStreamingContext(sparkConf, Seconds.apply(5)); //every 5 seconds

JavaPairInputDStream<String, PortableDataStream> fileStream = jstreamCtxt.fileStream(

"file:///home/user/stream", String.class, PortableDataStream.class, StreamInputFormat.class);

fileStream.foreachRDD(x -> {

List<Tuple2<String, PortableDataStream>> dataWithFileNames = x.collect();

for (Tuple2<String, PortableDataStream> dataAndFileNm : dataWithFileNames) {

String filePath = dataAndFileNm._1.split(":")[1];

PortableDataStream data = dataAndFileNm._2;

DataInputStream dis = data.open();

byte[] bs = new byte[dis.available()];

dis.readFully(bs);

System.out.println("###### file " + filePath);

System.out.println("###### data " + new String(bs));

Files.deleteIfExists(Paths.get(filePath));

}

});

jstreamCtxt.start();

jstreamCtxt.awaitTermination();

jstreamCtxt.close();

}

What if you want to process the files already in the folder when the streaming job started?… Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

12: XML Processing in Spark with XmlInputFormat

Step 1: Read the XML snippet in between the tags “<Record>”. Upload this file to HDFS “/user/cloudera/xml/orders.xml”.


<?xml version="1.0" encoding="UTF-8"?>
<Message xmlns="http://com.mytut">

   <Header>
       <MsgId>01-01-2017T22:00:30</MsgId>
       <MsgTimestamp>01-01-2017T22:00:30</MsgTimestamp>
   </Header>

   <Record>
       <OrderId>123</OrderId>
       <OrderName></OrderName>
   </Record>

</Message>

<?xml version="1.0" encoding="UTF-8"?>

</Header>

</Record>

</Message>

Step 2: You need the XmlInputFormat class as shown below. You can find this in the Mahout library. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Databricks – Spark ML – Dummy Variables

Prerequisite: Extends Databricks – Spark ML – Categorical Features – Linear Regression. Problem statement: Predict the house prices by land area in square feet, house color as in “White”, “Grey”, and “Cream”, and house locality as in “Eastwood”, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Docker Tutorial: Apache Spark (spark-shell & pyspark) on Cloudera quickstart

This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \
--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \
--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \
cloudera/quickstart /usr/bin/docker-quickstart

~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \

--privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \

--publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \

cloudera/quickstart /usr/bin/docker-quickstart

Step 2: Create a text file “simple.txt” in the local file system.

[root@quickstart /]# touch simple.txt
[root@quickstart /]# vi simple.txt

[root@quickstart /]# touch simple.txt

[root@quickstart /]# vi simple.txt

A big brown fox jumped over a big brown fence.

1	A big brown fox jumped over a big brown fence.

Step 3: Copy this file onto HDFS file system. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Q98 – Q104 Apache Hive Basics Interview questions & answers

Q98. What is Hive? A98. Hive is used for accessing and analyzing data in Hadoop using SQL syntax. It is known as the HiveQL. Q99. What is the difference between Hive internal tables & external tables? A99. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Spark inner & outer joins in Java with JavaPairRDDs

RDD inner join via JavaPairRDD Here is an inner join displaying all the orders with line items.

package com.mytutorial;

import java.util.Arrays;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;

import scala.Tuple2;

public class SparkRddJoins {

    public static void main(String[] args) {

        SparkConf conf = new SparkConf().setAppName("Spark Rdd joins")
                .setMaster("local[3]");
        
        List<Order> orders = Arrays.asList(new Order("123", "John"), new Order("456", "Smith"), new Order("789", "Samuel"));
        List<LineItem> items = Arrays.asList(new LineItem("123", "Pen"), new LineItem("456", "Pencil"));
        
        try (JavaSparkContext sc = new JavaSparkContext(conf);) {
            JavaRDD<Order> rddOrders = sc.parallelize(orders);
            JavaRDD<LineItem> rddLineItems = sc.parallelize(items);
            
            JavaPairRDD<String, Order> pairRddOrders = rddOrders.mapToPair(x -> {
                return new Tuple2<String, Order>(x.getOrderId(), x);
            });
            
            JavaPairRDD<String, LineItem> pairRddLineItems = rddLineItems.mapToPair(x -> {
                return new Tuple2<String, LineItem>(x.getOrderId(), x); 
            });
            
            JavaPairRDD<String, Tuple2<Order, LineItem>> joinedRdd = pairRddOrders.join(pairRddLineItems);
                                                                                           
            
            joinedRdd.foreach(x -> {
                Tuple2<Order, LineItem> orderAndLineItems = x._2();
                System.out.println("Order= " + orderAndLineItems._1() );
                System.out.println("LineItems= " + orderAndLineItems._2() );
                
            });
        }
    }
}

package com.mytutorial;

import java.util.Arrays;

import java.util.List;

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.JavaPairRDD;

import org.apache.spark.api.java.JavaRDD;

import org.apache.spark.api.java.JavaSparkContext;

import scala.Tuple2;

public class SparkRddJoins {

public static void main(String[] args) {

SparkConf conf = new SparkConf().setAppName("Spark Rdd joins")

.setMaster("local[3]");

List<Order> orders = Arrays.asList(new Order("123", "John"), new Order("456", "Smith"), new Order("789", "Samuel"));

List<LineItem> items = Arrays.asList(new LineItem("123", "Pen"), new LineItem("456", "Pencil"));

try (JavaSparkContext sc = new JavaSparkContext(conf);) {

JavaRDD<Order> rddOrders = sc.parallelize(orders);

JavaRDD<LineItem> rddLineItems = sc.parallelize(items);

JavaPairRDD<String, Order> pairRddOrders = rddOrders.mapToPair(x -> {

return new Tuple2<String, Order>(x.getOrderId(), x);

});

JavaPairRDD<String, LineItem> pairRddLineItems = rddLineItems.mapToPair(x -> {

return new Tuple2<String, LineItem>(x.getOrderId(), x);

});

JavaPairRDD<String, Tuple2<Order, LineItem>> joinedRdd = pairRddOrders.join(pairRddLineItems);

joinedRdd.foreach(x -> {

Tuple2<Order, LineItem> orderAndLineItems = x._2();

System.out.println("Order= " + orderAndLineItems._1() );

System.out.println("LineItems= " + orderAndLineItems._2() );

});

}

Outputs:


Order= Order [orderId=456, orderName=Smith]
LineItems= LineItem [orderId=456, productName=Pencil]
Order= Order [orderId=123, orderName=John]
LineItems= LineItem [orderId=123, productName=Pen]

Order= Order [orderId=456, orderName=Smith]

LineItems= LineItem [orderId=456, productName=Pencil]

Order= Order [orderId=123, orderName=John]

LineItems= LineItem [orderId=123, productName=Pen]

RDD left outer join with filtering via JavaPairRDD Here is a left outer join with filtering to display all the orders without any line items. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

13: Spark on Zeppelin – Dataframe date & timestamp

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 5 6 7 8 9 10 11 12 13 »

Blog Archives

11: Spark streaming with “textFileStream” simple tutorial

11. What are part- files in Hadoop & 6 ways to merge them

11A: Databricks – Spark ML – Pandas Dataframe & Matplotlib

12: Databricks – Spark ML – Categorical Features

12: Docker Tutorial: Hadoop Big Data configuration files on Cloudera quickstart

12: Spark on Zeppelin – Dataframe pivot

12: Spark streaming with “fileStream” and “PortableDataStream” simple tutorial

12: XML Processing in Spark with XmlInputFormat

13: Databricks – Spark ML – Dummy Variables

13: Docker Tutorial: Apache Spark (spark-shell & pyspark) on Cloudera quickstart

13: Q98 – Q104 Apache Hive Basics Interview questions & answers

13: Spark inner & outer joins in Java with JavaPairRDDs

13: Spark on Zeppelin – Dataframe date & timestamp

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer