Tutorials - Big Data Archives | Page 7 of 13 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 3 4 5 6 7 8 9 10 11 … 13 »

07: Getting started with Spark-submit

This extends the tutorial Setting up & getting started with Spark local mode with Sbt & Scala and Setting up Spark standalone on Mac. Spark-submit is used to submit a Spark job. In this tutorial we will be using the “local” master in “client” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Learn Spark Dataframes to do ETL in Java with examples

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. What is a Spark Dataframe? A DataFrame is an immutable distributed collection of data like an RDD, but unlike an RDD, data is organized into named columns...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Spark on Zeppelin – window functions in Scala

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Q. What are the different types of functions in Spark SQL? … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: spark-xml to split & read very large XML files

Processing very large XML files can be a bit tricky as they cannot be processed line by line in parallel as you would do with CSV files. The xml file has to be intact whilst matching the start and end entity tags, and if the tags are distributed in parts...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Apache Spark standalone cluster on Docker to read from AWS S3 bucket & write to MySQL database

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, MySQL DB, Spark master and Spark worker to form a cluster. “mysql” is the hostname.

version: "2" 

services:
  s3:
    container_name: s3
    image: minio/minio:RELEASE.2018-05-04T23-13-12Z
    environment:
      MINIO_ACCESS_KEY: user
      MINIO_SECRET_KEY: password
    ports:
      - 9000:9000
    volumes:
      - ./docker/s3/data:/data
      - ./docker/s3/config:/root/.minio
    networks:
      - infra
    command: server /data

  mysql:
    container_name: mysql
    image: mysql:5.7.21
    ports:
      - 3306:3306
    volumes:
      - ./docker/mysql/data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: password
    networks:
      - infra

  spark-master:
      build:
        dockerfile: spark.dockerfile
        context: ./docker/spark
      command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master
      hostname: spark-master
      environment:
        MASTER: spark://spark-master:7077
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: 127.0.0.1
      expose:
        - 7001
        - 7002
        - 7003
        - 7004
        - 7005
        - 7006
        - 7077
        - 6066
        - 5005
      ports:
        - 5005:5005
        - 6066:6066
        - 7077:7077
        - 8080:8080
      volumes:
        - ./docker/spark/spark-master:/conf
        - ./data:/tmp/data 
      networks:
        - infra 
  
  spark-worker-1:
      build:
        dockerfile: spark.dockerfile
        context: ./docker/spark
      command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
      hostname: spark-worker-1
      environment:
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: 127.0.0.1
        SPARK_WORKER_CORES: 2
        SPARK_WORKER_MEMORY: 2g
        SPARK_WORKER_PORT: 8881
        SPARK_WORKER_WEBUI_PORT: 8081
      links:
        - spark-master
      expose:
        - 7012
        - 7013
        - 7014
        - 7015
        - 7016
        - 8881
      ports:
        - 8081:8081
      volumes:
        - ./conf/spark-worker-1:/conf
        - ./data:/tmp/data 
      networks:
        - infra

networks:
  infra:
    external:
      name: docker_test_infra

version: "2"

services:

s3:

container_name: s3

image: minio/minio:RELEASE.2018-05-04T23-13-12Z

environment:

MINIO_ACCESS_KEY: user

MINIO_SECRET_KEY: password

ports:

- 9000:9000

volumes:

- ./docker/s3/data:/data

- ./docker/s3/config:/root/.minio

networks:

- infra

command: server /data

mysql:

container_name: mysql

image: mysql:5.7.21

ports:

- 3306:3306

volumes:

- ./docker/mysql/data:/var/lib/mysql

environment:

MYSQL_ROOT_PASSWORD: password

networks:

- infra

spark-master:

build:

dockerfile: spark.dockerfile

context: ./docker/spark

command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master

hostname: spark-master

environment:

MASTER: spark://spark-master:7077

SPARK_CONF_DIR: /conf

SPARK_PUBLIC_DNS: 127.0.0.1

expose:

- 7001

- 7002

- 7003

- 7004

- 7005

- 7006

- 7077

- 6066

- 5005

ports:

- 5005:5005

- 6066:6066

- 7077:7077

- 8080:8080

volumes:

- ./docker/spark/spark-master:/conf

- ./data:/tmp/data

networks:

- infra

spark-worker-1:

build:

dockerfile: spark.dockerfile

context: ./docker/spark

command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077

hostname: spark-worker-1

environment:

SPARK_CONF_DIR: /conf

SPARK_PUBLIC_DNS: 127.0.0.1

SPARK_WORKER_CORES: 2

SPARK_WORKER_MEMORY: 2g

SPARK_WORKER_PORT: 8881

SPARK_WORKER_WEBUI_PORT: 8081

links:

- spark-master

expose:

- 7012

- 7013

- 7014

- 7015

- 7016

- 8881

ports:

- 8081:8081

volumes:

- ./conf/spark-worker-1:/conf

- ./data:/tmp/data

networks:

- infra

networks:

infra:

external:

name: docker_test_infra

Step 2: The “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Databricks – Spark problem 1

Prerequisite: Extends Databricks – Spark Window functions.

Problem: Convert the below table

+------+--------+---------+-------------+
|emp_id|emp_name| emp_city|emp_expertise|
+------+--------+---------+-------------+
|     1|    John|   Sydney|         Java|
|     2|   Peter|Melbourne|        Scala|
|     3|     Sam| Brisbane|       Python|
|     4|   David|Melbourne|       Python|
|     5|  Elliot|   Sydney|         Java|
+------+--------+---------+-------------+

+------+--------+---------+-------------+

+------+--------+---------+-------------+

| 3| Sam| Brisbane| Python|

+------+--------+---------+-------------+

+-------------+----------------------------------------------------------+
|Column       |Content                                                   |
+-------------+----------------------------------------------------------+
|emp_id       |[[1, 1], [3, 1], [5, 1], [4, 1], [2, 1]]                  |
|emp_name     |[[Elliot, 1], [John, 1], [Sam, 1], [Peter, 1], [David, 1]]|
|emp_city     |[[Sydney, 2], [Melbourne, 2], [Brisbane, 1]]              |
|emp_expertise|[[Python, 2], [Java, 2], [Scala, 1]]                      |
+-------------+----------------------------------------------------------+

+-------------+----------------------------------------------------------+

|Column |Content |

+-------------+----------------------------------------------------------+

|emp_id |[[1, 1], [3, 1], [5, 1], [4, 1], [2, 1]] |

|emp_name |[[Elliot, 1], [John, 1], [Sam, 1], [Peter, 1], [David, 1]]|

|emp_city |[[Sydney, 2], [Melbourne, 2], [Brisbane, 1]] |

|emp_expertise|[[Python, 2], [Java, 2], [Scala, 1]] |

+-------------+----------------------------------------------------------+

Where each column is counted for its occurrence. For example, the emp_city column values “Melbourne”

…

08: Learn Spark how to convert RDD in Java to Dataframe with examples

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. Java RDD to Dataframe The following code reads a text file as shown below into a Java RDD. orders.txt


"ord123","John" 
"ord456","Peter"
"ord789","Samuel"

"ord123","John"

"ord456","Peter"

"ord789","Samuel"

The Java code that read the text...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Spark on Zeppelin – convert DataFrames to RDD[Row] and RDD[Row] to DataFrame

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Important: It is not a best practice to mutate values or to use RDD directly as opposed to using Dataframes....

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Spark writing RDDs to multiple text files & HAR to solve small files issue

We know that the following code snippets in Spark will write each JavaRDD element to a single file


employeesRdd.saveAsTextFile(pathToHdfs)

employeesRdd.saveAsTextFile(pathToHdfs)

What if you want to write each employee history to a separate file? Step 1: Create a JavaPairRDD from JavaRDD


JavaPairRDD<String, String> empPairRdd = employeesRdd.mapToPair(new PairFunction<Employee, String, String>() {

    @Override
    public Tuple2<String, String> call(Employee t) throws Exception {
        return new Tuple2<String, String>(t.getEmpId, t.getEmpHistoryDump());
    }
});

JavaPairRDD<String, String> empPairRdd = employeesRdd.mapToPair(new PairFunction<Employee, String, String>() {

@Override

public Tuple2<String, String> call(Employee t) throws Exception {

return new Tuple2<String, String>(t.getEmpId, t.getEmpHistoryDump());

}

});

Step 2: Create a MultipleOutputFormat, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08. Setting up & getting started with Hadoop on Mac

This tutorial outlines the basic steps to get started with Hadoop on Mac OS. This is a single-node cluster with YARN as the resource manager.

1. Install Xcode

Xcode can be installed via Apple appstore. Xcode is Apple’s Integrated Development Environment (IDE). Xcode is a large suite of software development tools and libraries from Apple.

…

09: Append to AVRO from Spark with distributed Zookeeper locking using Apache’s Curator framework

Step 1: The pom.xml file that has all the relevant dependencies to Spark, Avro & hadoop libraries.


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-spark</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-spark</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>4.8.1</junit.version>
        <hadoop.version>2.7.2</hadoop.version>
        <spark-version>1.3.0</spark-version>
        <avro.version>1.8.1</avro.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Hadoop API -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
            <scope>provided</scope>
        </dependency>

        <!-- Apache Spark -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark-version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
            <scope>provided</scope>
        </dependency>

        <!-- Apache Spark -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark-version}</version>
        </dependency>

        <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>spark-xml_2.11</artifactId>
            <version>0.4.0</version>
        </dependency>
        
        
        <!-- AVRO -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
         <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-mapred</artifactId>
            <version>${avro.version}</version>
            <classifier>hadoop2</classifier>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.avro</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency> 
    </dependencies>

    <build>
        <plugins>
            <!-- build an uber jar with hadoop dependencies included -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

100

101

102

103

104

105

106

107

108

109

110

111

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-spark</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-spark</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.7</maven.compiler.source>

<maven.compiler.target>1.7</maven.compiler.target>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<junit.version>4.8.1</junit.version>

<hadoop.version>2.7.2</hadoop.version>

<spark-version>1.3.0</spark-version>

<avro.version>1.8.1</avro.version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>${junit.version}</version>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

<scope>provided</scope>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.11</artifactId>

<version>${spark-version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

<scope>provided</scope>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-sql_2.11</artifactId>

<version>${spark-version}</version>

</dependency>

<groupId>com.databricks</groupId>

<artifactId>spark-xml_2.11</artifactId>

</dependency>

<groupId>org.apache.avro</groupId>

<version>${avro.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro-mapred</artifactId>

<version>${avro.version}</version>

<classifier>hadoop2</classifier>

<groupId>org.apache.avro</groupId>

</exclusion>

</exclusions>

</dependency>

</dependencies>

<build>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>

Step 2: Avro schema /schema/employee.avsc file under src/main/resources folder.


{
	"type": "record",
	"namespace": "com.myapp",
	"name": "organization",
	"fields": [
	    {
	    	"name": "name", 
	    	"type": "string", 
	    	"doc": "The employee name",
	    	"default": null
	    }
	    
	    ,
	    {
	    	"name": "title", 
	    	"type": "string", 
	    	"doc": "employee title",
	    	"default": null
	    }

	 ]
}

{

"type": "record",

"namespace": "com.myapp",

"name": "organization",

"fields": [

{

"name": "name",

"type": "string",

"doc": "The employee name",

"default": null

}

{

"name": "title",

"type": "string",

"doc": "employee title",

"default": null

}

]

}

Step 3: Spark job that creates random data into a RDD named “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

09: Databricks – Spark Problem 2

Prerequisite: Extends Databricks – Spark problem 1. Problem: Convert the below table

+--------+------------+---+
|emp_name|     product|qty|
+--------+------------+---+
|    John|    Notebook|  7|
|   Peter|Mobile phone| 12|
|     Sam|      Camera|  3|
|   David|Mobile phone|  9|
|  Elliot|    Notebook|  3|
|  Elliot|      Camera|  1|
|    John|Mobile phone| 13|
|   Peter|Mobile phone|  2|
+--------+------------+---+

+--------+------------+---+

|emp_name| product|qty|

+--------+------------+---+

| John| Notebook| 7|

| Peter|Mobile phone| 12|

| Sam| Camera| 3|

| David|Mobile phone| 9|

| Elliot| Notebook| 3|

| Elliot| Camera| 1|

| John|Mobile phone| 13|

| Peter|Mobile phone| 2|

+--------+------------+---+

+--------+------+------------+--------+
|emp_name|Camera|Mobile phone|Notebook|
+--------+------+------------+--------+
|  Elliot|     1|        null|       3|
|    John|  null|          13|       7|
|     Sam|     3|        null|    null|
|   Peter|  null|          14|    null|
|   David|  null|           9|    null|
+--------+------+------------+--------+

+--------+------+------------+--------+

+--------+------+------------+--------+

| Elliot| 1| null| 3|

| John| null| 13| 7|

| Sam| 3| null| null|

+--------+------+------------+--------+

Step 1: Create a new Python notebook, and attach it to a new cluster. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

09: Running a Spark job on YARN cluster in Cloudera

This assumes that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. It is also important to enable History server as per Before running a Spark job on YARN. Step 1: Open a “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 3 4 5 6 7 8 9 10 11 … 13 »

Blog Archives

07: Getting started with Spark-submit

07: Learn Spark Dataframes to do ETL in Java with examples

07: Spark on Zeppelin – window functions in Scala

07: spark-xml to split & read very large XML files

08: Apache Spark standalone cluster on Docker to read from AWS S3 bucket & write to MySQL database

08: Databricks – Spark problem 1

08: Learn Spark how to convert RDD in Java to Dataframe with examples

08: Spark on Zeppelin – convert DataFrames to RDD[Row] and RDD[Row] to DataFrame

08: Spark writing RDDs to multiple text files & HAR to solve small files issue

08. Setting up & getting started with Hadoop on Mac

09: Append to AVRO from Spark with distributed Zookeeper locking using Apache’s Curator framework

09: Databricks – Spark Problem 2

09: Running a Spark job on YARN cluster in Cloudera

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer