Tutorials - Big Data Archives | Page 4 of 13 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 2 3 4 5 6 7 8 … 13 »

03: Learn Spark & Parquet Write & Read in Java by example

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. AVRO (i.e row oriented) and Parquet (i.e. column oriented) file formats are HDFS (i.e. Hadoop Distributed File System) friendly binary data formats as they store data compressed...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

03: Spark on Zeppelin – DataFrame Operations in Scala

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As.

This tutorial extends Apache Zeppelin on Docker Tutorial – Docker pull from Docker hub and Spark stand-alone to read a file from local file system

val lines = sc.textFile("file:///zeppelin/seed/employees.txt")

case class Employee (id: Integer, name: String, location: String, salary: Double )

val dfEmployees = lines.map(s => s.split(",")).map(s => Employee(s(0).toInt, s(1), s(2), s(3).toDouble)).toDF()

val lines = sc.textFile("file:///zeppelin/seed/employees.txt")

case class Employee (id: Integer, name: String, location: String, salary: Double )

val dfEmployees = lines.map(s => s.split(",")).map(s => Employee(s(0).toInt, s(1), s(2), s(3).toDouble)).toDF()

1.…

03: Spark tutorial – reading a Sequence File from HDFS

This extends Spark submit – reading a file from HDFS. A SequenceFile is a flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats. Like CSV, Sequence files do not store meta data, hence only schema evolution is appending new fields to the end...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

03. Setting up Scala and Spark with Maven

This assumes that you set up Scala IDE for eclipse as per Setting up Scala IDE & using Maven plugins.

Step 1: Create a Scala project in Maven.


mvn archetype:generate "-DarchetypeGroupId=net.alchim31.maven"  "-DarchetypeArtifactId=scala-archetype-simple"  "-DgroupId=com.mytutorial" "-DartifactId=spark-and-scala" -B

mvn archetype:generate "-DarchetypeGroupId=net.alchim31.maven" "-DarchetypeArtifactId=scala-archetype-simple" "-DgroupId=com.mytutorial" "-DartifactId=spark-and-scala" -B

Step 2: Import Maven project into eclipse.

File ->

…

03. Upgrading Java version in Cloudera Quickstart

This extends Installing & getting started with Cloudera QuickStart on VMWare for windows to upgrade the Java version from 1.7.x to 1.8.x. Firstly, check if your Cloudera version supports Java 1.8. This example is running on Cloudera 5.12.

This post is obsolete with the new Cloudera version.

…

04: Apache Kafka JSON example with Java Producer & Consumer Tutorial

This extends Apache Kafka example with Java – getting started tutorial | Apache Kafka example with Java Producer & Consumer Tutorial. This tutorial looks at Producer/Consumer with JSON payload using Java & Kafka broker on Docker. Refer to previous getting started tutorial to run Kafka broker on Docker & …...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Apache Spark cluster – Docker compose tutorial with Java, Maven & spark-submit

This extends the previous Docker tutorials. This is a stand-alone Spark cluster tutorial on Docker compose. Step 1: The image docker file “spark.dockerfile” in the folder “docker-test/docker/spark/”.

FROM maven:3.5-jdk-8-alpine

RUN apk update \
      && apk upgrade \
      && apk add --update bash \
      && apk add --update curl \
      && rm -rf /var/cache/apk/*

ARG SPARK_ARCHIVE=https://www.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz

RUN curl -s $SPARK_ARCHIVE | tar -xz -C /usr/local/

ENV SPARK_HOME /usr/local/spark-2.3.0-bin-hadoop2.7
ENV PATH $PATH:$SPARK_HOME/bin

EXPOSE 4040 5005 6066 7077 8080 8081

WORKDIR $SPARK_HOME

FROM maven:3.5-jdk-8-alpine

RUN apk update \

&& apk upgrade \

&& apk add --update bash \

&& apk add --update curl \

&& rm -rf /var/cache/apk/*

ARG SPARK_ARCHIVE=https://www.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz

RUN curl -s $SPARK_ARCHIVE | tar -xz -C /usr/local/

ENV SPARK_HOME /usr/local/spark-2.3.0-bin-hadoop2.7

ENV PATH $PATH:$SPARK_HOME/bin

EXPOSE 4040 5005 6066 7077 8080 8081

WORKDIR $SPARK_HOME

Step 2: The pom.xml file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.myproject</groupId>
  <artifactId>hellodocker</artifactId>
  <version>1.0</version>
  
  <properties>
   <maven.compiler.source>1.8</maven.compiler.source>
   <maven.compiler.target>1.8</maven.compiler.target>
</properties>

  <dependencies>
    <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.3.0</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>  

  <build>
    <plugins>
      <plugin>
        <!-- Build an executable JAR -->
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <classpathPrefix>lib/</classpathPrefix>
              <mainClass>com.mypkg.HelloDocker</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<groupId>com.myproject</groupId>

<artifactId>hellodocker</artifactId>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

</properties>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.11</artifactId>

<scope>provided</scope>

</dependency>

</dependencies>

<build>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-jar-plugin</artifactId>

<mainClass>com.mypkg.HelloDocker</mainClass>

</manifest>

</archive>

</configuration>

</plugin>

</plugins>

</build>

</project>

Step 3: The Spark code “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Convert XML file To an Avro File with Apache Spark – writing & reading

This extends Convert XML file To an Avro File – writing & reading. Step 1: The pom.xml file should include the Apache Spark & Avro libraries as shown below.


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>my.app</groupId>
    <artifactId>datawrangling-java</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>datawrangling-java</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>4.8.1</junit.version>
        <avro.version>1.7.7</avro.version>
        <hadoop.version>2.7.2</hadoop.version>
        <spark-version>1.3.0</spark-version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- AVRO -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
         <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-mapred</artifactId>
            <version>${avro.version}</version>
            <classifier>hadoop2</classifier>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.avro</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency> 
        
        <!-- Hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- XML -->
        <dependency>
            <groupId>org.eclipse.persistence</groupId>
            <artifactId>org.eclipse.persistence.moxy</artifactId>
            <version>2.6.2</version>
        </dependency>

        <!-- Spark -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark-version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<artifactId>datawrangling-java</artifactId>

<version>0.0.1-SNAPSHOT</version>

<name>datawrangling-java</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<junit.version>4.8.1</junit.version>

<avro.version>1.7.7</avro.version>

<hadoop.version>2.7.2</hadoop.version>

<spark-version>1.3.0</spark-version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>${junit.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<version>${avro.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro-mapred</artifactId>

<version>${avro.version}</version>

<classifier>hadoop2</classifier>

<groupId>org.apache.avro</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.eclipse.persistence</groupId>

<artifactId>org.eclipse.persistence.moxy</artifactId>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.11</artifactId>

<version>${spark-version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

</dependencies>

</project>

Step 2: The report.xml file under “src/main/resources/data”. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Create new or append to an existing AVRO file tutorial

This extends Create or append a file to HDFS – Hadoop API tutorial to write an AVRO file to HDFS. Step 1: Include the AVRO library files in the pom.xml file.


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-hadoop</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-hadoop</name>
    <url>http://maven.apache.org</url>

    <properties>
        <hadoop.version>2.7.2</hadoop.version>
        <avro.version>1.8.1</avro.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <!-- Hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <!-- Required for the  FsInput class  -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-mapred</artifactId>
            <version>${avro.version}</version>
            <classifier>hadoop2</classifier>
        </dependency>

    </dependencies>
    <build>
        <plugins>
            <!-- build an uber jar with hadoop dependencies included -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-hadoop</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-hadoop</name>

<url>http://maven.apache.org</url>

<hadoop.version>2.7.2</hadoop.version>

<avro.version>1.8.1</avro.version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>${hadoop.version}</version>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>

<version>${hadoop.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<version>${avro.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro-mapred</artifactId>

<version>${avro.version}</version>

<classifier>hadoop2</classifier>

</dependency>

</dependencies>

<build>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>

Step 2: The AVRO files are schema based. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Databricks – Spark SCD Type 2

Prerequisite: Extends 03: Databricks – Spark SCD Type 1. What is SCD Type 2 SCD stands for Slowly Changing Dimension, and it was explained in 10 Data warehouse interview Q&As. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Learn how to connect to HBase from Spark using Java API

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. What is HBase? Apache HBase is a NoSQL database used for random and real-time read/write access to your Big Data. It is built on top of the...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Running a Simple Spark Job in local & cluster modes

Step 1: Create a simple maven Spark project using “-B” for non-interactive mode.


[cloudera@quickstart ~]$ mvn archetype:generate -DgroupId=com.mytutorial -DartifactId=simple-spark -B

[cloudera@quickstart ~]$ mvn archetype:generate -DgroupId=com.mytutorial -DartifactId=simple-spark -B

Step 2: Import the maven project “simple-spark” into eclipse. Step 3: The pom.xml file should have the relevant dependency jars as shown below.


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-spark</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-spark</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>4.8.1</junit.version>
        <hadoop.version>2.7.2</hadoop.version>
        <spark-version>1.3.0</spark-version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        
        <!-- Hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- Apache Spark -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark-version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-spark</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-spark</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<junit.version>4.8.1</junit.version>

<hadoop.version>2.7.2</hadoop.version>

<spark-version>1.3.0</spark-version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>${junit.version}</version>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.11</artifactId>

<version>${spark-version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

</dependencies>

</project>

Step 4: Write the simple Spark job “SimpleSparkJob.java” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04: Spark on Zeppelin – DataFrame joins in Scala

This tutorial extends the series: Spark on Apache Zeppelin Tutorials.

1. Create “Orders” DataFrame

import org.apache.spark.sql._
import org.apache.spark.sql.types._

val orders = Seq (
    Row ("order-1", "cust-234", 234.50),
    Row ("order-2", "cust-234", 347.00),
    Row ("order-3", "cust-337", 67.00),
    Row ("order-4", "cust-425", 84.50),
    Row ("order-5", "cust-456", 134.50)
)

val orderSchema = List(
  StructField("order_num", StringType, true),
  StructField("customer_num", StringType, true),
  StructField("amount", DoubleType, true)
)

val ordersDf = spark.createDataFrame(sc.parallelize(orders), StructType(orderSchema))

ordersDf.show()

import org.apache.spark.sql._

import org.apache.spark.sql.types._

val orders = Seq (

Row ("order-1", "cust-234", 234.50),

Row ("order-2", "cust-234", 347.00),

Row ("order-3", "cust-337", 67.00),

Row ("order-4", "cust-425", 84.50),

Row ("order-5", "cust-456", 134.50)

)

val orderSchema = List(

StructField("order_num", StringType, true),

StructField("customer_num", StringType, true),

StructField("amount", DoubleType, true)

)

val ordersDf = spark.createDataFrame(sc.parallelize(orders), StructType(orderSchema))

ordersDf.show()


+---------+------------+------+
|order_num|customer_num|amount|
+---------+------------+------+
|  order-1|    cust-234| 234.5|
|  order-2|    cust-234| 347.0|
|  order-3|    cust-337|  67.0|
|  order-4|    cust-425|  84.5|
|  order-5|    cust-456| 134.5|
+---------+------------+------+

+---------+------------+------+

|order_num|customer_num|amount|

+---------+------------+------+

| order-1| cust-234| 234.5|

| order-2| cust-234| 347.0|

| order-3| cust-337| 67.0|

| order-4| cust-425| 84.5|

| order-5| cust-456| 134.5|

+---------+------------+------+

2. Create “Customers” DataFrame

import org.apache.spark.sql._
import org.apache.spark.sql.types._

val customers = Seq (
    Row ("cust-234", "John"),
    Row ("cust-239", "Peter"),
    Row ("cust-337", "Samuel"),
    Row ("cust-425", "Jordan"),
    Row ("cust-456", "George")
)

val customerSchema = List(
  StructField("customer_num", StringType, true),
  StructField("customer_name", StringType, true)
)

val customersDf = spark.createDataFrame(sc.parallelize(customers), StructType(customerSchema))

customersDf.show()

import org.apache.spark.sql._

import org.apache.spark.sql.types._

val customers = Seq (

Row ("cust-234", "John"),

Row ("cust-239", "Peter"),

Row ("cust-337", "Samuel"),

Row ("cust-425", "Jordan"),

Row ("cust-456", "George")

)

val customerSchema = List(

StructField("customer_num", StringType, true),

StructField("customer_name", StringType, true)

)

val customersDf = spark.createDataFrame(sc.parallelize(customers), StructType(customerSchema))

customersDf.show()


+------------+-------------+
|customer_num|customer_name|
+------------+-------------+
|    cust-234|         John|
|    cust-239|        Peter|
|    cust-337|       Samuel|
|    cust-425|       Jordan|
|    cust-456|       George|
+------------+-------------+

+------------+-------------+

|customer_num|customer_name|

+------------+-------------+

| cust-234| John|

| cust-239| Peter|

| cust-337| Samuel|

| cust-425| Jordan|

| cust-456| George|

+------------+-------------+

You can perform a number of joins between DataFrames. Default is the inner join. Joins can be of: inner,

…

« 1 2 3 4 5 6 7 8 … 13 »

Blog Archives

03: Learn Spark & Parquet Write & Read in Java by example

03: Spark on Zeppelin – DataFrame Operations in Scala

03: Spark tutorial – reading a Sequence File from HDFS

03. Setting up Scala and Spark with Maven

03. Upgrading Java version in Cloudera Quickstart

04: Apache Kafka JSON example with Java Producer & Consumer Tutorial

04: Apache Spark cluster – Docker compose tutorial with Java, Maven & spark-submit

04: Convert XML file To an Avro File with Apache Spark – writing & reading

04: Create new or append to an existing AVRO file tutorial

04: Databricks – Spark SCD Type 2

04: Learn how to connect to HBase from Spark using Java API

04: Running a Simple Spark Job in local & cluster modes

04: Spark on Zeppelin – DataFrame joins in Scala

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer