04: Running a Simple Spark Job in local & cluster modes

Posted on July 22, 2016 by

Step 1: Create a simple maven Spark project using “-B” for non-interactive mode.


[cloudera@quickstart ~]$ mvn archetype:generate -DgroupId=com.mytutorial -DartifactId=simple-spark -B

[cloudera@quickstart ~]$ mvn archetype:generate -DgroupId=com.mytutorial -DartifactId=simple-spark -B

Step 2: Import the maven project “simple-spark” into eclipse. Step 3: The pom.xml file should have the relevant dependency jars as shown below.


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-spark</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-spark</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>4.8.1</junit.version>
        <hadoop.version>2.7.2</hadoop.version>
        <spark-version>1.3.0</spark-version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        
        <!-- Hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- Apache Spark -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark-version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-spark</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-spark</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<junit.version>4.8.1</junit.version>

<hadoop.version>2.7.2</hadoop.version>

<spark-version>1.3.0</spark-version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>${junit.version}</version>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>

<version>${hadoop.version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.11</artifactId>

<version>${spark-version}</version>

<groupId>javax.servlet</groupId>

</exclusion>

</exclusions>

</dependency>

</dependencies>

</project>

Step 4:…

Members Only Content

Based on our outdated book (published 2008): Java/JEE Job Interview Companion (Sold 35,000+ copies) via Amazon.com | Amazon.com reviews | Good reads reviews

"After landing a job I have kept on reading the book and recommend it to all my friends..."-- JavaRanch Forum

I can safely say you'd have to buy probably at least 15 individual books and supplement this with reading articles off the internet to get the equivalent diversity of coverage elsewhere...... -- Amazon.uk By Mr. Jeremy

... I was dejected with my rejections in interview about 8 years ago. I just read around 100 pages from your book and since then I have not flunked my interviews! Vivek via LinkedIn.com

This book does a very good job of highlighting the key topics that are likely to be discussed during interviews and giving various levels of answers. Josh Berry via Amazon.com

More reviews

The contents in this Java-Success are copyrighted and from EmpoweringTech pty ltd. The EmpoweringTech pty ltd has the right to correct or enhance the current content without any prior notice. These are general advice only, and one needs to take his/her own circumstances into consideration. The EmpoweringTech pty ltd will not be held liable for any damages caused or alleged to be caused either directly or indirectly by these materials and resources. Any trademarked names or labels used in this blog remain the property of their respective trademark owners. Links to external sites do not imply endorsement of the linked-to sites. Privacy Policy

Responsive WordPress Theme powered by CyberChimps

Top

300+ Big Data Interview FAQs

Big Data Tutorials

300+ Java Interview FAQs

Java Code Quality FAQs

16+ Java Key Areas to go places

800+ Java Interview Q&As

Java Tutorials

Java Architect FAQs

Java Coding Q&As

04: Running a Simple Spark Job in local & cluster modes

Members Only Content

Java Interview FAQs

Big Data Interview FAQs

800+ Java Interview Q&As

Reviews

More reviews

Disclaimer