This assumes that you have set up Java, Maven, and Eclipse.
To run JEE based web apps, you need an application server that contains a web container. There are a number of application servers like Apache Tomcat, JBoss, GlassFish, etc. This tutorial will use Apache Tomcat.
…
This tutorial outlines the basic steps to get started with Hadoop on Mac OS. This is a single-node cluster with YARN as the resource manager.
1. Install Xcode
Xcode can be installed via Apple appstore. Xcode is Apple’s Integrated Development Environment (IDE). Xcode is a large suite of software development tools and libraries from Apple.
…
Related Links 1. Getting started with Apache Tomcat application server by deploying a simple JEE Application. … Read more ›...
Related Links 1. Java Web App with Tomcat & Maven beginner video tutorial. 2. Getting started with Apache Tomcat application server by deploying a simple JEE Application. … Read more ›...
Step 1: The pom.xml file that has all the relevant dependencies to Spark, Avro & hadoop libraries.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.mytutorial</groupId> <artifactId>simple-spark</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <name>simple-spark</name> <url>http://maven.apache.org</url> <properties> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <junit.version>4.8.1</junit.version> <hadoop.version>2.7.2</hadoop.version> <spark-version>1.3.0</spark-version> <avro.version>1.8.1</avro.version> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> <!-- Hadoop API --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>javax.servlet</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> <scope>provided</scope> </dependency> <!-- Apache Spark --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>${spark-version}</version> <exclusions> <exclusion> <groupId>javax.servlet</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> <scope>provided</scope> </dependency> <!-- Apache Spark --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.11</artifactId> <version>${spark-version}</version> </dependency> <dependency> <groupId>com.databricks</groupId> <artifactId>spark-xml_2.11</artifactId> <version>0.4.0</version> </dependency> <!-- AVRO --> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>${avro.version}</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-mapred</artifactId> <version>${avro.version}</version> <classifier>hadoop2</classifier> <exclusions> <exclusion> <groupId>org.apache.avro</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> </dependency> </dependencies> <build> <plugins> <!-- build an uber jar with hadoop dependencies included --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> |
Step 2: Avro schema /schema/employee.avsc file under src/main/resources folder.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
{ "type": "record", "namespace": "com.myapp", "name": "organization", "fields": [ { "name": "name", "type": "string", "doc": "The employee name", "default": null } , { "name": "title", "type": "string", "doc": "employee title", "default": null } ] } |
Step 3: Spark job that creates random data into a RDD named “ … Read more ›...
Prerequisite: Extends Databricks – Spark problem 1. Problem: Convert the below table
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
+--------+------------+---+ |emp_name| product|qty| +--------+------------+---+ | John| Notebook| 7| | Peter|Mobile phone| 12| | Sam| Camera| 3| | David|Mobile phone| 9| | Elliot| Notebook| 3| | Elliot| Camera| 1| | John|Mobile phone| 13| | Peter|Mobile phone| 2| +--------+------------+---+ |
to
|
1 2 3 4 5 6 7 8 9 10 |
+--------+------+------------+--------+ |emp_name|Camera|Mobile phone|Notebook| +--------+------+------------+--------+ | Elliot| 1| null| 3| | John| null| 13| 7| | Sam| 3| null| null| | Peter| null| 14| null| | David| null| 9| null| +--------+------+------------+--------+ |
Step 1: Create a new Python notebook, and attach it to a new cluster. … Read more ›...
This assumes that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. It is also important to enable History server as per Before running a Spark job on YARN. Step 1: Open a “ … Read more ›...
Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Important: It is not a best practice to mutate values or to use RDD directly as opposed to using Dataframes....
This extends Setting up & getting started with Spark local mode with Sbt & Scala and Setting up & getting started with Hadoop on Mac.
The Spark job written in Scala will be reading the data from HDFS.
1. CSV data in HDFS
|
1 2 3 |
Maths, 85 English, 94 Science, 75 |
2.…
Industry strength code needs to statically & dynamically capture code quality. Also, more and more organizations are using “production quality” home assignments to shortlist candidates for job interviews. So, it really pays to set up code quality tools like SonarQube on your home development environment to get feedback on your code quality with the view to learm &
…
Related Links 1. JUnit with Mockito step by step. … Read more ›...
Input Data
scores.data in folder:/Users/arulk/projects representing marks of 4 students in 3 subjects:
|
1 2 3 4 5 |
Science, 80, 75, 89, 90 Maths, 90, 87, 78, 92 English, 78, 88, 65, 99 |
Calculate the max mark for each subject.
Step 1: Download Apache Pig from http://apache.mirror.digitalpacific.com.au/pig/ and extract the tar file.
…
Input Data & How Hadoop reads the Data
scores.data in folder:/Users/arulk/projects
|
1 2 3 4 5 6 |
Science, 80, 75, 89, 90 Maths, 90, 87, 78, 92 English, 78, 88, 65, 99 |
Mapper Input
The Hadoop “org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat” class read the input as key/value pairs. The default delimiter is tab.
…