This tutorial assumes that you have set up Cloudera as per “cloudera quickstart vm tutorial installation” YouTube videos that you can search Google or YouTube. You can install it on...
Blog Archives
01B: Spark tutorial – writing to HDFS from Spark using Hadoop API
Step 1: The “pom.xml” that defines the dependencies for Spark & Hadoop APIs. Step 2: The Spark job that writes numbers 1 to 10 to 10 different files on HDFS....
02: Spark tutorial – reading a file from HDFS
This extends Spark tutorial – writing a file from a local file system to HDFS.
This tutorial assumes that you have set up Cloudera as per “cloudera quickstart vm tutorial installation” YouTube videos that you can search Google or YouTube.
…
03: Spark tutorial – reading a Sequence File from HDFS
This extends Spark submit – reading a file from HDFS. A SequenceFile is a flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats....
04: Running a Simple Spark Job in local & cluster modes
Step 1: Create a simple maven Spark project using “-B” for non-interactive mode. Step 2: Import the maven project “simple-spark” into eclipse. Step 3: The pom.xml file should have the...
05: Spark SQL & CSV with DataFrame Tutorial
Step 1: Create a simple maven project. Step 2: Import the “simple-spark” maven project into eclipse or IDE of your choice. Step 3: Modify the pom.xml file include 1) relevant...
05a: Spark DataFrame simple tutorial
A DataFrame is an immutable distributed collection of data like an RDD, but unlike an RDD, data is organized into named columns of a table in a relational database. This...
06: Spark Streaming with Flume Avro Sink Tutorial
This extends Running a Simple Spark Job in local & cluster modes and Apache Flume with JMS source (Websphere MQ) and HDFS sink. In this tutorial a Flume sink will...
07: spark-xml to split & read very large XML files
Processing very large XML files can be a bit tricky as they cannot be processed line by line in parallel as you would do with CSV files. The xml file...
08: Spark writing RDDs to multiple text files & HAR to solve small files issue
We know that the following code snippets in Spark will write each JavaRDD element to a single file What if you want to write each employee history to a separate...
Top
