Step 1: Create a Java project with Maven
|
1 2 3 4 |
bash-4.1$ cd projects bash-4.1$ mvn archetype:generate -DgroupId=com.mytutorial -DartifactId=simple-storm -DinteractiveMode=false |
Step 2: Open eclipse and import it as an existing maven project by clicking File –> Import –> Existing Maven Projects –> “/home/cloudera/projects/simple-storm”, which has the pom.xml file. Step 3: Add Apache Storm project to pom.xml and then select the project...
This extends Installing & getting started with Apache Storm on Cloudera quickstart and Simple Apache Storm application running inside Eclipse in a local cluster to run the same example in Storm cluster mode. Step 1: Firstly modify the “storm.yaml” in the folder “/opt/storm/apache-storm-1.1.1/conf” … Read more ›...
Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook.
Step 1: Pull this from the docker hub,
…
This Spark tutorial extends Spark RDD joins in Scala tutorial and Setting up Spark and Scala with Maven. Step 1: Let’s take a simple example of joining a student to department. This will be written in an SQL world as:
|
1 2 3 4 5 6 |
SELECT d.name, count(*) FROM Student s JOIN Department d on s.deptId = d.id group by d.name |
Step 2: Let’s create classes to represent Student and...
This extends Spark tutorial – writing a file from a local file system to HDFS.
This tutorial assumes that you have set up Cloudera as per “cloudera quickstart vm tutorial installation” YouTube videos that you can search Google or YouTube. You can install it on VMWare (non commercial use) or on VirtualBox.
…
Prerequisite: At least 12GB+ RAM (i.e. 4GB+ for operating system & 8GB+ for Cloudera), although 16 GB+ is preferred. 80GB Hard Disk. Cloudera runs on CentOS, which is the community edition of the Linux. Windows system must support 64-bit.
This post is obsolete with the newer version of Cloudera.
…
This extends Setting up Java, Maven & Eclipse and Setting up Scala & practicing the concepts via REPL the Scala way for Java developers. Step 1: Scala IDE is an eclipse plugin. Following is the Scala IDE update site URL for Eclipse 4.4 (Luna) and Eclipse 4.5 (Mars). … Read...
What is Avro IDL? Avro IDL (i.e. Interface Description Language) is a high-level language to write Avro schemata. You can generate Java, C++, and Python objects from the Avro IDL files. These files generally have the “.avdl” extension. Step 1: Write the “order.avdl” … Read more ›...
This extends Apache Kafka example with Java Producer & Consumer Tutorial – Part 2. This covers Apache Kafka example with toxiproxy & producer timeout. The producer code needs to be robust to handle any network issues like timeouts & latency issues. In this tutorial a tool named toxiproxy is used to simulate network issues.
…
This extends 01: Docker tutorial with Java & Maven. Step 1: Create a Docker network where all 3 containers – Spark master (i.e. Stand-alone cluster manager), Spark worker, and Spark driver will be deployed to.
|
1 2 |
docker network create docker_test_infra || true |
Step 2: The image docker file “ … Read more ›...
This extends the Convert XML file To Sequence File With Hadoop libaries. Avro files are schema driven & support schema evolution, which means you can add new columns & modify existing columns. Step 1: The pom.xml file should include the Apache Spark libraries as shown below. … Read more ›...
Step 1: Create a simple maven project named “simple-hadoop“.
|
1 2 3 |
mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-hadoop |
Step 2: Import the “simple-hadoop” maven project into eclipse or IDE of your choice. Step 3: Modify the pom.xml file include 1) relevant Hadoop libraries 2) The shade plugin to create a single jar (i.e. … Read more ›...
Prerequisite: Extends Databricks getting started – Spark, Shell, SQL.
What is SCD Type 1
SCD stands for Slowly Changing Dimension, and it was explained in 10 Data warehouse interview Q&As.
…
I have a good news, I got job offer. I like to share with you from my bottom of my heart feeling thankful to you.