This tutorial extends Getting started with Databricks. Currently spark does not support recursion like you can use in SQL via “Common Table Expression“.
Step 1: Login to Databricks notebook:
https://community.cloud.databricks.com/login.html.
Step 2: Create a CLUSTER and it will take a few minutes to come up.
…
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.
|
1 2 3 4 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \ cloudera/quickstart /usr/bin/docker-quickstart |
Install maven Step 2: Download maven from “http://mirror.ventraip.net.au/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz”
|
1 2 |
[root@quickstart /]# sudo curl -L -O http://mirror.ventraip.net.au/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz |
|
1 2 3 4 5 |
[root@quickstart /]# cp apache-maven-3.6.1-bin.tar.gz /opt [root@quickstart /]# cd /opt [root@quickstart /]# sudo tar xvfz apache-maven-3.6.1-bin.tar.gz [root@quickstart /]# sudo rm -f apache-maven-3.6.1-bin.tar.gz |
Step 3: Add it to your profile so that you can run the “mvn” … Read more ›...
This tutorial extends Getting started with Databricks. You can run these in the Databricks notebook.
Step 1: Login to Databricks notebook:
https://community.cloud.databricks.com/login.html.
Step 2: Create a CLUSTER and it will take a few minutes to come up.
…
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.
|
1 2 3 4 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \ cloudera/quickstart /usr/bin/docker-quickstart |
Add new user & add to sudoers Step 2: “-m” creates a home directory “/home/kafka”, and the new user will be added to “ … Read more ›...
This requires 26: Docker Tutorial: Apache Kafka install, create topic & publish message on Cloudera quickstart. It is important that you continue from the previous tutorial. Add maven path to ~/.bash_profile Step 1: Maven is already in the quickstart/cloudera Docker image.
|
1 |
[kafka@quickstart /]$ vi ~/.bash_profile |
Modify so that “mvn” … Read more ›...
This extends 27: Docker Tutorial: Apache Kafka with Java API on Cloudera quickstart Install Java 8 Step 1: Install Java 8.
|
1 |
[kafka@quickstart my-app]$ sudo yum install java-1.8.0-openjdk-devel |
|
1 2 3 4 5 |
[kafka@quickstart my-app]$ java -version openjdk version "1.8.0_212" OpenJDK Runtime Environment (build 1.8.0_212-b04) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode) |
|
1 2 3 4 5 6 7 8 9 |
[kafka@quickstart my-app]$ update-alternatives --config java There is 1 program that provides 'java'. Selection Command ----------------------------------------------- *+ 1 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java |
JAVA_HOME Step 2: Set JAVA_HOME to Java 8 so that Maven uses Java 8. … Read more ›...
Prerequisite: Docker is installed on your Windows or Mac, and you have a basic understanding of Docker. Docker tutorials step by step | Hadoop, Hive, Impala & Spark on Cloudera quickstart on Docker tutorials Step 1: Pull the modified image “gdancik/cloudera” … Read more ›...
This extends the tutorial 1. Apache Pig Getting started and 2. Apache Pig: Regex (Regular expressions).
Input Data
scores.xml in folder:/Users/arulk/projects representing marks of 4 students in 3 subjects:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
<scores> <subject> <name>Science</name> <marks> <mark>80</mark> <mark>75</mark> <mark>89</mark> <mark>90</mark> </marks> </subject> <subject> <name>Maths</name> <marks> <mark>90</mark> <mark>87</mark> <mark>78</mark> <mark>92</mark> </marks> </subject> <subject> <name>English</name> <marks> <mark>78</mark> <mark>88</mark> <mark>65</mark> <mark>99</mark> </marks> </subject> </scores> |
Step 1: Start pig in local file system mode.
…
Extends Getting started with HBase. HBase is a NoSQL database. HBase is a columnar database. #1. In HBase, you create the Table & column-families “create” command to create a table with column families. You can later on use “ … Read more ›...
This extends Docker Tutorial: Apache Kafka with Python 3 on Cloudera quickstart Step 1: Create the pyspark streaming code in python. driver.py
|
1 |
(my-app_env) [root@quickstart my-app]# vi driver.py |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
from pyspark import SparkConf, SparkContext from pyspark.streaming import StreamingContext from mypackage import simple if __name__ == "__main__": conf = SparkConf().setAppName("Simple App") conf = conf.setMaster("local[*]") sc = SparkContext(conf=conf) ssc = StreamingContext(sc, 10) # every 10 seconds simple.SimpleSpark().myfunc(ssc) ssc.start(); ssc.awaitTermination(); |
simple.py – receive messages from a topic
|
1 |
(my-app_env) [root@quickstart my-app]# vi mypackage/simple.py |
|
1 2 3 4 5 6 7 8 9 10 11 |
from pyspark.streaming import StreamingContext from pyspark.streaming.kafka import KafkaUtils class SimpleSpark: def myfunc(self, ssc): # 2181 is the zookeeper port & 9092 is the kafka broker port kafkaStream = KafkaUtils.createStream(ssc, 'localhost:2181', 'spark-streaming', {'MyTestTopic':1}) lines = kafkaStream.map(lambda x : x[1]) lines.pprint() |
Build the .egg file Step 2: Create setup.py to build the .egg file, … Read more ›...
This extends 27: Docker Tutorial: Apache Kafka with Java API on Cloudera quickstart. Pre-requisite: Java 8 & Kafka installed with the “MyTestTopic” as per the previous tutorials. Install scala & sbt We will install Scala 2.10.7 from “https://www.scala-lang.org/download/2.10.7.html” … Read more ›...
HBase is a NoSQL database used in Hadoop world to store “Big Data”. This extends Understanding HBase (NoSQL) database basics in 7 steps.
Step 1: Create a Maven based Java project
|
1 2 3 |
mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-hbase |
Step 2: Import the project into eclipse & modify the pom.xml file
to add Hadoop &
…
Step 1: Create an HBase table named “sequence_numbers” with one column-family “i”. You can create this via HBase shell.
|
1 2 3 4 |
$ hbase shell hbase(main):001:0> create 'sequence_numbers', {NAME => 'i'} |
Step 2: You can use the HBase API in Java to insert a new record via “assigCurrentValue()“, and perform atomic operations in “ … Read more ›...