This tutorial extends Getting started with Databricks. You can run these in the Databricks notebook.
Step 1: Login to Databricks notebook:
https://community.cloud.databricks.com/login.html.
Step 2: Create a CLUSTER and it will take a few minutes to come up.
…
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.
|
1 2 3 4 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 \ cloudera/quickstart /usr/bin/docker-quickstart |
Add new user & add to sudoers Step 2: “-m” creates a home directory “/home/kafka”, and the new user will be added to “ … Read more ›...
This requires 26: Docker Tutorial: Apache Kafka install, create topic & publish message on Cloudera quickstart. It is important that you continue from the previous tutorial. Add maven path to ~/.bash_profile Step 1: Maven is already in the quickstart/cloudera Docker image.
|
1 |
[kafka@quickstart /]$ vi ~/.bash_profile |
Modify so that “mvn” … Read more ›...
This extends 27: Docker Tutorial: Apache Kafka with Java API on Cloudera quickstart Install Java 8 Step 1: Install Java 8.
|
1 |
[kafka@quickstart my-app]$ sudo yum install java-1.8.0-openjdk-devel |
|
1 2 3 4 5 |
[kafka@quickstart my-app]$ java -version openjdk version "1.8.0_212" OpenJDK Runtime Environment (build 1.8.0_212-b04) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode) |
|
1 2 3 4 5 6 7 8 9 |
[kafka@quickstart my-app]$ update-alternatives --config java There is 1 program that provides 'java'. Selection Command ----------------------------------------------- *+ 1 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java |
JAVA_HOME Step 2: Set JAVA_HOME to Java 8 so that Maven uses Java 8. … Read more ›...
Prerequisite: Docker is installed on your Windows or Mac, and you have a basic understanding of Docker. Docker tutorials step by step | Hadoop, Hive, Impala & Spark on Cloudera quickstart on Docker tutorials Step 1: Pull the modified image “gdancik/cloudera” … Read more ›...
This extends the tutorial 1. Apache Pig Getting started and 2. Apache Pig: Regex (Regular expressions).
Input Data
scores.xml in folder:/Users/arulk/projects representing marks of 4 students in 3 subjects:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
<scores> <subject> <name>Science</name> <marks> <mark>80</mark> <mark>75</mark> <mark>89</mark> <mark>90</mark> </marks> </subject> <subject> <name>Maths</name> <marks> <mark>90</mark> <mark>87</mark> <mark>78</mark> <mark>92</mark> </marks> </subject> <subject> <name>English</name> <marks> <mark>78</mark> <mark>88</mark> <mark>65</mark> <mark>99</mark> </marks> </subject> </scores> |
Step 1: Start pig in local file system mode.
…
Extends Getting started with HBase. HBase is a NoSQL database. HBase is a columnar database. #1. In HBase, you create the Table & column-families “create” command to create a table with column families. You can later on use “ … Read more ›...
This extends Docker Tutorial: Apache Kafka with Python 3 on Cloudera quickstart Step 1: Create the pyspark streaming code in python. driver.py
|
1 |
(my-app_env) [root@quickstart my-app]# vi driver.py |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
from pyspark import SparkConf, SparkContext from pyspark.streaming import StreamingContext from mypackage import simple if __name__ == "__main__": conf = SparkConf().setAppName("Simple App") conf = conf.setMaster("local[*]") sc = SparkContext(conf=conf) ssc = StreamingContext(sc, 10) # every 10 seconds simple.SimpleSpark().myfunc(ssc) ssc.start(); ssc.awaitTermination(); |
simple.py – receive messages from a topic
|
1 |
(my-app_env) [root@quickstart my-app]# vi mypackage/simple.py |
|
1 2 3 4 5 6 7 8 9 10 11 |
from pyspark.streaming import StreamingContext from pyspark.streaming.kafka import KafkaUtils class SimpleSpark: def myfunc(self, ssc): # 2181 is the zookeeper port & 9092 is the kafka broker port kafkaStream = KafkaUtils.createStream(ssc, 'localhost:2181', 'spark-streaming', {'MyTestTopic':1}) lines = kafkaStream.map(lambda x : x[1]) lines.pprint() |
Build the .egg file Step 2: Create setup.py to build the .egg file, … Read more ›...
This extends 27: Docker Tutorial: Apache Kafka with Java API on Cloudera quickstart. Pre-requisite: Java 8 & Kafka installed with the “MyTestTopic” as per the previous tutorials. Install scala & sbt We will install Scala 2.10.7 from “https://www.scala-lang.org/download/2.10.7.html” … Read more ›...
HBase is a NoSQL database used in Hadoop world to store “Big Data”. This extends Understanding HBase (NoSQL) database basics in 7 steps.
Step 1: Create a Maven based Java project
|
1 2 3 |
mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-hbase |
Step 2: Import the project into eclipse & modify the pom.xml file
to add Hadoop &
…
Step 1: Create an HBase table named “sequence_numbers” with one column-family “i”. You can create this via HBase shell.
|
1 2 3 4 |
$ hbase shell hbase(main):001:0> create 'sequence_numbers', {NAME => 'i'} |
Step 2: You can use the HBase API in Java to insert a new record via “assigCurrentValue()“, and perform atomic operations in “ … Read more ›...
Performance testing is an important process of any software development. You can bring this up yourself to many open-ended questions like — Tell me about yourself? What are some of your recent accomplishments that you are proud of? Setting up a performance/load test script is always the first step in...
This extends the Apache HTTPClient Tutorial to invoke RESTFul service with a POST request.
The pom.xml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.mytutorial</groupId> <artifactId>httpRestClient</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <name>httpRestClient</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.2.1</version> </dependency> </dependencies> </project> |
The client using HttpClient & JAXBContext
The “JAXBContext” is from the Java 6 onwards. The sample code includes
1) Exception handling.
2) “BasicCredentialsProvider”
…