Prerequisite: Extends Spark ML – StringIndexer & OneHotEncoder – LinearRegression. Q. What is a Classification type prediction? How does it differ from Linear Regression? A. Classification type predictions are 1) Is Email spam or not? … Read more ›...
CSV is a row based storage, and Parquet is columnar in nature, and it is designed from the ground up for efficient storage, compression and encoding, which gives better performance. Run the cloudera/quickstart This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. … Read more ›...
Create LineItems Hive table Step 1: Create a file “line-item1.txt” on HDFS under “/user/cloudera/learn-hdfs/lineitems” as
|
1 2 3 4 |
123,Pen 456,Pencil |
Step 2: You create a Hive table “lineitems” in the database “learnhadoop”.
|
1 2 3 4 5 6 7 8 9 |
CREATE EXTERNAL TABLE IF NOT EXISTS learnhadoop.lineitems ( order_id int ,product_name string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 'hdfs:////user/cloudera/learn-hdfs/lineitems'; |
Step 3: The “orders” table can be created in a similar manner as shown above. … Read more ›...
Prerequisite: Extends 15: Databricks – Spark ML – Classification with Logistic Regression & Databricks – Spark ML – StringIndexer & OneHotEncoder. Problem statement: Predict the likelihood of leaning towards a political party based on age & … Read more ›...
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.
|
1 2 3 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart |
Get parquet-tools Step 2: Install wget. The “uname -a” gets you the info of the kernel. … Read more ›...
Prerequisite: Extends Databricks – Spark ML Multiclass Logistic Regression & Pipeline. Q: What is K-Folds Cross Validation in ML? How does it differ from the train-split validation? A: Once we are done with training our model, … Read more ›...
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.
|
1 2 3 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart |
Connect to MySQL DB Step 2: Hive & Impala store the metadata in a database like MySQL. … Read more ›...
Prerequisite: Extends Databricks – Spark ML Multiclass Logistic Regression & Pipeline. Q: What is a Decision Tree Classifier? A: A Decision Tree a Supervised Machine Learning where the data is continuously split into Root Node, … Read more ›...
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker. Step 1: Run the container on a command line.
|
1 2 3 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart |
Connect to MySQL DB Step 2: Hive & Impala store the metadata in a database like MySQL. … Read more ›...
Prerequisite: Extends Databricks – Spark ML Decision Tree Classifier. Q. What is the difference between a decision tree classifier and a random forrest classifier? A. As is implied by the names “Tree” … Read more ›...
This extends Docker Tutorial: BigData on Cloudera quickstart via Docker.
Step 1: Run the container on a command line.
|
1 2 3 |
~/projects/docker-hadoop]$ docker run --hostname=quickstart.cloudera \ --privileged=true -t -i -v /Users/arulkumarankumaraswamipillai/projects/docker-hadoop:/src \ --publish-all=true -p 8888:8888 -p 80:80 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart |
Create a .csv in HDFS
Step 2: Create a csv file “employee.csv” in the local file system.
…
This extends the tutorial 1. Apache Pig Getting started.
Input Data
scores.xml in folder:/Users/arulk/projects representing marks of 4 students in 3 subjects:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
<scores> <subject> <name>Science</name> <marks> <mark>80</mark> <mark>75</mark> <mark>89</mark> <mark>90</mark> </marks> </subject> <subject> <name>Maths</name> <marks> <mark>90</mark> <mark>87</mark> <mark>78</mark> <mark>92</mark> </marks> </subject> <subject> <name>English</name> <marks> <mark>78</mark> <mark>88</mark> <mark>65</mark> <mark>99</mark> </marks> </subject> </scores> |
Step 1: Start pig in local file system mode.
…
This extends the Part 1 tutorial 1. Hadoop MapReduce Basic Tutorial. The key difference of this tutorial is using a “TextInputFormat” instead of “KeyValueTextInputFormat“.
TextInputFormat reads
The key as line offset number starting from 0 and the values as “Science,
…
I have a good news, I got job offer. I like to share with you from my bottom of my heart feeling thankful to you.