TUT - PySpark on Databricks Archives | 800+ Java & Big Data Interview Q&As

Blog Archives

1 2 3 »

01: Databricks getting started – PySpark, Shell, and SQL

Step 1: Signup to Databricks community edition – https://databricks.com/try-databricks. Fill in the details and you can leave your mobile number blank. Select “COMMUNITY EDITION” ==“GET STARTED“.

If you have a Cloud account then you can use it.

…

02: Databricks – Spark schemas, casting & PySpark API

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL.

Q: What is a Dataframe?
A: A DataFrame is a data abstraction or a domain-specific language (DSL) for working with structured and semi-structured data,

…

03: Databricks – Spark SCD Type 1

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL.

What is SCD Type 1

SCD stands for Slowly Changing Dimension, and it was explained in 10 Data warehouse interview Q&As.

…

04: Databricks – Spark SCD Type 2

Prerequisite: Extends 03: Databricks – Spark SCD Type 1. What is SCD Type 2 SCD stands for Slowly Changing Dimension, and it was explained in 10 Data warehouse interview Q&As. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04a: Databricks – Spark SCD Type 1 with Merge

Prerequisite: Extends 03: Databricks – Spark SCD Type 1. What is SCD Type 1 SCD stands for Slowly Changing Dimension, and it was explained in 10 Data warehouse interview Q&As. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04b: Databricks – Spark SCD Type 2 with Merge

Databricks SCD Type 2 with null mergeKey

Prerequisite: … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Databricks – Spark UDFs

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL.

What is a UDF?

User-Defined Functions (aka UDF) is a feature of Spark SQL to define new Column-based functions that extend the vocabulary of Spark SQL’s DSL for transforming Datasets.

…

06: Databricks – Spark Window functions

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL. What is a window function? Q. What are the different types of functions in Spark SQL? A. There are 4 types of functions: 1) Built-in functions: from org.apache.spark.sql. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Databricks – groupBy, collect_list & explode

Prerequisite: Extends Databricks – Spark Window functions. Step 1: Create a new Python notebook, and attach it to a cluster. Step 2: Let’s create some data using pyspark.

emp_list = (
  (1, 'John', 'Java'),
  (2, 'Peter', 'Scala'),
  (2, 'Peter', 'Python'),
  (2, 'Peter', 'Spark'),
  (1, 'John', 'JEE'),
  (6, 'Elliot', 'Unix',)
)
 
data_schema = [StructField('emp_id', IntegerType(), False), StructField('emp_name', StringType(), False), StructField('emp_expertise', StringType(), False)]
final_struc = StructType(fields=data_schema)
 
df = spark.sparkContext.parallelize(emp_list);
 
emp_df = spark.createDataFrame(df, final_struc)
emp_df.show()

emp_list = (

(1, 'John', 'Java'),

(2, 'Peter', 'Scala'),

(2, 'Peter', 'Python'),

(2, 'Peter', 'Spark'),

(1, 'John', 'JEE'),

(6, 'Elliot', 'Unix',)

)

data_schema = [StructField('emp_id', IntegerType(), False), StructField('emp_name', StringType(), False), StructField('emp_expertise', StringType(), False)]

final_struc = StructType(fields=data_schema)

df = spark.sparkContext.parallelize(emp_list);

emp_df = spark.createDataFrame(df, final_struc)

emp_df.show()

Output:

+------+--------+-------------+
|emp_id|emp_name|emp_expertise|
+------+--------+-------------+
|     1|    John|         Java|
|     2|   Peter|        Scala|
|     2|   Peter|       Python|
|     2|   Peter|        Spark|
|     1|    John|          JEE|
|     6|  Elliot|         Unix|
+------+--------+-------------+

+------+--------+-------------+

|emp_id|emp_name|emp_expertise|

+------+--------+-------------+

| 1| John| Java|

| 2| Peter| Scala|

| 2| Peter| Python|

| 2| Peter| Spark|

| 1| John| JEE|

| 6| Elliot| Unix|

+------+--------+-------------+

agg( ) &… Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Databricks – Spark problem 1

Prerequisite: Extends Databricks – Spark Window functions.

Problem: Convert the below table

+------+--------+---------+-------------+
|emp_id|emp_name| emp_city|emp_expertise|
+------+--------+---------+-------------+
|     1|    John|   Sydney|         Java|
|     2|   Peter|Melbourne|        Scala|
|     3|     Sam| Brisbane|       Python|
|     4|   David|Melbourne|       Python|
|     5|  Elliot|   Sydney|         Java|
+------+--------+---------+-------------+

+------+--------+---------+-------------+

+------+--------+---------+-------------+

| 3| Sam| Brisbane| Python|

+------+--------+---------+-------------+

+-------------+----------------------------------------------------------+
|Column       |Content                                                   |
+-------------+----------------------------------------------------------+
|emp_id       |[[1, 1], [3, 1], [5, 1], [4, 1], [2, 1]]                  |
|emp_name     |[[Elliot, 1], [John, 1], [Sam, 1], [Peter, 1], [David, 1]]|
|emp_city     |[[Sydney, 2], [Melbourne, 2], [Brisbane, 1]]              |
|emp_expertise|[[Python, 2], [Java, 2], [Scala, 1]]                      |
+-------------+----------------------------------------------------------+

+-------------+----------------------------------------------------------+

|Column |Content |

+-------------+----------------------------------------------------------+

|emp_id |[[1, 1], [3, 1], [5, 1], [4, 1], [2, 1]] |

|emp_name |[[Elliot, 1], [John, 1], [Sam, 1], [Peter, 1], [David, 1]]|

|emp_city |[[Sydney, 2], [Melbourne, 2], [Brisbane, 1]] |

|emp_expertise|[[Python, 2], [Java, 2], [Scala, 1]] |

+-------------+----------------------------------------------------------+

Where each column is counted for its occurrence. For example, the emp_city column values “Melbourne”

…

09: Databricks – Spark Problem 2

Prerequisite: Extends Databricks – Spark problem 1. Problem: Convert the below table

+--------+------------+---+
|emp_name|     product|qty|
+--------+------------+---+
|    John|    Notebook|  7|
|   Peter|Mobile phone| 12|
|     Sam|      Camera|  3|
|   David|Mobile phone|  9|
|  Elliot|    Notebook|  3|
|  Elliot|      Camera|  1|
|    John|Mobile phone| 13|
|   Peter|Mobile phone|  2|
+--------+------------+---+

+--------+------------+---+

|emp_name| product|qty|

+--------+------------+---+

| John| Notebook| 7|

| Peter|Mobile phone| 12|

| Sam| Camera| 3|

| David|Mobile phone| 9|

| Elliot| Notebook| 3|

| Elliot| Camera| 1|

| John|Mobile phone| 13|

| Peter|Mobile phone| 2|

+--------+------------+---+

+--------+------+------------+--------+
|emp_name|Camera|Mobile phone|Notebook|
+--------+------+------------+--------+
|  Elliot|     1|        null|       3|
|    John|  null|          13|       7|
|     Sam|     3|        null|    null|
|   Peter|  null|          14|    null|
|   David|  null|           9|    null|
+--------+------+------------+--------+

+--------+------+------------+--------+

+--------+------+------------+--------+

| Elliot| 1| null| 3|

| John| null| 13| 7|

| Sam| 3| null| null|

+--------+------+------------+--------+

Step 1: Create a new Python notebook, and attach it to a new cluster. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

10: Databricks – Spark ML – Linear Regression

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL. You can try these tutorials in Scala using Databricks Notebook. There are Scala tutorials covered in Spark using Scala on Zeppelin Notebook. Problem statement: Predict the land prices by land area in square feet based on a given set of known...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11: Databricks – Spark ML – Multivariate Linear Regression

Prerequisite: Extends Databricks – Spark ML – Linear Regression. Problem statement: Predict the house prices by land area in square feet, no of bedrooms, and how old the house is?, which is the age. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

1 2 3 »

Blog Archives

01: Databricks getting started – PySpark, Shell, and SQL

02: Databricks – Spark schemas, casting & PySpark API

03: Databricks – Spark SCD Type 1

04: Databricks – Spark SCD Type 2

04a: Databricks – Spark SCD Type 1 with Merge

04b: Databricks – Spark SCD Type 2 with Merge

05: Databricks – Spark UDFs

06: Databricks – Spark Window functions

07: Databricks – groupBy, collect_list & explode

08: Databricks – Spark problem 1

09: Databricks – Spark Problem 2

10: Databricks – Spark ML – Linear Regression

11: Databricks – Spark ML – Multivariate Linear Regression

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer