Q98. What is Hive? A98. Hive is used for accessing and analyzing data in Hadoop using SQL syntax. It is known as the HiveQL. Q99. What is the difference between Hive internal tables & external tables? A99. … Read more ›...
800+ Java & Big Data Interview Q&As
130+ job offers to choose from as a self-taught Java & Big Data contractor for 22+ years. It pays to prepare.
Blog Archives
14: 50+ SQL scenarios based interview Q&As on order of SQL executions
This post reiterates the importance of understanding the order of execution of the SQL statements. Also, always think of the data granularity (aka grain). This post also demonstrates that the final query may look complex, but if you break it down it is much easier to comprehend.
…
14: PySpark with Apache Kafka Interview Q&As with tutorials – part 2
This extends 13: PySpark with Apache Kafka Interview Q&As with tutorials – part 1 Q42: How would you go about displaying the array of JSON data in a tabular format? A42 In this post we will expand & … Read more ›...
14: Q105 – Q108 Spark “map” vs “flatMap” interview questions & answers
Q105. What is the difference between “map” and “flatMap” operations in Spark?
A105. The map and flatMap are transformation operations in Spark.
map transformation is applied to each element of RDD and it returns the result as a new RDD.
…
15: 50+ SQL scenarios based interview Q&As on pivoting & aggregate functions
In this post let’s look at an SQL scenario, and three different ways to solve this scenario. There could be more ways to solve an SQL scenario. Problem Statement Given the below “orders” table write a query to find the order status.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
line_item_id order_id line_item_status ---------------------------------------------- 1 ORD123 DELIVERED 2 ORD123 DELIVERED 3 ORD123 DELIVERED 1 ORD124 DELIVERED 2 ORD124 SHIPPED 1 ORD125 PENDING 2 ORD125 SHIPPED 3 ORD125 DELIVERED 1 ORD126 SHIPPED 2 ORD126 SHIPPED 1 ORD127 PENDING 2 ORD127 PENDING |
1) If all line items are ‘DELIVERED’...
15: PySpark with Apache Kafka Interview Q&As with tutorials – part 3
This extends PySpark with Apache Kafka Interview Q&As with tutorials – part 1 and PySpark with Apache Kafka Interview Q&As with tutorials – part 2 Q46: How will you go about writing the streamed structured to a parquet file? A46: In the last tutorial we wrote a console sink, …...
15: Q109 – Q113 Spark RDD partitioning and “mapPartitions” interview questions & answers
Q109. What is the difference between “map” and “mapPartitions” transformations in Spark? A109. The method map converts each element of the source RDD into a single element of the result RDD by applying a function. The method mapPartitions converts each partition of the source RDD into multiple elements of the...
15+ Amazon EC2 instances interview Q&As
Q01. What is an Amazon EC2 Service? A01. EC2 stands for Elastic Cloud Computing that provides resizable (i.e. scalable) computing capacity in the cloud. You can use Amazon EC2 to launch as many virtual servers as you need. In EC2 you can configure security and networking as well as manage...
15+ AWS Services overview interview Q&As
Below questions are based on the above diagram, and extends AWS Q&As on VPC, Subnets, Availability Zones, VPN, Route tables, NACLs & Security Groups: Q1. What is the difference between CloudWatch & CloudTrail? A1. Both provide logging capabilities. … Read more ›...
15+ SQL optimisation interview questions & answers with scenarios
1) Select fields instead of using SELECT *. The “*” is used as a short hand to query all available data from a table. If a table has many fields and rows, this takes database resources by querying a lot of unnecessary data. … Read more ›...
16: 50+ SQL scenarios based interview Q&As on joins to reconcile source & target tables
In SQL often we need to merge records from a source table to a target table. The source table could be a staging table where the source system sent new & updated records to be reconciled with the a target table. Let’s look at an example as to how to...
16: PySpark to read JSON files Interview Q&As with tutorials
Q51: How will you read the below JSON file employee.json into a Pyspark Dataframe?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
{ "company_name": "XYZ Org", "company_id": 123, "executive_name": "Jonathan", "employees": [ { "emp_id": 1, "emp_name": "John", "emp_city": "Sydney", "emp_salary": 35000 }, { "emp_id": 2, "emp_name": "Peter", "emp_city": "Melbourne", "emp_salary": 45000 }, { "emp_id": 3, "emp_name": "Sam", "emp_city": "Sydney", "emp_salary": 55000 }, { "emp_id": 4, "emp_name": "Danny", "emp_city": "Melbourne", "emp_salary": 45000 }, { "emp_id": 7, "emp_city": "Sydney", "emp_salary": 35000 }, { "emp_id": 8, "emp_name": "Lisa", "emp_city": "Sydney", "emp_salary": 25000 } ] } |
A51: Here is the PySpark code with schema to read the above JSON file. read_json_spark.py: Read with a static schema
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
from pyspark.sql import SparkSession import pyspark.sql.functions as F from pyspark.sql.types import * from pyspark import sql def read_file(spark): # defne the schema schema = StructType([StructField('company_name', StringType(), False), StructField('company_id', IntegerType(), False), StructField('executive_name', StringType(), False), StructField('employees', ArrayType(StructType([ \ StructField('emp_id', StringType(), False), \ StructField('emp_name', StringType(), False),\ StructField('emp_city', StringType(), True),\ StructField('emp_salary', DoubleType(), True)])))]) # read a json file emp_df = spark.read.format("json")\ .option("multiline", True)\ .schema(schema)\ .load("/tmp/employee.json") emp_df.printSchema() emp_df.show(20, False) if __name__ == '__main__': spark = SparkSession.builder.appName("Spark read a JSON file")\ .master("local[*]")\ .getOrCreate() read_file(spark) |
Run As:
|
1 2 3 |
$ spark-submit read_json_spark.py |
Outputs
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
...... |-- company_name: string (nullable = true) |-- company_id: integer (nullable = true) |-- executive_name: string (nullable = true) |-- employees: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- emp_id: string (nullable = true) | | |-- emp_name: string (nullable = true) | | |-- emp_city: string (nullable = true) | | |-- emp_salary: double (nullable = true) ...... +------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ |company_name|company_id|executive_name|employees | +------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ |XYZ Org |123 |Jonathan |[{1, John, Sydney, 35000.0}, {2, Peter, Melbourne, 45000.0}, {3, Sam, Sydney, 55000.0}, {4, Danny, Melbourne, 45000.0}, {7, NULL, Sydney, 35000.0}, {8, Lisa, Sydney, 25000.0}]| +------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ...... |
Explode the array Q52: How will you explode the...
16: Q114 – Q115 CAP theorem interview Q&As
Q114. What does CAP stand for in CAP theorem? A114. In a distributed system having two or more nodes, and maintaining two or more copies of your data for fault tolerance, the CAP theorem can be depicted & explained as below: Consistency – … Read more ›...
Top