Q1. What is Spark SQL?
A1. Apache Spark SQL is a module for structured data processing in Spark. Spark SQL integrates relational processing (i.e. SQL) with Spark’s functional programming using Scala, Java, etc weave SQL queries with Dataframes/Datasets based transformations. It provides support for various data sources as shown below:
Q2.
…
This extends 09: 50+ SQL scenarios based interview Q&As on GROUP_CONCAT() & STRING_AGG().
In the previous post we looked at how to implode or concatenate multiple row column values into a single column. In this post let’s look how we can explode or split delimited column value to multiple rows.
…
Java developers need to learn Coding Scala Way. Scala way of coding is quite different & you need to unlearn Java way of coding
Question: Given a list of numbers, how will you group the numbers by how many times they occur? for example output for “List(3,
…
Q01. How can you clone a table on Databricks? A01. You can create a copy of an existing Delta Lake table on Databricks at a specific version using the clone command. Clones can be either deep or shallow. … Read more ›...
Given the below employee.csv file in /tmp folder:
|
1 2 3 4 5 6 7 8 9 |
emp_id,emp_name,emp_city,emp_salary 1, John, Sydney, 35000.00 2, Peter, Melbourne, 45000.00 3, Sam, Sydney,55000.00 4, Danny, Melbourne, 45000.00 7, Roger, Sydney, 35000.00 8, Lisa, Sydney, 25000.00 |
Q26: How will you read the above file & calculate 10% bonus and write it as parquet file employee.parquet? A26: Here is the code that reads the .csv file, … Read more ›...
Python’s general purpose built-in containers are dict ( E.g. {‘a’:1, ‘b’:2, ‘c’:3} ), list ( E.g. [1,2,3,4] ), set ( E.g. {1,2,3,4 } ), and tuple ( E.g. (1,2,3,4) ). Following are specialised container datatypes providing alternatives. … Read more ›...
Q66. What is the purpose of pattern matching in Scala? A66. It is for the “flow control“. Most imperative languages support “if-then-else”, case statements, and loops like while, for, foreach, etc. Scala does support some of these, but the most often used “ … Read more ›...
Q80. Why is schema design for HBase is different from relational database design? A80. HBase is a columnar NoSQL database. This means no two rows in a table need to have the same columns. In a columnar database table, each row contains cells containing
|
1 2 3 |
row-key (unique) + column-family + column qualifier + timestamp |
Values are stored against each...
Q1. What are the key differences between Apache Hive & Apache Impala? When would you use one over the other? A1. Apache Hive is a batch processing framework and used as an ETL/ELT tool, whilst Apache Impala is for interactive SQL queries by the data analysts & … Read more...
Q1. What is Apache Nifi? What is Apache NiFi not? A1. NiFi is data flow tool for moving data from point A (e.g. MySQL DB) to point B (e.g. AWS S3 bucket), with a GUI based drag and drop experience. Non-Functional features like security, Data lineage/provenance (i.e. … Read more...
LEAD(..) & LAG(..) are Window functions, which perform operations for each row of the partition or window. LEAD() and LAG() are used to accesses a value stored in a row below and a value stored in a row above respectively. An ORDER BY clause is required when working with LEAD...
We already saw in Java: Producer and Consumer Java Multi-threading code. This extends the tutorial “Simple Akka tutorial in Scala step by step” for setting up the environment with right plugins and libraries with Eclipse & Maven.
The Akka Actors sending &…
This extends 10: PySpark working with parquet file format Interview Q&As with tutorials. Given the below employee.csv file in /tmp folder:
|
1 2 3 4 5 6 7 8 9 |
emp_id,emp_name,emp_city,emp_salary 1, John, Sydney, 35000.00 2, Peter, Melbourne, 45000.00 3, Sam, Sydney,55000.00 4, Danny, Melbourne, 45000.00 7, Roger, Sydney, 35000.00 8, Lisa, Sydney, 25000.00 |
AVRO is a popular data serialization format used in big data processing systems such as Hadoop, Spark, and Kafka. … Read more ›...
I have a good news, I got job offer. I like to share with you from my bottom of my heart feeling thankful to you.