Quick Prep Big Data Interview Archives | Page 16 of 23 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 12 13 14 15 16 17 18 19 20 … 23 »

19: 50+ SQL scenarios based interview Q&As on effective_date

Here is another SQL scenario based interview question to determine a candidate’s ability to translate a business requirement into an SQL query. These are scenarios where you want to find a value at a certain point in time. Lean more about SCD Type 2 at 13 Data Warehouse interview Q&As...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

19: PySpark on handling duplicates Interview Q&As with tutorials

Given the below data, where the record with the emp_name “Elliot” is repeated.


+--------+------------+---+
|emp_name|     product|qty|
+--------+------------+---+
|    John|    Notebook|  3|
|   Peter|Mobile phone| 12|
|     Sam|      Camera|  3|
|   David|Mobile phone|  9|
|  Elliot|    Notebook|  3|
|  Elliot|    Notebook|  3|
|    John|Mobile phone| 13|
|   Peter|Mobile phone|  2|
+--------+------------+---+

+--------+------------+---+

|emp_name| product|qty|

+--------+------------+---+

| John| Notebook| 3|

| Peter|Mobile phone| 12|

| Sam| Camera| 3|

| David|Mobile phone| 9|

| Elliot| Notebook| 3|

| John|Mobile phone| 13|

| Peter|Mobile phone| 2|

+--------+------------+---+

distinct(..) Q: How will you eliminate duplicate rows? A: Use the distinct(..) function. problem.py:



from pyspark.sql import SparkSession
from pyspark.sql.types import *
from pyspark.sql import functions as F
from pyspark.sql import DataFrame
from functools import reduce

def solve(spark):

    sales = (
        ('John', 'Notebook', 3),
        ('Peter', 'Mobile phone', 12),
        ('Sam', 'Camera', 3),
        ('David', 'Mobile phone', 9),
        ('Elliot', 'Notebook', 3),
        ('Elliot', 'Notebook', 3),
        ('John', 'Mobile phone', 13),
        ('Peter', 'Mobile phone', 2)
     )
 
    data_schema = [StructField('emp_name', StringType(), False), StructField('product',StringType(),False), StructField('qty', IntegerType(), False)]
    final_struc = StructType(fields=data_schema)
 
    rdd = spark.sparkContext.parallelize(sales);
 
    df_sales = spark.createDataFrame(rdd, final_struc)
    df_sales.distinct()\
            .show()
    
if __name__ == '__main__':
    spark = SparkSession.builder.appName("Spark problem solving")\
                                .master("local[*]")\
                                .getOrCreate()
    solve(spark)

from pyspark.sql import SparkSession

from pyspark.sql.types import *

from pyspark.sql import functions as F

from pyspark.sql import DataFrame

from functools import reduce

def solve(spark):

sales = (

('John', 'Notebook', 3),

('Peter', 'Mobile phone', 12),

('Sam', 'Camera', 3),

('David', 'Mobile phone', 9),

('Elliot', 'Notebook', 3),

('John', 'Mobile phone', 13),

('Peter', 'Mobile phone', 2)

)

data_schema = [StructField('emp_name', StringType(), False), StructField('product',StringType(),False), StructField('qty', IntegerType(), False)]

final_struc = StructType(fields=data_schema)

rdd = spark.sparkContext.parallelize(sales);

df_sales = spark.createDataFrame(rdd, final_struc)

df_sales.distinct()\

.show()

if __name__ == '__main__':

spark = SparkSession.builder.appName("Spark problem solving")\

.master("local[*]")\

.getOrCreate()

solve(spark)

Run as:


$ spark-submit problem.py

$ spark-submit problem.py

Outputs:


+--------+------------+---+
|emp_name|     product|qty|
+--------+------------+---+
|    John|    Notebook|  3|
|   Peter|Mobile phone| 12|
|     Sam|      Camera|  3|
|   David|Mobile phone|  9|
|  Elliot|    Notebook|  3|
|    John|Mobile phone| 13|
|   Peter|Mobile phone|  2|
+--------+------------+---+

+--------+------------+---+

|emp_name| product|qty|

+--------+------------+---+

| John| Notebook| 3|

| Peter|Mobile phone| 12|

| Sam| Camera| 3|

| David|Mobile phone| 9|

| Elliot| Notebook| 3|

| John|Mobile phone| 13|

| Peter|Mobile phone| 2|

+--------+------------+---+

dropDuplicates(..) Q: What if you want to remove records based on duplicate values in selected columns?...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

19: Python operator overloading tutorial

Q. What do you understand by Python being a dynamically, and strongly typed language? A. In the example below, it is important to understand that the type is not changing. Two objects are created – an int object first, and then a str object. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

19: Q109 – Q113 Scala ADT (Algebraic Data Types) Interview Q&As

Q109 How do you model data in functional programming (i.e. FP)? A109 Using ADT (i.e. Algebraic Data Types), you can model data with logical ORs and logical ANDs. “OR” is also known as the “Sum” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

19: Q125 – Q130 Apache Zookeeper interview Q&As

Q125. What is Apache Zookeeper & where can you use it? A125. Distributed applications like HBase, Solr, Kafka, etc run services on multiple nodes (i.e. computers) for scalability (i.e more nodes can be added) & redundancy (i.e. several nodes provide the same service & if one node is down another...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

20: 50+ SQL scenarios based interview Q&As on first visit vs repeated visits

Let’s look at a very common problem statement of finding out the repeat customers, repeat visits, etc. The key to this problem is first determining the first visit or first purchase and then comparing it with the remaining records. Problem Statement 1 Given the below table, how will you determine...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

20: PySpark Dataframe Vs Pandas Dataframe Interview Q&As with tutorials

This extends NumPy Vs Pandas interview questions & answers – part 1 Q: What is the difference between PySpark Dataframe & a Pandas Dataframe? A: PySpark is a library where the operations are quicker than Pandas Dataframe library because of its parallel execution over multiple CPU cores & … Read...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

20: Python function overloading tutorial

Overloading in the context of programming, refers to the ability of a function or an operator to behave in different ways depending on the parameters that are passed to the function, or the operands that the operator acts on. Overloading a method is great for reusability because instead of writing...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

20: Q114 – Q120 Scala fold vs reduce Q&As

fold, map, and flatMap are key functional programming functions. These functions are used in three contexts: collections (E.g. List, Sequence, Array, Map, etc), Options, and Futures. Let’s look at the function “fold” here. Q114 What does fold & … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

20: Q131 – Q137 FAQ Apache Solr with HDFS interview questions & answers

Covers Solr interview Q&As in general, and how Solr can be used with Big Data in HDFS to index & query the data. You can get sub-seconds query performance when query the data on HDFS that are indexed using Solr. Q131. How does Solr accomplish fast search performance? … Read...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

21: 50+ SQL scenarios based interview Q&As on distinct count of all columns

We all know how to get a distinct count of a column. It can be done with COUNT(DISTINCT column_name). For example:


SELECT COUNT(DISTINCT user_id)
FROM web_store

SELECT COUNT(DISTINCT user_id)

FROM web_store

Outputs:


COUNT(DISTINCT user_id)
-------------------------
6

COUNT(DISTINCT user_id)

-------------------------

In this post let’s look at various approaches to count unique values. Using DISTINCT, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

21: PySpark map vs flatMap Interview Q&As with tutorials

Q01. What is the difference between map and flatMap operations in Spark?
A01. The map and flatMap are transformation operations in Spark.

map transformation is applied to each element of RDD or a DataFrame and it returns the result as a new RDD or a DataFrame.

…

21: Python function overriding tutorial

Q. What is method overriding? A. Overriding is the ability of a class to change the implementation of a method provided by one of its ancestors (i.e. parent). Overriding is one of OOP concepts, which makes inheritance exploit its power. Through method overriding a class may leverage its parent class’s...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 12 13 14 15 16 17 18 19 20 … 23 »

Blog Archives

19: 50+ SQL scenarios based interview Q&As on effective_date

19: PySpark on handling duplicates Interview Q&As with tutorials

19: Python operator overloading tutorial

19: Q109 – Q113 Scala ADT (Algebraic Data Types) Interview Q&As

19: Q125 – Q130 Apache Zookeeper interview Q&As

20: 50+ SQL scenarios based interview Q&As on first visit vs repeated visits

20: PySpark Dataframe Vs Pandas Dataframe Interview Q&As with tutorials

20: Python function overloading tutorial

20: Q114 – Q120 Scala fold vs reduce Q&As

20: Q131 – Q137 FAQ Apache Solr with HDFS interview questions & answers

21: 50+ SQL scenarios based interview Q&As on distinct count of all columns

21: PySpark map vs flatMap Interview Q&As with tutorials

21: Python function overriding tutorial

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer