Quick Prep Big Data Interview Archives | Page 14 of 23 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 10 11 12 13 14 15 16 17 18 … 23 »

15: Python multithreading tutorial

In Python, multithreading is primarily used for I/O-bound tasks like downloading files or querying databases. Due to the Global Interpreter Lock (i.e. GIL), Python threads cannot run in true parallel for CPU-heavy tasks instead, they take turns, which is known as concurrency.


import threading
import time

def job(name, delay):
    print(f"Job {name} started...")
    time.sleep(delay)  # Simulates I/O
    print(f"Job {name} finished")


# Create and start threads
t1 = threading.Thread(target=job, args=("A", 2))
t2 = threading.Thread(target=job, args=("B", 1))
t1.start();
t2.start()

# Wait for completion
t1.join();
t2.join()

import threading

import time

def job(name, delay):

print(f"Job {name} started...")

time.sleep(delay) # Simulates I/O

print(f"Job {name} finished")

# Create and start threads

t1 = threading.Thread(target=job, args=("A", 2))

t2 = threading.Thread(target=job, args=("B", 1))

t1.start();

t2.start()

# Wait for completion

t1.join();

t2.join()

Outputs:


Job A started...Job B started...

Job B finished
Job A finished

Job A started...Job B started...

Job B finished

Job A finished

GIL GIL is a lock...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15: Q109 – Q113 Spark RDD partitioning and “mapPartitions” interview questions & answers

Q109. What is the difference between “map” and “mapPartitions” transformations in Spark? A109. The method map converts each element of the source RDD into a single element of the result RDD by applying a function. The method mapPartitions converts each partition of the source RDD into multiple elements of the...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15: Q88 – Q92 Scala Functors Interview Q&As

This extends Category Theory With Scala Interview Q&As. Q88. What do you understand by the terms functors, monads, and applicatives ? A88. Functors, Monads, and Applicatives are concepts or patterns that abstract complexities in computations, and are NOT traits or classes. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15+ Amazon EC2 instances interview Q&As

Q01. What is an Amazon EC2 Service? A01. EC2 stands for Elastic Cloud Computing that provides resizable (i.e. scalable) computing capacity in the cloud. You can use Amazon EC2 to launch as many virtual servers as you need. In EC2 you can configure security and networking as well as manage...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15+ AWS Services overview interview Q&As

Below questions are based on the above diagram, and extends AWS Q&As on VPC, Subnets, Availability Zones, VPN, Route tables, NACLs & Security Groups: Q1. What is the difference between CloudWatch & CloudTrail? A1. Both provide logging capabilities. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15+ SQL optimisation interview questions & answers with scenarios

1) Select fields instead of using SELECT *. The “*” is used as a short hand to query all available data from a table. If a table has many fields and rows, this takes database resources by querying a lot of unnecessary data. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

15B: Scala Functors Interview Q&As by example

Prerequisite: Scala values, proper types, first order types & higher kinded types. Q. Is it possible to further abstract the below code?


package com.functor

import scala.util.Try
import scala.util.Success

object FunctorExample extends App {

  def timesByTwoList(input: List[Int]): List[Int] = input.map(_ * 2)

  def timesByTwoOption(input: Option[Int]): Option[Int] = input.map(_ * 2)
  
  def timesByTwoTry(input: Try[Int]): Try[Int] = input.map(_ * 2)
  
  println(timesByTwoList(List(1,2,3)));
  println(timesByTwoOption(Some(4)));
  println(timesByTwoTry(Success(5)));

}

package com.functor

import scala.util.Try

import scala.util.Success

object FunctorExample extends App {

def timesByTwoList(input: List[Int]): List[Int] = input.map(_ * 2)

def timesByTwoOption(input: Option[Int]): Option[Int] = input.map(_ * 2)

def timesByTwoTry(input: Try[Int]): Try[Int] = input.map(_ * 2)

println(timesByTwoList(List(1,2,3)));

println(timesByTwoOption(Some(4)));

println(timesByTwoTry(Success(5)));

}

Output:


List(2, 4, 6)
Some(8)
Success(10)

List(2, 4, 6)

Some(8)

Success(10)

A. Why have different timesByTwoXXX functions for each container type like List, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: 50+ SQL scenarios based interview Q&As on joins to reconcile source & target tables

In SQL often we need to merge records from a source table to a target table. The source table could be a staging table where the source system sent new & updated records to be reconciled with the a target table. Let’s look at an example as to how to...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: Coding Scala Way – Scala concurrency styles

This extends Scala concurrency & closure vs function definition. This post looks at Scala concurrency styles – blocking & non-blocking. onComplete – non-blocking


import scala.concurrent.ExecutionContext.Implicits.global //thread pool execution context
import scala.concurrent.{ Await, Future }
import scala.actors.threadpool.AtomicInteger
import scala.concurrent.duration._ // 5 seconds
import scala.util.{ Failure, Success }

object SimpleFuture extends App {

  val counter = new AtomicInteger

  //can succeed or fail
  def futureInc() = Future {
    Thread.sleep(2000);
    println("Incrementing .....")
    counter.incrementAndGet()
  }

  val f1 = futureInc  
  val f2 = futureInc   

  f1.onComplete {
    case Success(val1) => {
      println("val1 = " + val1)
      f2.onComplete {
        case Success(val2) => {
          println("val1 = " + val2)
          println("Sum of generated numbers = " + (val1 + val2))
        }

        case Failure(e) => e.printStackTrace

      }
    }
    case Failure(e) => e.printStackTrace

  }

  println("I am not blocked .......");
  println("Other operations .......");

  Thread.sleep(5000) //

}

import scala.concurrent.ExecutionContext.Implicits.global //thread pool execution context

import scala.concurrent.{ Await, Future }

import scala.actors.threadpool.AtomicInteger

import scala.concurrent.duration._ // 5 seconds

import scala.util.{ Failure, Success }

object SimpleFuture extends App {

val counter = new AtomicInteger

//can succeed or fail

def futureInc() = Future {

Thread.sleep(2000);

println("Incrementing .....")

counter.incrementAndGet()

}

val f1 = futureInc

val f2 = futureInc

f1.onComplete {

case Success(val1) => {

println("val1 = " + val1)

f2.onComplete {

case Success(val2) => {

println("val1 = " + val2)

println("Sum of generated numbers = " + (val1 + val2))

}

case Failure(e) => e.printStackTrace

}

case Failure(e) => e.printStackTrace

}

println("I am not blocked .......");

println("Other operations .......");

Thread.sleep(5000) //

}

Output:


I am not blocked .......
Other operations .......
Incrementing .....
Incrementing .....
val1 = 2
val1 = 1
Sum of generated numbers = 3

I am not blocked .......

Other operations .......

Incrementing .....

val1 = 2

val1 = 1

Sum of generated numbers = 3

onSuccess – non-blocking The above “onComplete” call is highly nested. “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: PySpark to read JSON files Interview Q&As with tutorials

Q51: How will you read the below JSON file employee.json into a Pyspark Dataframe?


{
    "company_name": "XYZ Org",
    "company_id": 123,
    "executive_name": "Jonathan",
    "employees": [
        {
            "emp_id": 1,
            "emp_name": "John",
            "emp_city": "Sydney",
            "emp_salary": 35000
        },
        {
            "emp_id": 2,
            "emp_name": "Peter",
            "emp_city": "Melbourne",
            "emp_salary": 45000
        },
        {
            "emp_id": 3,
            "emp_name": "Sam",
            "emp_city": "Sydney",
            "emp_salary": 55000
        },
        {
            "emp_id": 4,
            "emp_name": "Danny",
            "emp_city": "Melbourne",
            "emp_salary": 45000
        },
        {
            "emp_id": 7,
            "emp_city": "Sydney",
            "emp_salary": 35000
        },
        {
            "emp_id": 8,
            "emp_name": "Lisa",
            "emp_city": "Sydney",
            "emp_salary": 25000
        }
    ]
}

{

"company_name": "XYZ Org",

"company_id": 123,

"executive_name": "Jonathan",

"employees": [

{

"emp_id": 1,

"emp_name": "John",

"emp_city": "Sydney",

"emp_salary": 35000

{

"emp_id": 2,

"emp_name": "Peter",

"emp_city": "Melbourne",

"emp_salary": 45000

{

"emp_id": 3,

"emp_name": "Sam",

"emp_city": "Sydney",

"emp_salary": 55000

{

"emp_id": 4,

"emp_name": "Danny",

"emp_city": "Melbourne",

"emp_salary": 45000

{

"emp_id": 7,

"emp_city": "Sydney",

"emp_salary": 35000

{

"emp_id": 8,

"emp_name": "Lisa",

"emp_city": "Sydney",

"emp_salary": 25000

}

]

}

A51: Here is the PySpark code with schema to read the above JSON file. read_json_spark.py: Read with a static schema


from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from pyspark.sql.types import *
from pyspark import sql

def read_file(spark):

    # defne the schema
    schema = StructType([StructField('company_name', StringType(), False),
                         StructField('company_id', IntegerType(), False),
                         StructField('executive_name', StringType(), False),
                         StructField('employees', ArrayType(StructType([ \
                                 StructField('emp_id', StringType(), False), \
                                 StructField('emp_name', StringType(), False),\
                                 StructField('emp_city', StringType(), True),\
                                 StructField('emp_salary', DoubleType(), True)])))])

    # read a json file
    emp_df = spark.read.format("json")\
                             .option("multiline", True)\
                             .schema(schema)\
                             .load("/tmp/employee.json")  

    
    emp_df.printSchema()

    emp_df.show(20, False)

 
if __name__ == '__main__':
    spark = SparkSession.builder.appName("Spark read a JSON file")\
                                .master("local[*]")\
                                .getOrCreate()
    read_file(spark)

from pyspark.sql import SparkSession

import pyspark.sql.functions as F

from pyspark.sql.types import *

from pyspark import sql

def read_file(spark):

# defne the schema

schema = StructType([StructField('company_name', StringType(), False),

StructField('company_id', IntegerType(), False),

StructField('executive_name', StringType(), False),

StructField('employees', ArrayType(StructType([ \

StructField('emp_id', StringType(), False), \

StructField('emp_name', StringType(), False),\

StructField('emp_city', StringType(), True),\

StructField('emp_salary', DoubleType(), True)])))])

# read a json file

emp_df = spark.read.format("json")\

.option("multiline", True)\

.schema(schema)\

.load("/tmp/employee.json")

emp_df.printSchema()

emp_df.show(20, False)

if __name__ == '__main__':

spark = SparkSession.builder.appName("Spark read a JSON file")\

.master("local[*]")\

.getOrCreate()

read_file(spark)

Run As:


$ spark-submit read_json_spark.py

$ spark-submit read_json_spark.py

Outputs


......
 |-- company_name: string (nullable = true)
 |-- company_id: integer (nullable = true)
 |-- executive_name: string (nullable = true)
 |-- employees: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- emp_id: string (nullable = true)
 |    |    |-- emp_name: string (nullable = true)
 |    |    |-- emp_city: string (nullable = true)
 |    |    |-- emp_salary: double (nullable = true)
......
+------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|company_name|company_id|executive_name|employees                                                                                                                                                                      |
+------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|XYZ Org     |123       |Jonathan      |[{1, John, Sydney, 35000.0}, {2, Peter, Melbourne, 45000.0}, {3, Sam, Sydney, 55000.0}, {4, Danny, Melbourne, 45000.0}, {7, NULL, Sydney, 35000.0}, {8, Lisa, Sydney, 25000.0}]|
+------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
......

......

|-- company_name: string (nullable = true)

|-- company_id: integer (nullable = true)

|-- executive_name: string (nullable = true)

|-- employees: array (nullable = true)

| |-- element: struct (containsNull = true)

| | |-- emp_id: string (nullable = true)

| | |-- emp_name: string (nullable = true)

| | |-- emp_city: string (nullable = true)

| | |-- emp_salary: double (nullable = true)

......

+------------+----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

|XYZ Org |123 |Jonathan |[{1, John, Sydney, 35000.0}, {2, Peter, Melbourne, 45000.0}, {3, Sam, Sydney, 55000.0}, {4, Danny, Melbourne, 45000.0}, {7, NULL, Sydney, 35000.0}, {8, Lisa, Sydney, 25000.0}]|

......

Explode the array Q52: How will you explode the...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: Python multithreading producer/consumer & semaphores tutorial

Producer/Consumer in Python The Consumer and Producer threads run indefinitely while checking the status of the queue. The Producer thread is responsible for putting items into the queue if it is not full while the Consumer thread consumes items if there are any. You need to know the following three...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: Q114 – Q115 CAP theorem interview Q&As

Q114. What does CAP stand for in CAP theorem? A114. In a distributed system having two or more nodes, and maintaining two or more copies of your data for fault tolerance, the CAP theorem can be depicted & explained as below: Consistency – … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

16: Q93 – Q98 Scala monads interview Q&As

This extends Scala Functors Interview Q&As. You need to remember the following.


Applying function of signature:
-----------------------------
"T => R"                           to return  "C[T] => C[R]"    is a  Functor
"T => C[R]"                        to return  "C[T] => C[R]"    is a  Monad
"C[T => R]"                        to return  "C[T] => C[R]"    is an Applicative

Applying function of signature:

-----------------------------

"T => R" to return "C[T] => C[R]" is a Functor

"T => C[R]" to return "C[T] => C[R]" is a Monad

"C[T => R]" to return "C[T] => C[R]" is an Applicative

Where T & R are raw types like String, Int, Employee, etc, and C is a container or wrapper type like List, Option, Future, List, etc. So, T =R could be a type abstraction for Int =Int,...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 10 11 12 13 14 15 16 17 18 … 23 »

Blog Archives

15: Python multithreading tutorial

15: Q109 – Q113 Spark RDD partitioning and “mapPartitions” interview questions & answers

15: Q88 – Q92 Scala Functors Interview Q&As

15+ Amazon EC2 instances interview Q&As

15+ AWS Services overview interview Q&As

15+ SQL optimisation interview questions & answers with scenarios

15B: Scala Functors Interview Q&As by example

16: 50+ SQL scenarios based interview Q&As on joins to reconcile source & target tables

16: Coding Scala Way – Scala concurrency styles

16: PySpark to read JSON files Interview Q&As with tutorials

16: Python multithreading producer/consumer & semaphores tutorial

16: Q114 – Q115 CAP theorem interview Q&As

16: Q93 – Q98 Scala monads interview Q&As

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer