Big Data - 09: Spark SQL Archives

Blog Archives

06: Learn how to access Hive from Spark via SparkSQL & Dataframes by example

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. This example extends Learn Hive to write…

Members Only Content

…

10 Spark SQL Interview Q&As

Q1. What is Spark SQL?
A1. Apache Spark SQL is a module for structured data processing in Spark. Spark SQL integrates relational processing (i.e. SQL) with Spark’s functional programming using Scala, Java, etc weave SQL queries with Dataframes/Datasets based transformations. It provides support for various data sources as shown below:

Q2.…

17: Spark interview Q&As with coding examples in pyspark (i.e. python)

Q01. How will you create a Spark context? A01.

from pyspark.sql import SparkSession

 spark = SparkSession.builder.appName("my spark job")
 spark.master('local[*]')

 spark.config('spark.jars.packages', 'com.amazonaws:aws-java-sdk:1.11.297,org.apache.hadoop:hadoop-aws:2.8.3,mysql:mysql-connector-java:5.1.46')
              .config('spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version', 2)
              .config('spark.speculation', 'false')

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("my spark job")

spark.

…

Apache Spark SQL join types interview Q&As

Q1. What are the different Spark SQL join types?
A1. There are different SQL join types like inner join, left/right outer joins, full outer join, left semi-join, left anti-join and self-join.

Q2. Given the below tables, can give examples of the above join types?


package com.myapp

import org.apache.spark.sql.SparkSession

object MySparkApp4 {

  case class Customer(code: String, name: String)
  case class Order(id: String, product: String, custCode: String)

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[*]").appName("Spark

…