Q01. Can you summarise the Spark eco system?
A01. Apache Spark is a general purpose cluster computing system. It provides high-level API in Java, Scala, Python, and R. It has 6 components Core, Spark SQL, Spark Streaming, Spark MLlib, Spark GraphX, and SparkR. All the functionalities being provided by Apache Spark are built on the top of Spark Core.…
Q1. Why are data cleansing & pre-processing important in analytics & machine learning?
A1. Garbage in gets you garbage out. No matter how good your machine learning algorithm is.
Q2. What are the general steps of cleansing data
A2. General steps involve Deduplication, dropping/imputing missing values, fixing structural errors, removing the outliers, encoding the categorical values and scaling down the features.…
Q01. Where is Apache Spark used in the Hadoop eco system?
A01. Spark is essentially a data processing framework that is faster & more flexible than “Map Reduce”. The Spark itself has grown into an eco system with Spark SQL, Spark streaming, Spark UI, GraphX, MLlib, and SparkR. Apache Spark can run on Hadoop clusters, as a standalone system or on the cloud.…
Q105. What is the difference between “map” and “flatMap” operations in Spark? A105. The map and flatMap are transformation operations in Spark. map transformation is applied to each element of…
Login Register 100+ Free Java interview FAQs 100+ Free Big Data interview FAQs
Q109. What is the difference between “map” and “mapPartitions” transformations in Spark? A109. The method map converts each element of the source RDD into a single element of the result…
Login Register 100+ Free Java interview FAQs 100+ Free Big Data interview FAQs
Q01. How will you create a Spark context? A01.
|
1 2 3 4 5 6 7 8 9 |
from pyspark.sql import SparkSession spark = SparkSession.builder.appName("my spark job") spark. |
Q38 What causes small files to be created? A38 Files that are in few KBs or MBs are considered to be small files. Ideally anything smaller than the block size…
Login Register 100+ Free Java interview FAQs 100+ Free Big Data interview FAQs
There are so many different ways to solve the big data problems at hand in Spark, but some approaches can impact on performance, and lead to performance and memory issues. Here are some best practices to keep in mind when writing Spark jobs.
#1 Favor DataFrames, Datasets, or SparkSQL over RDDs: Spark DataFrames, Datasets or SparkSQL are optimised, hence faster than RDDs, especially when working with structured data.…
This extends 40+ Apache Spark best practices & optimisation interview FAQs – Part-1, where best practices 1-10 were covered with examples & diagrams. #11 Use Spark UI: Running Spark jobs…
Login Register 100+ Free Java interview FAQs 100+ Free Big Data interview FAQs
This extends 40+ Apache Spark best practices & optimisation interview FAQs – Part-2 Spark UI. #31 Bucketing is another data optimisation technique that groups data with the same bucket value…
Login Register 100+ Free Java interview FAQs 100+ Free Big Data interview FAQs