FAQs Data: 04: ScalaTest of Spark using IntelliJ IDEA IDE

This extends FAQs Data: 03: ScalaTest using IntelliJ IDEA IDE and FAQs Data: 02: Getting started with Spark Scala on IntelliJ IDEA with Maven.

Step 1: Firstly, let’s rewrite the MySparkApp.scala into more testable & maintainable methods as shown below under “src/main/scala

Step 2: Let’s create an empty abstract base class “MyTestBase” for our tests in “src/test/scala”. Even though it is empty, in future it will be useful for mocking & having common behaviour among all the tests.

Step 3: Let’s create a test class “SimpleSparkTest” for our tests in “src/test/scala”. This will test for creation of SparkSession & creation of a Dataframe.

When you run the BDD test “SimpleSparkTest” above it runs successfully.

Spark Scala Test on IntelliJ IDEA

Spark Scala Test on IntelliJ IDEA

Negative test

to see if our tests are working fine by purposely creating a side effect in the “MySparkApp” method “createDataframe(…)” by sneaking in 2 additional items Python & PySpark as shown below.

When you run your test again it will fail with the below assert error:

300+ Java & Big Data Interview FAQs

16+ Java Key Areas Interview Q&As

800+ Java Interview Q&As

300+ Java & Big Data Tutorials