FAQs Data: 01: Getting started with Spark Scala on IntelliJ IDEA

Even though you can download required libraries via Maven pom.xml, in this tutorial will directly download so that you understand the key steps. Intellij IDEA is the most popular IDE for Spark development on Scala. Here are the 12 easy steps that will get you started with your Spark with Scala on IntelliJ IDEA IDE.

Step 1: Download Java 8 & set up the JAVA_HOME Environment variable.

Step 2: Install the latest IntelliJ IDEA IDE.

Step 3: Inside IntelliJ IDEA set up Scala Plugin & download Scala SDK.

IntelliJ IDEA –> Preferences –> Plugins

Intellij IDEA Scala plugin

Intellij IDEA Scala plugin

Step 4: Create a new project within Intellij IDEA with File –> New –> Maven. Select the “Project SDK“, which is the Java you downloaded.

Intellij IDEA New Maven Project

Intellij IDEA New Maven Project

Click on “NEXT“, and name the project “spark-on-scala“.

Step 5: Rename src.main/java to src/main/scala via right mouse click context menu & “Refactor“.

Step 6: File –> Project Structure. Add Scala SDK via “Global Libraries” & then add Scala SDK library via “Modules“. Click on the “+” icon & add as a Library.

Intellij IDEA IDE - Project Structure

Intellij IDEA IDE – Project Structure

Once added the Scala module, you can see the Scala libraries in the “External Libraries” as shown below:

Intellij IDEA - External Libraries

Intellij IDEA – External Libraries

Step 7: Download the Spark Scala library from https://spark.apache.org/downloads.html as a zipped file & unzip it a folder of your choice.

Step 8: Add the Spark module to your project via “File” –> “Project Structure” –> “Module“. Click on “+” icon & select “JARS or Directories” and point to the Spark folder with jars, which you downloaded & unzipped. E.g. “/Users/ak/projects/spark-3.2.1-bin-hadoop3.2/jars”.

This will now display all the Spark transitive libraries in the “External Libraries“.

Step 9: Create a new package under “src/main/scala” by right mouse clicking to get the context menu, and then “New” –> “Package“. Name it “com.myapp“.

Step 10: Create a singleton Scala object under “src/main/scala/com/myapp” by right mouse clicking to get the context menu, and then “New” –> “Scala Class” –> “Object“. Name it “MySparkApp.scala”.

Step 11: Write your Spark code in Scala as shown below:

Intellij IDEA IDE - Spark Scala App

Intellij IDEA IDE – Spark Scala App

Step 12: The output on the terminal console will be:

In the next post we will see how we can bring the Scala & Spark libraries via Maven pom.xml. FAQs Data: 02: Getting started with Spark Scala on IntelliJ IDEA with Maven


300+ Java & Big Data Interview FAQs

16+ Java Key Areas Interview Q&As

800+ Java Interview Q&As

300+ Java & Big Data Tutorials

Top