Q01. What are the different types of data that get stored in a data lake or data warehouse? A01. An enterprise stores different types of data. #1. Transactional Data Transactional data describes business events like placing an order. Example 1: If you take a Telecom service provider there could be...
Q01. What is a data pipeline? A01. A data pipeline is a means of moving data from one place (i.e source systems e.g. OLTP) to a destination such as a data lake or warehouse by simultaneously optimising and transforming the data. … Read more ›...
This extends Remotely debugging Spark submit Jobs in Java. Running Spark in local mode When you run Spark in local mode, both the Driver and Executor will be running in the same JVM and is very handy to debug the logic of your transformations. … Read more ›...
Uses DB-FIDDLE for the code example with PostgreSQL v17.
Q01 How will you go about defining tables with primary key, foreign keys and unique constraints?
A01 The main purpose of a primary key in a table is to uniquely identify each record.
…
Even though you can download required libraries via Maven pom.xml, in this tutorial will directly download so that you understand the key steps. Intellij IDEA is the most popular IDE for Spark development on Scala. Here are the 12 easy steps that will get you started with your Spark with Scala on IntelliJ IDEA IDE.
…
This extends FAQs Data: 01: Getting started with Spark Scala on IntelliJ IDEA to package the Spark & run it via command-line. The dependent libraries configured via Maven pom.xml. Make sure Maven is installed on your machine & set in the path.
Step 1: Extend the previous pom.xml file from
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.example</groupId> <artifactId>spark-on-scala</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> </properties> </project> |
TO:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.example</groupId> <artifactId>spark-on-scala2</artifactId> <version>1.0-SNAPSHOT</version> <name>${project.artifactId}</name> <description>My wonderfull scala app</description> <inceptionYear>2018</inceptionYear> <licenses> <license> <name>My License</name> <url>http://....</url> <distribution>repo</distribution> </license> </licenses> <properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <encoding>UTF-8</encoding> <scala.version>2.12.15</scala.version> <scala.compat.version>2.12</scala.compat.version> <spec2.version>4.2.0</spec2.version> <spark.version>3.2.1</spark.version> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_${scala.compat.version}</artifactId> <version>${spark.version}</version> <scope>provided</scope> </dependency> <!-- Test --> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> <dependency> <groupId>org.scalatest</groupId> <artifactId>scalatest_${scala.compat.version}</artifactId> <version>3.1.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.specs2</groupId> <artifactId>specs2-core_${scala.compat.version}</artifactId> <version>${spec2.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.specs2</groupId> <artifactId>specs2-junit_${scala.compat.version}</artifactId> <version>${spec2.version}</version> <scope>test</scope> </dependency> </dependencies> <build> <sourceDirectory>src/main/scala</sourceDirectory> <testSourceDirectory>src/test/scala</testSourceDirectory> <plugins> <plugin> <!-- see http://davidb.github.com/scala-maven-plugin --> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.3.2</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> <configuration> <args> <arg>-dependencyfile</arg> <arg>${project.build.directory}/.scala_dependencies</arg> </args> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.21.0</version> <configuration> <!-- Tests will be run with scalatest-maven-plugin instead --> <skipTests>true</skipTests> </configuration> </plugin> <plugin> <groupId>org.scalatest</groupId> <artifactId>scalatest-maven-plugin</artifactId> <version>2.0.0</version> <configuration> <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory> <junitxml>.</junitxml> <filereports>WDF TestSuite.txt</filereports> </configuration> <executions> <execution> <id>test</id> <goals> <goal>test</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> |
Step 2: Rename “src/test/java”
…
This extends FAQs Data: 01: Getting started with Spark Scala on IntelliJ IDEA.
Step 1: Note the relevant test libraries in the pom.xml file.
scalatest-maven-plugin is used to run the tests & scalatest_* &
…
This extends FAQs Data: 03: ScalaTest using IntelliJ IDEA IDE and FAQs Data: 02: Getting started with Spark Scala on IntelliJ IDEA with Maven.
Step 1: Firstly, let’s rewrite the MySparkApp.scala into more testable & maintainable methods as shown below under “
…
What is Apache Hadoop YARN? Apache Hadoop YARN (Yet Another Resource Negotiator) is the prerequisite for Enterprise Hadoop for dynamic allocation oc the cluster resources. For example, when you run a Spark job on YARN as opposed to in standalone mode, you can will take advantage of categorizing, isolating, and...
Q1. How would you go about.. building an ETL pipeline to handle 3k TPS? A1. Building a data pipeline (i.e. aka ETL) for 3000 TPS (i.e. Transactions per second), which about ~260 million records per day requires distributed, highly scalable, … Read more ›...
Q01. What is InfluxDb?
A01. InfluxDB is an open-source time series database. It is a Big Data, NoSQL database that allows for massive scalability, high availability, fast write, and fast read. As a NoSQL, InfluxDB stores time-series data,
…
Q1. What are the different ways to access files or data stored in HDFS? A1. You can access files or data stored in HDFS in many different ways. HDFS Command-Line Interface The HDFS command-line interface (CLI), called hdfs (or hadoop) shell, … Read more ›...
Q1. What is a multi-tenant architecture? A1. A “tenant” in cloud computing is a single customer or to be more specific a group of users who share a common access with specific privileges to the software instance. … Read more ›...