Blog Archives

01: Docker tutorial with Java & Maven

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As.

Step 1: Create a Java project “docker-test” with “HelloDocker.java” file under “src/main/java”

Read more ›



02: Apache Spark – local mode on Docker tutorial with Java & Maven

This extends 01: Docker tutorial with Java & Maven. This runs Spark in local mode. You build the Spark code as a jar file and run it as a Java application in the docker container with the “java” command. You can either create a fat or uber jar with the...



03: Apache Spark cluster – Docker compose tutorial with Java, Maven & spark-shell

This extends 01: Docker tutorial with Java & Maven. Step 1: Create a Docker network where all 3 containers – Spark master (i.e. Stand-alone cluster manager), Spark worker, and Spark driver will be deployed to. Step 2: The image docker file “ … Read more ›...



04: Apache Spark cluster – Docker compose tutorial with Java, Maven & spark-submit

This extends the previous Docker tutorials. This is a stand-alone Spark cluster tutorial on Docker compose. Step 1: The image docker file “spark.dockerfile” in the folder “docker-test/docker/spark/”. Step 2: The pom.xml file. Step 3: The Spark code “ … Read more ›...



05: Apache Spark local mode on Docker to read from AWS S3 bucket

Install minio, which is Amazon S3 compatible API Minio is an open source object storage server with Amazon S3 compatible API. Step 1: The “docker-compose.yml” with minio to emulate AWS S3. Step 2: You can start the minio service as a daemon (i.e with -d) by running Step 3: You...



06: Apache Spark standalone cluster on Docker to read from AWS S3 bucket

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, Spark master and Spark worker to form a cluster. Step 2: The “spark.dockerfile” … Read more ›...



07: Apache Spark standalone cluster on Docker to read from & write to AWS S3 bucket

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, Spark master and Spark worker to form a cluster. Step 2: The “spark.dockerfile” … Read more ›...



08: Apache Spark standalone cluster on Docker to read from AWS S3 bucket & write to MySQL database

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, MySQL DB, Spark master and Spark worker to form a cluster. “mysql” is the hostname. Step 2: The “ … Read more ›...



800+ Java Q&As & tutorials

Top