This extends 01: Docker tutorial with Java & Maven. This runs Spark in local mode. You build the Spark code as a jar file and run it as a Java application in the docker container with the “java” command. You can either create a fat or uber jar with the...

This extends 01: Docker tutorial with Java & Maven. Step 1: Create a Docker network where all 3 containers – Spark master (i.e. Stand-alone cluster manager), Spark worker, and Spark driver will be deployed to. Step 2: The image docker file “ … Read more ›...

This extends the previous Docker tutorials. This is a stand-alone Spark cluster tutorial on Docker compose. Step 1: The image docker file “spark.dockerfile” in the folder “docker-test/docker/spark/”. Step 2: The pom.xml file. Step 3: The Spark code “ … Read more ›...

Install minio, which is Amazon S3 compatible API Minio is an open source object storage server with Amazon S3 compatible API. Step 1: The “docker-compose.yml” with minio to emulate AWS S3. Step 2: You can start the minio service as a daemon (i.e with -d) by running Step 3: You...

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, Spark master and Spark worker to form a cluster. Step 2: The “spark.dockerfile” … Read more ›...

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, Spark master and Spark worker to form a cluster. Step 2: The “spark.dockerfile” … Read more ›...