05: Apache Spark local mode on Docker to read from AWS S3 bucket

Posted on May 28, 2018 by

Install minio, which is Amazon S3 compatible API Minio is an open source object storage server with Amazon S3 compatible API. Step 1: The “docker-compose.yml” with minio to emulate AWS S3.

version: "2" 
services:
  s3:
    container_name: s3
    image: minio/minio:RELEASE.2018-05-04T23-13-12Z
    environment:
      MINIO_ACCESS_KEY: user
      MINIO_SECRET_KEY: password
    ports:
      - 9000:9000
    volumes:
      - ./docker/s3/data:/data
      - ./docker/s3/config:/root/.minio
    networks:
      - infra
    command: server /data
networks:
  infra:
    external:
      name: docker_test_infra

version: "2"

services:

s3:

container_name: s3

image: minio/minio:RELEASE.2018-05-04T23-13-12Z

environment:

MINIO_ACCESS_KEY: user

MINIO_SECRET_KEY: password

ports:

- 9000:9000

volumes:

- ./docker/s3/data:/data

- ./docker/s3/config:/root/.minio

networks:

- infra

command: server /data

networks:

infra:

external:

name: docker_test_infra

Step 2: You can start the minio service…

05: Apache Spark local mode on Docker to read from AWS S3 bucket

Members Only Content

🔥 300+ Java Interview FAQs

🔥 300+ Big Data Interview FAQs

Companion Tech Interview FAQs

16+ Java Key Areas to go places

800+ Java interview Q&As

Java & Big Data Tutorials

How to open more doors?

Disclaimer