Tutorials Archives | Page 15 of 30 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 11 12 13 14 15 16 17 18 19 … 30 »

07: Getting started with Spark-submit

This extends the tutorial Setting up & getting started with Spark local mode with Sbt & Scala and Setting up Spark standalone on Mac. Spark-submit is used to submit a Spark job. In this tutorial we will be using the “local” master in “client” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Learn Spark Dataframes to do ETL in Java with examples

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. What is a Spark Dataframe? A DataFrame is an immutable distributed collection of data like an RDD, but unlike an RDD, data is organized into named columns...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Spark on Zeppelin – window functions in Scala

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Q. What are the different types of functions in Spark SQL? … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: spark-xml to split & read very large XML files

Processing very large XML files can be a bit tricky as they cannot be processed line by line in parallel as you would do with CSV files. The xml file has to be intact whilst matching the start and end entity tags, and if the tags are distributed in parts...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Spring Cloud with Resilience4j CircuitBreaker

Q. What is Resilience4j?
A. Resilience4j is a fault-tolerance library implementing the Circuit Breaker pattern inspired by Netflix Hystrix, but offers a more convenient API and a number of other features like Rate Limiter to block too many frequent requests to some services,

…

07. ⏯ Spring DI with Maven beginner video tutorial

Related Links 1. Spring DIP, DI & IoC in detail interview Q&As. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Apache Spark standalone cluster on Docker to read from AWS S3 bucket & write to MySQL database

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, MySQL DB, Spark master and Spark worker to form a cluster. “mysql” is the hostname.

version: "2" 

services:
  s3:
    container_name: s3
    image: minio/minio:RELEASE.2018-05-04T23-13-12Z
    environment:
      MINIO_ACCESS_KEY: user
      MINIO_SECRET_KEY: password
    ports:
      - 9000:9000
    volumes:
      - ./docker/s3/data:/data
      - ./docker/s3/config:/root/.minio
    networks:
      - infra
    command: server /data

  mysql:
    container_name: mysql
    image: mysql:5.7.21
    ports:
      - 3306:3306
    volumes:
      - ./docker/mysql/data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: password
    networks:
      - infra

  spark-master:
      build:
        dockerfile: spark.dockerfile
        context: ./docker/spark
      command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master
      hostname: spark-master
      environment:
        MASTER: spark://spark-master:7077
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: 127.0.0.1
      expose:
        - 7001
        - 7002
        - 7003
        - 7004
        - 7005
        - 7006
        - 7077
        - 6066
        - 5005
      ports:
        - 5005:5005
        - 6066:6066
        - 7077:7077
        - 8080:8080
      volumes:
        - ./docker/spark/spark-master:/conf
        - ./data:/tmp/data 
      networks:
        - infra 
  
  spark-worker-1:
      build:
        dockerfile: spark.dockerfile
        context: ./docker/spark
      command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
      hostname: spark-worker-1
      environment:
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: 127.0.0.1
        SPARK_WORKER_CORES: 2
        SPARK_WORKER_MEMORY: 2g
        SPARK_WORKER_PORT: 8881
        SPARK_WORKER_WEBUI_PORT: 8081
      links:
        - spark-master
      expose:
        - 7012
        - 7013
        - 7014
        - 7015
        - 7016
        - 8881
      ports:
        - 8081:8081
      volumes:
        - ./conf/spark-worker-1:/conf
        - ./data:/tmp/data 
      networks:
        - infra

networks:
  infra:
    external:
      name: docker_test_infra

version: "2"

services:

s3:

container_name: s3

image: minio/minio:RELEASE.2018-05-04T23-13-12Z

environment:

MINIO_ACCESS_KEY: user

MINIO_SECRET_KEY: password

ports:

- 9000:9000

volumes:

- ./docker/s3/data:/data

- ./docker/s3/config:/root/.minio

networks:

- infra

command: server /data

mysql:

container_name: mysql

image: mysql:5.7.21

ports:

- 3306:3306

volumes:

- ./docker/mysql/data:/var/lib/mysql

environment:

MYSQL_ROOT_PASSWORD: password

networks:

- infra

spark-master:

build:

dockerfile: spark.dockerfile

context: ./docker/spark

command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master

hostname: spark-master

environment:

MASTER: spark://spark-master:7077

SPARK_CONF_DIR: /conf

SPARK_PUBLIC_DNS: 127.0.0.1

expose:

- 7001

- 7002

- 7003

- 7004

- 7005

- 7006

- 7077

- 6066

- 5005

ports:

- 5005:5005

- 6066:6066

- 7077:7077

- 8080:8080

volumes:

- ./docker/spark/spark-master:/conf

- ./data:/tmp/data

networks:

- infra

spark-worker-1:

build:

dockerfile: spark.dockerfile

context: ./docker/spark

command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077

hostname: spark-worker-1

environment:

SPARK_CONF_DIR: /conf

SPARK_PUBLIC_DNS: 127.0.0.1

SPARK_WORKER_CORES: 2

SPARK_WORKER_MEMORY: 2g

SPARK_WORKER_PORT: 8881

SPARK_WORKER_WEBUI_PORT: 8081

links:

- spark-master

expose:

- 7012

- 7013

- 7014

- 7015

- 7016

- 8881

ports:

- 8081:8081

volumes:

- ./conf/spark-worker-1:/conf

- ./data:/tmp/data

networks:

- infra

networks:

infra:

external:

name: docker_test_infra

Step 2: The “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Databricks – Spark problem 1

Prerequisite: Extends Databricks – Spark Window functions.

Problem: Convert the below table

+------+--------+---------+-------------+
|emp_id|emp_name| emp_city|emp_expertise|
+------+--------+---------+-------------+
|     1|    John|   Sydney|         Java|
|     2|   Peter|Melbourne|        Scala|
|     3|     Sam| Brisbane|       Python|
|     4|   David|Melbourne|       Python|
|     5|  Elliot|   Sydney|         Java|
+------+--------+---------+-------------+

+------+--------+---------+-------------+

+------+--------+---------+-------------+

| 3| Sam| Brisbane| Python|

+------+--------+---------+-------------+

+-------------+----------------------------------------------------------+
|Column       |Content                                                   |
+-------------+----------------------------------------------------------+
|emp_id       |[[1, 1], [3, 1], [5, 1], [4, 1], [2, 1]]                  |
|emp_name     |[[Elliot, 1], [John, 1], [Sam, 1], [Peter, 1], [David, 1]]|
|emp_city     |[[Sydney, 2], [Melbourne, 2], [Brisbane, 1]]              |
|emp_expertise|[[Python, 2], [Java, 2], [Scala, 1]]                      |
+-------------+----------------------------------------------------------+

+-------------+----------------------------------------------------------+

|Column |Content |

+-------------+----------------------------------------------------------+

|emp_id |[[1, 1], [3, 1], [5, 1], [4, 1], [2, 1]] |

|emp_name |[[Elliot, 1], [John, 1], [Sam, 1], [Peter, 1], [David, 1]]|

|emp_city |[[Sydney, 2], [Melbourne, 2], [Brisbane, 1]] |

|emp_expertise|[[Python, 2], [Java, 2], [Scala, 1]] |

+-------------+----------------------------------------------------------+

Where each column is counted for its occurrence. For example, the emp_city column values “Melbourne”

…

08: Docker Tutorial – client & server with docker-compose.yml

Q1. What is docker compose? A1. Docker compose allows us to configure and start multiple containers within the same host. This is handy if you don’t want to start each container separately. Q2. How does docker compose differ from Docker swarm? … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Learn Spark how to convert RDD in Java to Dataframe with examples

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. Java RDD to Dataframe The following code reads a text file as shown below into a Java RDD. orders.txt


"ord123","John" 
"ord456","Peter"
"ord789","Samuel"

"ord123","John"

"ord456","Peter"

"ord789","Samuel"

The Java code that read the text...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Spark on Zeppelin – convert DataFrames to RDD[Row] and RDD[Row] to DataFrame

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Important: It is not a best practice to mutate values or to use RDD directly as opposed to using Dataframes....

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Spark writing RDDs to multiple text files & HAR to solve small files issue

We know that the following code snippets in Spark will write each JavaRDD element to a single file


employeesRdd.saveAsTextFile(pathToHdfs)

employeesRdd.saveAsTextFile(pathToHdfs)

What if you want to write each employee history to a separate file? Step 1: Create a JavaPairRDD from JavaRDD


JavaPairRDD<String, String> empPairRdd = employeesRdd.mapToPair(new PairFunction<Employee, String, String>() {

    @Override
    public Tuple2<String, String> call(Employee t) throws Exception {
        return new Tuple2<String, String>(t.getEmpId, t.getEmpHistoryDump());
    }
});

JavaPairRDD<String, String> empPairRdd = employeesRdd.mapToPair(new PairFunction<Employee, String, String>() {

@Override

public Tuple2<String, String> call(Employee t) throws Exception {

return new Tuple2<String, String>(t.getEmpId, t.getEmpHistoryDump());

}

});

Step 2: Create a MultipleOutputFormat, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

08: Spring boot with OpenAPI (Swagger) tutorial

This tutorial extends Create a simple REST API with Spring Boot.

Q1. What is Swagger?
A1. The OpenAPI (used to be called Swagger) automates the documentation of REST APIs. Documentation of any REST Service API is a must because

1) Consumers of the services need to know which all services are available,

…

Blog Archives

07: Getting started with Spark-submit

07: Learn Spark Dataframes to do ETL in Java with examples

07: Spark on Zeppelin – window functions in Scala

07: spark-xml to split & read very large XML files

07: Spring Cloud with Resilience4j CircuitBreaker

07. ⏯ Spring DI with Maven beginner video tutorial

08: Apache Spark standalone cluster on Docker to read from AWS S3 bucket & write to MySQL database

08: Databricks – Spark problem 1

08: Docker Tutorial – client & server with docker-compose.yml

08: Learn Spark how to convert RDD in Java to Dataframe with examples

08: Spark on Zeppelin – convert DataFrames to RDD[Row] and RDD[Row] to DataFrame

08: Spark writing RDDs to multiple text files & HAR to solve small files issue

08: Spring boot with OpenAPI (Swagger) tutorial

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer