Tutorials Archives | Page 14 of 30 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 10 11 12 13 14 15 16 17 18 … 30 »

06: Simple Akka remoting tutorial in Java – producer & consumer

This tutorial extends the previous tutorial Simple Akka tutorial in Java step by step. This tutorial is all about remote actors. In other words, inter process communication. The “ProducerMain.java” and “ConsumerMain.java” run as different processes in host “127.0.0.1” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

06: Spark on Zeppelin – RDD operation zipWithIndex

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Q. Why do we need zipWithIndex? A. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

06: Spark Streaming with Flume Avro Sink Tutorial

This extends Running a Simple Spark Job in local & cluster modes and Apache Flume with JMS source (Websphere MQ) and HDFS sink. In this tutorial a Flume sink will ingest the data from a source like JMS, HDFS, etc and pass it to an “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

06: Spring Cloud declarative REST client Feign

Q. What use Feign client?
A. Feign is a declarative web service client, which makes creation of web service clients a breeze.

1) Unlike a RestTemplate, Feign client acts as a load balancer. Spring Cloud integrates Netflix/ribbon and Netflix/eureka to provide a load balanced http client when using Feign.

…

06. Getting started with GitHub, EGit & Eclipse for Maven based Java projects

Create a GitHub account Step 1: Go to the following link and “Sign up” a personal account to store & share your open-source projects. https://github.com/join Step 2: Login to your account, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

06a: ⏯ Spring 4 MVC RESTful POST method Video Tutorial

Extends Video Tutorial 1. ⏯ Spring 4 MVC RESTful Web Service Video Tutorial. Related Links 1. Spring 4 MVC RESTful GET & POST methods Web Services Tutorial. 1. “12 Rules” to write RESTFul Web Service API/a. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

06a. ⏯ Getting started with GitHub, EGit & Eclipse video tutorial

Correction: Where it says “maven” in the audio, should be corrected to “subversion“. Related Links 1. Getting started with GitHub, EGit & Eclipse for Maven based Java projects. 2. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

06b. ⏯ GitHub pull request with EGit video tutorial

Related Links 1. Getting started with GitHub, EGit & Eclipse for Maven based Java projects. 2. 8 Git Source control system interview questions & answers. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Apache Kafka Streaming with stateless operations

This extends 06: Apache Kafka Streaming with JSON & Java Tutorial – stateful operations groupByKey, windowing & aggregate and Getting started with Apache Kafka on Mac Tutorial.

Kafka Streams DSL (i.e. Domain Specific Language) is built on top of the Streams Processor API.

…

07: Apache Spark standalone cluster on Docker to read from & write to AWS S3 bucket

This extends Apache Spark local mode read from AWS S3 bucket with Docker. Step 1: The “docker-compose.yml” with minio to emulate AWS S3, Spark master and Spark worker to form a cluster.

version: "2" 

services:
  s3:
    container_name: s3
    image: minio/minio:RELEASE.2018-05-04T23-13-12Z
    environment:
      MINIO_ACCESS_KEY: user
      MINIO_SECRET_KEY: password
    ports:
      - 9000:9000
    volumes:
      - ./docker/s3/data:/data
      - ./docker/s3/config:/root/.minio
    networks:
      - infra
    command: server /data

  spark-master:
      build:
        dockerfile: spark.dockerfile
        context: ./docker/spark
      command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master
      hostname: spark-master
      environment:
        MASTER: spark://spark-master:7077
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: 127.0.0.1
      expose:
        - 7001
        - 7002
        - 7003
        - 7004
        - 7005
        - 7006
        - 7077
        - 6066
        - 5005
      ports:
        - 5005:5005
        - 6066:6066
        - 7077:7077
        - 8080:8080
      volumes:
        - ./docker/spark/spark-master:/conf
        - ./data:/tmp/data 
      networks:
        - infra 
  
  spark-worker-1:
      build:
        dockerfile: spark.dockerfile
        context: ./docker/spark
      command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
      hostname: spark-worker-1
      environment:
        SPARK_CONF_DIR: /conf
        SPARK_PUBLIC_DNS: 127.0.0.1
        SPARK_WORKER_CORES: 2
        SPARK_WORKER_MEMORY: 2g
        SPARK_WORKER_PORT: 8881
        SPARK_WORKER_WEBUI_PORT: 8081
      links:
        - spark-master
      expose:
        - 7012
        - 7013
        - 7014
        - 7015
        - 7016
        - 8881
      ports:
        - 8081:8081
      volumes:
        - ./conf/spark-worker-1:/conf
        - ./data:/tmp/data 
      networks:
        - infra

networks:
  infra:
    external:
      name: docker_test_infra

version: "2"

services:

s3:

container_name: s3

image: minio/minio:RELEASE.2018-05-04T23-13-12Z

environment:

MINIO_ACCESS_KEY: user

MINIO_SECRET_KEY: password

ports:

- 9000:9000

volumes:

- ./docker/s3/data:/data

- ./docker/s3/config:/root/.minio

networks:

- infra

command: server /data

spark-master:

build:

dockerfile: spark.dockerfile

context: ./docker/spark

command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master

hostname: spark-master

environment:

MASTER: spark://spark-master:7077

SPARK_CONF_DIR: /conf

SPARK_PUBLIC_DNS: 127.0.0.1

expose:

- 7001

- 7002

- 7003

- 7004

- 7005

- 7006

- 7077

- 6066

- 5005

ports:

- 5005:5005

- 6066:6066

- 7077:7077

- 8080:8080

volumes:

- ./docker/spark/spark-master:/conf

- ./data:/tmp/data

networks:

- infra

spark-worker-1:

build:

dockerfile: spark.dockerfile

context: ./docker/spark

command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077

hostname: spark-worker-1

environment:

SPARK_CONF_DIR: /conf

SPARK_PUBLIC_DNS: 127.0.0.1

SPARK_WORKER_CORES: 2

SPARK_WORKER_MEMORY: 2g

SPARK_WORKER_PORT: 8881

SPARK_WORKER_WEBUI_PORT: 8081

links:

- spark-master

expose:

- 7012

- 7013

- 7014

- 7015

- 7016

- 8881

ports:

- 8081:8081

volumes:

- ./conf/spark-worker-1:/conf

- ./data:/tmp/data

networks:

- infra

networks:

infra:

external:

name: docker_test_infra

Step 2: The “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Avro IDL (e.g. avdl) to Java objects & Avro Schemas (i.e. avsc) tutorial

Avro IDL (i.e Interface Definition Language) schema can be specified with two type of files “avpr” (i.e. AVro PRotocol file) & “avdl” (i.e. AVro iDL). Step 1: Create a maven based Java project from a command-line

$ mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-avro

1	$ mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-avro

Step 2: Import it into eclipse as a maven project. … Read more...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Databricks – groupBy, collect_list & explode

Prerequisite: Extends Databricks – Spark Window functions. Step 1: Create a new Python notebook, and attach it to a cluster. Step 2: Let’s create some data using pyspark.

emp_list = (
  (1, 'John', 'Java'),
  (2, 'Peter', 'Scala'),
  (2, 'Peter', 'Python'),
  (2, 'Peter', 'Spark'),
  (1, 'John', 'JEE'),
  (6, 'Elliot', 'Unix',)
)
 
data_schema = [StructField('emp_id', IntegerType(), False), StructField('emp_name', StringType(), False), StructField('emp_expertise', StringType(), False)]
final_struc = StructType(fields=data_schema)
 
df = spark.sparkContext.parallelize(emp_list);
 
emp_df = spark.createDataFrame(df, final_struc)
emp_df.show()

emp_list = (

(1, 'John', 'Java'),

(2, 'Peter', 'Scala'),

(2, 'Peter', 'Python'),

(2, 'Peter', 'Spark'),

(1, 'John', 'JEE'),

(6, 'Elliot', 'Unix',)

)

data_schema = [StructField('emp_id', IntegerType(), False), StructField('emp_name', StringType(), False), StructField('emp_expertise', StringType(), False)]

final_struc = StructType(fields=data_schema)

df = spark.sparkContext.parallelize(emp_list);

emp_df = spark.createDataFrame(df, final_struc)

emp_df.show()

Output:

+------+--------+-------------+
|emp_id|emp_name|emp_expertise|
+------+--------+-------------+
|     1|    John|         Java|
|     2|   Peter|        Scala|
|     2|   Peter|       Python|
|     2|   Peter|        Spark|
|     1|    John|          JEE|
|     6|  Elliot|         Unix|
+------+--------+-------------+

+------+--------+-------------+

|emp_id|emp_name|emp_expertise|

+------+--------+-------------+

| 1| John| Java|

| 2| Peter| Scala|

| 2| Peter| Python|

| 2| Peter| Spark|

| 1| John| JEE|

| 6| Elliot| Unix|

+------+--------+-------------+

agg( ) &… Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

07: Docker Tutorial – Spring boot with Docker maven plugin

This tutorial extends the 06: Docker Tutorial – Spring boot application. In the previous tutorial the Dockerfile has the jar filename & the version. Every time the version changes the Dockerfile has to be changed. Also, we have to manually type the Docker commands by opening a terminal. … Read...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 10 11 12 13 14 15 16 17 18 … 30 »

Blog Archives

06: Simple Akka remoting tutorial in Java – producer & consumer

06: Spark on Zeppelin – RDD operation zipWithIndex

06: Spark Streaming with Flume Avro Sink Tutorial

06: Spring Cloud declarative REST client Feign

06. Getting started with GitHub, EGit & Eclipse for Maven based Java projects

06a: ⏯ Spring 4 MVC RESTful POST method Video Tutorial

06a. ⏯ Getting started with GitHub, EGit & Eclipse video tutorial

06b. ⏯ GitHub pull request with EGit video tutorial

07: Apache Kafka Streaming with stateless operations

07: Apache Spark standalone cluster on Docker to read from & write to AWS S3 bucket

07: Avro IDL (e.g. avdl) to Java objects & Avro Schemas (i.e. avsc) tutorial

07: Databricks – groupBy, collect_list & explode

07: Docker Tutorial – Spring boot with Docker maven plugin

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer