Tutorials - Big Data Archives | Page 5 of 13 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 2 3 4 5 6 7 8 9 … 13 »

04. Setting up & getting started with sbt

SBT (i.e. Simple Build Tool) for Scala requires Java & Scala. This tutorial is based on Java 8 (i.e. 1.8.0) and Scala 2.12.7. Scala runs on the JVM, so Java and Scala stacks can be freely mixed. You can call Java libraries from Scala. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04a: Databricks – Spark SCD Type 1 with Merge

Prerequisite: Extends 03: Databricks – Spark SCD Type 1. What is SCD Type 1 SCD stands for Slowly Changing Dimension, and it was explained in 10 Data warehouse interview Q&As. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

04b: Databricks – Spark SCD Type 2 with Merge

Databricks SCD Type 2 with null mergeKey

Prerequisite: … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Apache Kafka Streaming with JSON & Java Tutorial

This extends Apache Kafka JSON example with Java Producer & Consumer Tutorial. In this tutorial, let’s look at KafkaStreams, which enables you to consume from Kafka topics, analyse, transform or aggregate data, and potentially, send it to another Kafka topic.

This kafka tutorial is stateless,

…

05: Apache Spark local mode on Docker to read from AWS S3 bucket

Install minio, which is Amazon S3 compatible API Minio is an open source object storage server with Amazon S3 compatible API. Step 1: The “docker-compose.yml” with minio to emulate AWS S3.

version: "2" 

services:
  s3:
    container_name: s3
    image: minio/minio:RELEASE.2018-05-04T23-13-12Z
    environment:
      MINIO_ACCESS_KEY: user
      MINIO_SECRET_KEY: password
    ports:
      - 9000:9000
    volumes:
      - ./docker/s3/data:/data
      - ./docker/s3/config:/root/.minio
    networks:
      - infra
    command: server /data

networks:
  infra:
    external:
      name: docker_test_infra

version: "2"

services:

s3:

container_name: s3

image: minio/minio:RELEASE.2018-05-04T23-13-12Z

environment:

MINIO_ACCESS_KEY: user

MINIO_SECRET_KEY: password

ports:

- 9000:9000

volumes:

- ./docker/s3/data:/data

- ./docker/s3/config:/root/.minio

networks:

- infra

command: server /data

networks:

infra:

external:

name: docker_test_infra

Step 2: You can start the minio service as a daemon (i.e with -d) by running

$ docker-compose up --build -d

1 2	$ docker-compose up --build -d

Step...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Apache Zeppelin on Docker – Hadoop Tutorial

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This tutorial extends the series: Spark on Apache Zeppelin Tutorials. Step 1: Pull Docker image from the Docker Hub – … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Convert XML file To an Avro File with avro-maven-plugin & Apache Spark

This extends 04: Convert XML file To an Avro File with Apache Spark – writing & reading. Instead of using the GenericRecord, let’s generate an avro schema object from the avro schema. Step 1: The pom.xml file should include the Apache Spark & … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Create or append a Sequence file to HDFS – Hadoop API tutorial

The following tutorial extends Create or append a file to HDFS – Hadoop API tutorial, and Create or append an AVRO file to HDFS – Hadoop & AVRO API tutorial. In this tutorial we will write to a Sequence file, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Databricks – Spark UDFs

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL.

What is a UDF?

User-Defined Functions (aka UDF) is a feature of Spark SQL to define new Column-based functions that extend the vocabulary of Spark SQL’s DSL for transforming Datasets.

…

05: Learn Hive to write to and read from AVRO & Parquet files by examples

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. What is Apache Hive? Hive allows SQL developers to write Hive Query Language (HQL) statements that are similar to standard SQL statements. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Setting up & getting started with Spark local mode with Sbt & Scala

This extends Setting up & getting started with sbt. Spark 2.3.2 requires Scala 2.11.x. You need to use the right Scala version. Step 1: On Eclipse set the Scala compiler to 2.11. Right mouse click on the project “sbt-tutorial” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Spark on Zeppelin – semi-structured log file

This tutorial extends the series: Spark on Apache Zeppelin Tutorials. Step 1: Pull apache/zeppelin image from the docker hub, and build the image with the following command.

$ docker pull apache/zeppelin:0.7.3

1 2	$ docker pull apache/zeppelin:0.7.3

“docker images” will show the image that was created. Step 2: Run the above image to create a container with the following...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Spark SQL & CSV with DataFrame Tutorial

Step 1: Create a simple maven project.


mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-spark

mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-spark

Step 2: Import the “simple-spark” maven project into eclipse or IDE of your choice. Step 3: Modify the pom.xml file include 1) relevant Spark libraries 2) The shade plugin to create a single jar (i.e. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 2 3 4 5 6 7 8 9 … 13 »

Blog Archives

04. Setting up & getting started with sbt

04a: Databricks – Spark SCD Type 1 with Merge

04b: Databricks – Spark SCD Type 2 with Merge

05: Apache Kafka Streaming with JSON & Java Tutorial

05: Apache Spark local mode on Docker to read from AWS S3 bucket

05: Apache Zeppelin on Docker – Hadoop Tutorial

05: Convert XML file To an Avro File with avro-maven-plugin & Apache Spark

05: Create or append a Sequence file to HDFS – Hadoop API tutorial

05: Databricks – Spark UDFs

05: Learn Hive to write to and read from AVRO & Parquet files by examples

05: Setting up & getting started with Spark local mode with Sbt & Scala

05: Spark on Zeppelin – semi-structured log file

05: Spark SQL & CSV with DataFrame Tutorial

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer