Tutorials Archives | Page 12 of 30 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 8 9 10 11 12 13 14 15 16 … 30 »

05: Convert XML file To an Avro File with avro-maven-plugin & Apache Spark

This extends 04: Convert XML file To an Avro File with Apache Spark – writing & reading. Instead of using the GenericRecord, let’s generate an avro schema object from the avro schema. Step 1: The pom.xml file should include the Apache Spark & … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Create or append a Sequence file to HDFS – Hadoop API tutorial

The following tutorial extends Create or append a file to HDFS – Hadoop API tutorial, and Create or append an AVRO file to HDFS – Hadoop & AVRO API tutorial. In this tutorial we will write to a Sequence file, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Databricks – Spark UDFs

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL.

What is a UDF?

User-Defined Functions (aka UDF) is a feature of Spark SQL to define new Column-based functions that extend the vocabulary of Spark SQL’s DSL for transforming Datasets.

…

05: Docker Tutorial – Client & Server on separate containers & networking

In this tutorial, let’s run a Java socket client & a server in two separate Docker containers. Both the client & server are continuously run, and networked by creating a Docker network. MyClient.java

package com.myapp;

import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.Socket;
import java.util.Scanner;

public class MyClient {
    public static void main(String[] args) throws IOException {
        if (args.length != 1) {
            System.err
                    .println("Pass the server IP as the sole command line argument");
            return;
        }
        try (Socket socket = new Socket(args[0], 8888);
                Scanner sc = new Scanner(System.in);
                Scanner in = new Scanner(socket.getInputStream())) {

            OutputStream os = socket.getOutputStream();
            PrintWriter out = new PrintWriter(os);

            while (true) {
                while(sc.hasNextLine()) {
                    String msg = sc.nextLine();
                    System.out.println("Client sending: " + msg);
                    out.println(msg);
                    out.flush();

                    System.out.println("Server response: " + in.nextLine());
                }
            }
        }
    }
}

package com.myapp;

import java.io.IOException;

import java.io.OutputStream;

import java.io.PrintWriter;

import java.net.Socket;

import java.util.Scanner;

public class MyClient {

public static void main(String[] args) throws IOException {

if (args.length != 1) {

System.err

.println("Pass the server IP as the sole command line argument");

return;

}

try (Socket socket = new Socket(args[0], 8888);

Scanner sc = new Scanner(System.in);

Scanner in = new Scanner(socket.getInputStream())) {

OutputStream os = socket.getOutputStream();

PrintWriter out = new PrintWriter(os);

while (true) {

while(sc.hasNextLine()) {

String msg = sc.nextLine();

System.out.println("Client sending: " + msg);

out.println(msg);

out.flush();

System.out.println("Server response: " + in.nextLine());

}

MyServer.java

package com.myapp;

import java.io.IOException;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.Scanner;

public class MyServer {

    public static void main(String[] args) throws IOException {
        try (ServerSocket listener = new ServerSocket(8888)) {
            System.out.println("The server is running...");
            while (true) {
                try (Socket socket = listener.accept();
                        Scanner in = new Scanner(socket.getInputStream())) {
                    PrintWriter out = new PrintWriter(socket.getOutputStream(),
                            true);
                    while (in.hasNextLine()) {
                        String msg = in.nextLine();
                        System.out.println("Server received: " + msg);
                        out.println(msg.toUpperCase());
                    }
                }
            }
        }
    }

}

package com.myapp;

import java.io.IOException;

import java.io.PrintWriter;

import java.net.ServerSocket;

import java.net.Socket;

import java.util.Scanner;

public class MyServer {

public static void main(String[] args) throws IOException {

try (ServerSocket listener = new ServerSocket(8888)) {

System.out.println("The server is running...");

while (true) {

try (Socket socket = listener.accept();

Scanner in = new Scanner(socket.getInputStream())) {

PrintWriter out = new PrintWriter(socket.getOutputStream(),

true);

while (in.hasNextLine()) {

String msg = in.nextLine();

System.out.println("Server received: " + msg);

out.println(msg.toUpperCase());

}

Dockerfile.client

FROM alpine:3.6

MAINTAINER java-success.com
 
WORKDIR /app

USER root

# install required packages
RUN apk add --no-cache openssh openssl openjdk8 rsync bash procps nss maven

# set JAVA_HOME
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk
ENV PATH $PATH:$JAVA_HOME/bin

COPY src/main/java/com/myapp/MyClient.java com/myapp/MyClient.java

RUN javac com/myapp/MyClient.java

EXPOSE 8888
ENTRYPOINT ["java"]
CMD ["com.myapp.MyClient", "myserver"]

FROM alpine:3.6

MAINTAINER java-success.com

WORKDIR /app

USER root

# install required packages

RUN apk add --no-cache openssh openssl openjdk8 rsync bash procps nss maven

# set JAVA_HOME

ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk

ENV PATH $PATH:$JAVA_HOME/bin

COPY src/main/java/com/myapp/MyClient.java com/myapp/MyClient.java

RUN javac com/myapp/MyClient.java

EXPOSE 8888

ENTRYPOINT ["java"]

CMD ["com.myapp.MyClient", "myserver"]

Note that “ … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Drools with rules in an excel spreadsheet – Global variables & variable capture

This extends 04: Drools with rules in an excel spreadsheet – Data validation .

Step 1: Enhance the Trade.java POJO with “actualValues” field and “getFieldValue(String fieldNames)” method.


package com.mytutorial.pojo;

import java.lang.reflect.Field;
import org.apache.commons.lang.StringUtils;

public class Trade {
    private String tradeId;
    private String tradeDate;
    private String symbol;
    private String price;
    private String volume;
    private String totalPrice;
    private String buyerId;
    private String sellerId;

    private String actualValues;

    public Trade(String tradeId, String tradeDate, String symbol, String price, String volume, String totalPrice,
            String buyerId, String sellerId) {
        super();
        this.tradeId = tradeId;
        this.tradeDate = tradeDate;
        this.symbol = symbol;
        this.price = price;
        this.volume = volume;
        this.totalPrice = totalPrice;
        this.buyerId = buyerId;
        this.sellerId = sellerId;
    }

    public String getTradeId() {
        return tradeId;
    }

    public void setTradeId(String tradeId) {
        this.tradeId = tradeId;
    }

    public String getTradeDate() {
        return tradeDate;
    }

    public void setTradeDate(String tradeDate) {
        this.tradeDate = tradeDate;
    }

    public String getSymbol() {
        return symbol;
    }

    public void setSymbol(String symbol) {
        this.symbol = symbol;
    }

    public String getPrice() {
        return price;
    }

    public void setPrice(String price) {
        this.price = price;
    }

    public String getVolume() {
        return volume;
    }

    public void setVolume(String volume) {
        this.volume = volume;
    }

    public String getTotalPrice() {
        return totalPrice;
    }

    public void setTotalPrice(String totalPrice) {
        this.totalPrice = totalPrice;
    }

    public String getBuyerId() {
        return buyerId;
    }

    public void setBuyerId(String buyerId) {
        this.buyerId = buyerId;
    }

    public String getSellerId() {
        return sellerId;
    }

    public void setSellerId(String sellerId) {
        this.sellerId = sellerId;
    }

    public String getActualValues() {
        return actualValues;
    }

    public void setActualValues(String actualValues) {
        this.actualValues = actualValues;
    }

    //reflection to get field values
    public String getFieldValue(String fieldNames) throws NoSuchFieldException, SecurityException, IllegalArgumentException, IllegalAccessException {

        if (StringUtils.isEmpty(fieldNames)) {
            throw new IllegalArgumentException();
        }

        String[] splits = fieldNames.split("\\,");
        StringBuilder sb = new StringBuilder();

        for (int i = 0; i < splits.length; i++) {
            Field field = this.getClass().getDeclaredField(splits[i].trim());
            field.setAccessible(true);
            sb.append((String) field.get(this));

            if (i < splits.length - 1) {
                sb.append("|");
            }
        }

        return sb.toString();
    }

    @Override
    public String toString() {
        return "Trade [tradeId=" + tradeId + ", tradeDate=" + tradeDate + ", symbol=" + symbol + ", price=" + price
                + ", volume=" + volume + ", totalPrice=" + totalPrice + ", buyerId=" + buyerId + ", sellerId="
                + sellerId + ", actualValues=" + actualValues + "]";
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

package com.mytutorial.pojo;

import java.lang.reflect.Field;

import org.apache.commons.lang.StringUtils;

public class Trade {

private String tradeId;

private String tradeDate;

private String symbol;

private String price;

private String volume;

private String totalPrice;

private String buyerId;

private String sellerId;

private String actualValues;

public Trade(String tradeId, String tradeDate, String symbol, String price, String volume, String totalPrice,

String buyerId, String sellerId) {

super();

this.tradeId = tradeId;

this.tradeDate = tradeDate;

this.symbol = symbol;

this.price = price;

this.volume = volume;

this.totalPrice = totalPrice;

this.buyerId = buyerId;

this.sellerId = sellerId;

}

public String getTradeId() {

return tradeId;

}

public void setTradeId(String tradeId) {

this.tradeId = tradeId;

}

public String getTradeDate() {

return tradeDate;

}

public void setTradeDate(String tradeDate) {

this.tradeDate = tradeDate;

}

public String getSymbol() {

return symbol;

}

public void setSymbol(String symbol) {

this.symbol = symbol;

}

public String getPrice() {

return price;

}

public void setPrice(String price) {

this.price = price;

}

public String getVolume() {

return volume;

}

public void setVolume(String volume) {

this.volume = volume;

}

public String getTotalPrice() {

return totalPrice;

}

public void setTotalPrice(String totalPrice) {

this.totalPrice = totalPrice;

}

public String getBuyerId() {

return buyerId;

}

public void setBuyerId(String buyerId) {

this.buyerId = buyerId;

}

public String getSellerId() {

return sellerId;

}

public void setSellerId(String sellerId) {

this.sellerId = sellerId;

}

public String getActualValues() {

return actualValues;

}

public void setActualValues(String actualValues) {

this.actualValues = actualValues;

}

//reflection to get field values

public String getFieldValue(String fieldNames) throws NoSuchFieldException, SecurityException, IllegalArgumentException, IllegalAccessException {

if (StringUtils.isEmpty(fieldNames)) {

throw new IllegalArgumentException();

}

String[] splits = fieldNames.split("\\,");

StringBuilder sb = new StringBuilder();

for (int i = 0; i < splits.length; i++) {

Field field = this.getClass().getDeclaredField(splits[i].trim());

field.setAccessible(true);

sb.append((String) field.get(this));

if (i < splits.length - 1) {

sb.append("|");

}

return sb.toString();

}

@Override

public String toString() {

return "Trade [tradeId=" + tradeId + ", tradeDate=" + tradeDate + ", symbol=" + symbol + ", price=" + price

+ ", volume=" + volume + ", totalPrice=" + totalPrice + ", buyerId=" + buyerId + ", sellerId="

+ sellerId + ", actualValues=" + actualValues + "]";

}

Step 2: trades-rules.xls with “

…

05: Kubernetes useful commands with kubectl for development & debugging

This extends Install Kubernetes (i.e. minikube) on Mac.

Step 1: Start the Docker.app on Mac, and then start the minikube cluster from a terminal.


~]$ minikube start

~]$ minikube start

Step 2: Create a pod.yaml manifest file to create a new pod that runs nginx image.

…

05: Learn Hive to write to and read from AVRO & Parquet files by examples

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc. What is Apache Hive? Hive allows SQL developers to write Hive Query Language (HQL) statements that are similar to standard SQL statements. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Setting up & getting started with Spark local mode with Sbt & Scala

This extends Setting up & getting started with sbt. Spark 2.3.2 requires Scala 2.11.x. You need to use the right Scala version. Step 1: On Eclipse set the Scala compiler to 2.11. Right mouse click on the project “sbt-tutorial” … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Spark on Zeppelin – semi-structured log file

This tutorial extends the series: Spark on Apache Zeppelin Tutorials. Step 1: Pull apache/zeppelin image from the docker hub, and build the image with the following command.

$ docker pull apache/zeppelin:0.7.3

1 2	$ docker pull apache/zeppelin:0.7.3

“docker images” will show the image that was created. Step 2: Run the above image to create a container with the following...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: Spark SQL & CSV with DataFrame Tutorial

Step 1: Create a simple maven project.


mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-spark

mvn archetype:generate -B -DgroupId=com.mytutorial -DartifactId=simple-spark

Step 2: Import the “simple-spark” maven project into eclipse or IDE of your choice. Step 3: Modify the pom.xml file include 1) relevant Spark libraries 2) The shade plugin to create a single jar (i.e. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05: strconv – How will you convert a string to other data types & other data types to string in Golang?

Q01. How will you convert a string to other data types & other data types to string in Golang?
A01. Package strconv implements conversions to and from string representations of basic data types. The most common numeric conversions are Atoi (string to int) and Itoa (int to string).

…

05. Spring Cloud routing with Netflix’s Zuul

Q. What is Netflix’s Zuul usecase in Micro Services Architecture? A. In a typical micro service architecture you have many micro services running on different hosts and ports. The challenge with this type of architecture is how different clients like browser based Web Applications, Mobile based apps, … Read more...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

05. ⏯ Getting started with Eclipse & Maven beginner video tutorial

« 1 … 8 9 10 11 12 13 14 15 16 … 30 »

Blog Archives

05: Convert XML file To an Avro File with avro-maven-plugin & Apache Spark

05: Create or append a Sequence file to HDFS – Hadoop API tutorial

05: Databricks – Spark UDFs

05: Docker Tutorial – Client & Server on separate containers & networking

05: Drools with rules in an excel spreadsheet – Global variables & variable capture

05: Kubernetes useful commands with kubectl for development & debugging

05: Learn Hive to write to and read from AVRO & Parquet files by examples

05: Setting up & getting started with Spark local mode with Sbt & Scala

05: Spark on Zeppelin – semi-structured log file

05: Spark SQL & CSV with DataFrame Tutorial

05: strconv – How will you convert a string to other data types & other data types to string in Golang?

05. Spring Cloud routing with Netflix’s Zuul

05. ⏯ Getting started with Eclipse & Maven beginner video tutorial

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer