Tutorials Archives | Page 17 of 30 | 800+ Java & Big Data Interview Q&As

Blog Archives

« 1 … 13 14 15 16 17 18 19 20 21 … 30 »

1. HBase Tutorial Getting Started

In standalone mode, HBase does not use HDFS — it uses the local filesystem instead — and it runs all HBase daemons and a local ZooKeeper all up in the same JVM. Zookeeper binds to a well known port so clients may talk to HBase.

Step 1: Download HBase from “http://hbase.apache.org/”.

…

10: Databricks – Spark ML – Linear Regression

Prerequisite: Extends Databricks getting started – Spark, Shell, SQL. You can try these tutorials in Scala using Databricks Notebook. There are Scala tutorials covered in Spark using Scala on Zeppelin Notebook. Problem statement: Predict the land prices by land area in square feet based on a given set of known...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

10: Getting started with MySQL database beginner tutorial

All Java developers not only must know “SQL”, but also a database is required for “JDBC tutorials”, “Hibernate Tutorials”, and so on. Any decent Java application will require a database. So, your self-taught Java projects must perform CRUD (i.e. Create Read, Update and Delete) operations on the database.

…

10: Solving AlreadyBeingCreatedException & LeaseExpiredException thrown from your Spark jobs

What is wrong with the following Spark code snippet?


resultsRdd.foreachPartition(results -> {
    
    final Configuration hc = new Configuration();
    final FileSystem hdfs = FileSystem.get(hc);
    
    String prefix = UUID.randomUUID().toString();
    long nowInMillis = DateTime.now().toInstant().getMillis();
    
    String strOutFile = "/path/to/output/file.csv";

    while (results.hasNext()) {
        Tuple2<String, String> x = results.next();

        if (x._2 != null) {

            String decodedPayload = x._2;
            final Path outFilePath = new Path(strOutFile);
            
            FSDataOutputStream ouputOs = null;
            if (hdfs.exists(outFilePath)) {
                ouputOs = hdfs.append(outFilePath);
            } else {
                ouputOs = hdfs.create(outFilePath);
            }
            ouputOs.writeBytes(decodedPayload);
            ouputOs.close();
        } 
    }
});

resultsRdd.foreachPartition(results -> {

final Configuration hc = new Configuration();

final FileSystem hdfs = FileSystem.get(hc);

String prefix = UUID.randomUUID().toString();

long nowInMillis = DateTime.now().toInstant().getMillis();

String strOutFile = "/path/to/output/file.csv";

while (results.hasNext()) {

Tuple2<String, String> x = results.next();

if (x._2 != null) {

String decodedPayload = x._2;

final Path outFilePath = new Path(strOutFile);

FSDataOutputStream ouputOs = null;

if (hdfs.exists(outFilePath)) {

ouputOs = hdfs.append(outFilePath);

} else {

ouputOs = hdfs.create(outFilePath);

}

ouputOs.writeBytes(decodedPayload);

ouputOs.close();

}

});

You are likely to get AlreadyBeingCreatedException & LeaseExpiredException thrown as multiple executors try to either create or append to the same file in HDFS in parallel. HDFS allows only one writer. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

10: Spark on Zeppelin – union, udf and explode

Pre-requisite: Docker is installed on your machine for Mac OS X (E.g. $ brew cask install docker) or Windows 10. Docker interview Q&As. This extends setting up Apache Zeppelin Notebook. Step 1: Pull this from the docker hub, … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

10: Spark RDDs to HBase & HBase to Spark RDDs

Step 1: pom.xml with library dependencies. It is important to note that 1) “https://repository.cloudera.com/artifactory/cloudera-repos/” is added as the “Cloudera Maven Repository” and 2) hbase-spark dependency is used for writing to HBase from Spark RDDs & … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

10. Setting up & getting started with Hive

Hive is popular SQL based tool to process Big Data. It stores & retrieves data to/from HDFS.

Prerequisite: Xcode & Hadoop are installed as outlined in Setting up & getting started with Hadoop on Mac

1. Install Hive

Let’s install hive-2.3.4 with hadoop 2.7.7.

…

11: Databricks – Spark ML – Multivariate Linear Regression

Prerequisite: Extends Databricks – Spark ML – Linear Regression. Problem statement: Predict the house prices by land area in square feet, no of bedrooms, and how old the house is?, which is the age. … Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11: Docker Tutorial: Hadoop Big Data CLIs on Cloudera quickstart

There are a number of CLIs that you can run from the edge node, which is the gateway node to the Hadoop cluster consisting of master & slave (aka worker) nodes. Let’s look at the different CLIs (i.e Command Line Interfaces)

Most of the CLIs listed below are in /usr/bin

[root@quickstart /]# ls -ltr  /usr/bin | grep 'hdfs\|impala-shell\|beeline\|hive\|spark-shell\|pyspark\|spark-submit'
-rwxr-xr-x   1 root root      8144 Sep 22  2015 db_archive
-rwxr-xr-x   1 root root       328 Mar 23  2016 hiveserver2
-rwxr-xr-x   1 root root       321 Mar 23  2016 hive
-rwxr-xr-x   1 root root       324 Mar 23  2016 beeline
-rwxr-xr-x   1 root root       504 Mar 23  2016 sqoop-create-hive-table
-rwxr-xr-x   1 root root   1138056 Mar 23  2016 git-upload-archive
-rwxr-xr-x   1 root root       141 Mar 23  2016 spark-submit
-rwxr-xr-x   1 root root       140 Mar 23  2016 spark-shell
-rwxr-xr-x   1 root root       185 Mar 23  2016 pyspark
-rwxr-xr-x   1 root root       192 Mar 23  2016 hdfs
-rwxr-xr-x   1 root root      1856 Mar 23  2016 impala-shell

-rwxr-xr-x 1 root root 8144 Sep 22 2015 db_archive

-rwxr-xr-x 1 root root 328 Mar 23 2016 hiveserver2

-rwxr-xr-x 1 root root 321 Mar 23 2016 hive

-rwxr-xr-x 1 root root 324 Mar 23 2016 beeline

-rwxr-xr-x 1 root root 504 Mar 23 2016 sqoop-create-hive-table

-rwxr-xr-x 1 root root 1138056 Mar 23 2016 git-upload-archive

-rwxr-xr-x 1 root root 141 Mar 23 2016 spark-submit

-rwxr-xr-x 1 root root 140 Mar 23 2016 spark-shell

-rwxr-xr-x 1 root root 185 Mar 23 2016 pyspark

-rwxr-xr-x 1 root root 192 Mar 23 2016 hdfs

-rwxr-xr-x 1 root root 1856 Mar 23 2016 impala-shell

hdfs CLI

The following command will give you the commands you can use with “hdfs”

…

11: Spark on Zeppelin – Dataframe groupBy, collect_list, explode & window

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11: Spark streaming with “textFileStream” simple tutorial

Using Spark streaming data can be ingested from many sources like Kafka, Flume, HDFS, Unix/Windows File system, etc. In this example, let’s run the Spark in a local mode to ingest data from a Unix file system. Step 1: The pom.xml file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-spark</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-spark</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <hadoop.version>2.7.2</hadoop.version>
        <avro.version>1.8.1</avro.version>
        <spark.version>1.6.0-cdh5.8.3</spark.version>
        <scala.version>2.10.4</scala.version>
        <scala.binary.version>2.10</scala.binary.version>
    </properties>

    <repositories>
        <repository>
            <id>central</id>
            <name>Maven Central</name>
            <url>http://repo1.maven.org/maven2/</url>
        </repository>
        <repository>
            <id>cloudera</id>
            <url>http://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- Spark libraries -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
    
    <build>
        <plugins>
            <!-- build an uber jar with hadoop dependencies included -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-spark</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-spark</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

<hadoop.version>2.7.2</hadoop.version>

<avro.version>1.8.1</avro.version>

<spark.version>1.6.0-cdh5.8.3</spark.version>

<scala.version>2.10.4</scala.version>

<scala.binary.version>2.10</scala.binary.version>

</properties>

<id>central</id>

<name>Maven Central</name>

<url>http://repo1.maven.org/maven2/</url>

</repository>

<id>cloudera</id>

<url>http://repository.cloudera.com/artifactory/cloudera-repos/</url>

</repository>

</repositories>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

</dependency>

<groupId>org.scala-lang</groupId>

<artifactId>scala-library</artifactId>

<version>${scala.version}</version>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-sql_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

</dependencies>

<build>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>

Using textFileStream(..) textFileStream watches a directory for...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11. What are part- files in Hadoop & 6 ways to merge them

What are the part-xxxx files generated by Hadoop? When you invoke rdd.saveAsTextFile(…) or rdd.saveAsNewAPIHadoopFile(…) from Spark you will get part- files. When you do “INSERT INTO” command in Hive, the execution results in multiple part files in HDFS. You will have one part-xxxx file per partition in the RDD you...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

11A: Databricks – Spark ML – Pandas Dataframe & Matplotlib

Prerequisite: Extends 11: Databricks – Spark ML – Multivariate Linear Regression. How do you convert Pyspark dataframe to Pndas Dataframe? df.toPandas() converts Pyspark Dataframe to Pandas Dataframe.

from pyspark.sql.types import *

input_data = (
 (400,3,5,480000.00),
 (500,3,20,400000.00,), 
 (600,5,None,850000.00),
 (700,4,10,770000.00),
 (800,4,15,800000.00),
 (1000,6,20,1100000.00),
 (1200,7,2,1600000.00) 
)
 
data_schema = [StructField('land_area', IntegerType(), False), StructField('bedrooms', IntegerType(), True), StructField('age', IntegerType(), True), StructField('house_price',DoubleType(),False)]
final_struc = StructType(fields=data_schema)
 
df = spark.sparkContext.parallelize(input_data)
 
df_input= spark.createDataFrame(df, final_struc)
 
pandas_df = df_input.toPandas()

print(pandas_df)

from pyspark.sql.types import *

input_data = (

(400,3,5,480000.00),

(500,3,20,400000.00,),

(600,5,None,850000.00),

(700,4,10,770000.00),

(800,4,15,800000.00),

(1000,6,20,1100000.00),

(1200,7,2,1600000.00)

)

data_schema = [StructField('land_area', IntegerType(), False), StructField('bedrooms', IntegerType(), True), StructField('age', IntegerType(), True), StructField('house_price',DoubleType(),False)]

final_struc = StructType(fields=data_schema)

df = spark.sparkContext.parallelize(input_data)

df_input= spark.createDataFrame(df, final_struc)

pandas_df = df_input.toPandas()

print(pandas_df)

Output:

df_input:pyspark.sql.dataframe.DataFrame
land_area:integer
bedrooms:integer
age:integer
house_price:double

   land_area  bedrooms   age  house_price
0        400         3   5.0     480000.0
1        500         3  20.0     400000.0
2        600         5   NaN     850000.0
3        700         4  10.0     770000.0
4        800         4  15.0     800000.0
5       1000         6  20.0    1100000.0
6       1200         7   2.0    1600000.0

df_input:pyspark.sql.dataframe.DataFrame

land_area:integer

bedrooms:integer

age:integer

house_price:double

land_area bedrooms age house_price

0 400 3 5.0 480000.0

1 500 3 20.0 400000.0

2 600 5 NaN 850000.0

3 700 4 10.0 770000.0

4 800 4 15.0 800000.0

5 1000 6 20.0 1100000.0

6 1200 7 2.0 1600000.0

How to perform statistical data exploration?… Read more ›...

This content is for 30-Day-Full-Access, 100-Day-Full-Access, 200-Day-Full-Access, and 365-Day-Full-Access members only. Register 80+ Free Java FAQs 80+ Free Big Data FAQs

Already a member? Log in here

« 1 … 13 14 15 16 17 18 19 20 21 … 30 »

Blog Archives

1. HBase Tutorial Getting Started

10: Databricks – Spark ML – Linear Regression

10: Getting started with MySQL database beginner tutorial

10: Solving AlreadyBeingCreatedException & LeaseExpiredException thrown from your Spark jobs

10: Spark on Zeppelin – union, udf and explode

10: Spark RDDs to HBase & HBase to Spark RDDs

10. Setting up & getting started with Hive

11: Databricks – Spark ML – Multivariate Linear Regression

11: Docker Tutorial: Hadoop Big Data CLIs on Cloudera quickstart

11: Spark on Zeppelin – Dataframe groupBy, collect_list, explode & window

11: Spark streaming with “textFileStream” simple tutorial

11. What are part- files in Hadoop & 6 ways to merge them

11A: Databricks – Spark ML – Pandas Dataframe & Matplotlib

300+ Java Interview Questions - FAQs

10+ Companion Techs Interview Q&As

300+ Big Data Interview Questions - FAQs

16+ Java Tech Key Areas Interview Q&As

Your Career Tips to go places

Tutorials on Java & Big Data

300+ Java Interview Questions Answered

Disclaimer