01: Learn Hadoop API by examples in Java

Posted on November 2, 2017 by

These Hadoop tutorials assume that you have installed Cloudera QuickStart, which has the Hadoop eco system like HDFS, Spark, Hive, HBase, YARN, etc.

What is Hadoop & HDFS? Hadoop based data hub architecture & basics | Hadoop eco system basics Q&As style.

List files in HDFS

The following Java code uses the Hadoop API to list files in HDFS.

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
/**
 * Similar to:   hdfs dfs -ls hdfs://xx.xxx.xxx.xx:8020/user/someuser/test
 *
 */
public class HadoopSimple {
    public static void main(String[] args) throws IOException {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://ip-address:8020/");  
        String hdfsPath = "/user/someuser/some_folder_path";
        
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path(hdfsPath);
        RemoteIterator<LocatedFileStatus> files = fs.listFiles(path, true); // true means recursive
        
        while (files.hasNext()) {
            LocatedFileStatus file = files.next();
            System.out.println(file);
        }
        
        fs.close();
    }
}

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.LocatedFileStatus;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.fs.RemoteIterator;

/**

* Similar to: hdfs dfs -ls hdfs://xx.xxx.xxx.xx:8020/user/someuser/test

public class HadoopSimple {

public static void main(String[] args) throws IOException {

Configuration conf = new Configuration();

conf.set("fs.defaultFS", "hdfs://ip-address:8020/");

String hdfsPath = "/user/someuser/some_folder_path";

FileSystem fs = FileSystem.get(conf);

Path path = new Path(hdfsPath);

RemoteIterator<LocatedFileStatus> files = fs.listFiles(path, true); // true means recursive

while (files.hasNext()) {

LocatedFileStatus file = files.next();

System.out.println(file);

}

fs.close();

}

You can use the handle on “fs” to perform operations on a file like

fs.rename(src, dst);                      //rename a file
fs.copyFromLocalFile(src, dst); 
fs.delete(f, recursive);
FSDataOutputStream os = fs.append(path);  //apend to an existing file

fs.rename(src, dst); //rename a file

fs.copyFromLocalFile(src, dst);

fs.delete(f, recursive);

FSDataOutputStream os = fs.append(path); //apend to an existing file

List files in a local Unix file system

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
public class HadoopSimple {
    public static void main(String[] args) throws IOException {
        Configuration conf = new Configuration();
        
        String hdfsPath = "file:///home/user/path_to_a_folder";
        
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path(hdfsPath);
        RemoteIterator<LocatedFileStatus> files = fs.listFiles(path, true); // true means recursive
        
        while (files.hasNext()) {
            LocatedFileStatus file = files.next();
            System.out.println(file);
        }
        
        fs.close();
    }
}

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.LocatedFileStatus;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.fs.RemoteIterator;

public class HadoopSimple {

public static void main(String[] args) throws IOException {

Configuration conf = new Configuration();

String hdfsPath = "file:///home/user/path_to_a_folder";

FileSystem fs = FileSystem.get(conf);

Path path = new Path(hdfsPath);

RemoteIterator<LocatedFileStatus> files = fs.listFiles(path, true); // true means recursive

while (files.hasNext()) {

LocatedFileStatus file = files.next();

System.out.println(file);

}

fs.close();

}

Append contents to a file in HDFS

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
public class HadoopSimple {
    public static void main(String[] args) throws IOException {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://ip-address:8020/");  
        String hdfsPath = "/user/someuser/test/";
        
        FileSystem fs = FileSystem.get(conf);
        Path path = new Path(hdfsPath);
        RemoteIterator<LocatedFileStatus> files = fs.listFiles(path, true); // true means recursive
        
        while (files.hasNext()) {
            LocatedFileStatus file = files.next();
            System.out.println(file);
            
            if(file.isFile() && file.getPath().getName().equalsIgnoreCase("test.txt")) {
                FSDataOutputStream os = fs.append(file.getPath());
                os.write("Some test\n".getBytes());
                os.close();
            }
        }
        
        
        fs.close();
    }
}

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FSDataOutputStream;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.LocatedFileStatus;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.fs.RemoteIterator;

public class HadoopSimple {

public static void main(String[] args) throws IOException {

Configuration conf = new Configuration();

conf.set("fs.defaultFS", "hdfs://ip-address:8020/");

String hdfsPath = "/user/someuser/test/";

FileSystem fs = FileSystem.get(conf);

Path path = new Path(hdfsPath);

RemoteIterator<LocatedFileStatus> files = fs.listFiles(path, true); // true means recursive

while (files.hasNext()) {

LocatedFileStatus file = files.next();

System.out.println(file);

if(file.isFile() && file.getPath().getName().equalsIgnoreCase("test.txt")) {

FSDataOutputStream os = fs.append(file.getPath());

os.write("Some test\n".getBytes());

os.close();

}

fs.close();

}

The above Java code is equivalent to the following command-line.

echo -e "some text" | hdfs dfs -appendToFile - hdfs://ip-address:8020//user/someuser/test/test.txt

echo -e "some text" | hdfs dfs -appendToFile - hdfs://ip-address:8020//user/someuser/test/test.txt

How do you find the name node URI?

Option 1: On the edge node Via /etc/hadoop/conf/core-site.xml.

 <property>
    <name>fs.defaultFS</name>
    <value>hdfs://<ip-address>:8020</value>
  </property>

<name>fs.defaultFS</name>

</property>

Option 2: If you are on Cloudera, go to Cloudera Manager, and click on “HDFS“, and then select NameNode to get its configuration details including the ip address.

Option 3: If you are on Cloudera, go to Cloudera Manager, and click on “HDFS“, and then click on the Actions drop down and click the “Download Client Configuration“, which will have all the config files including the core-site.xml as a zip file.

What libraries do you need in the classpath?

The Hadoop examples shown in this post must have the relevant JARs shown in the pom.xml file. The spark-core will transitively bring in the Hadoop libraries.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-spark</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-spark</name>
    <url>http://maven.apache.org</url>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <hadoop.version>2.7.2</hadoop.version>
        <scala.version>2.10.4</scala.version>
        <scala.binary.version>2.10</scala.binary.version>
    </properties>
    
    <repositories>
        <repository>
            <id>central</id>
            <name>Maven Central</name>
            <url>http://repo1.maven.org/maven2/</url>
        </repository>
        <repository>
            <id>cloudera</id>
            <url>http://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <!-- Spark libraries -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-spark</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-spark</name>

<url>http://maven.apache.org</url>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

<hadoop.version>2.7.2</hadoop.version>

<scala.version>2.10.4</scala.version>

<scala.binary.version>2.10</scala.binary.version>

</properties>

<id>central</id>

<name>Maven Central</name>

<url>http://repo1.maven.org/maven2/</url>

</repository>

<id>cloudera</id>

<url>http://repository.cloudera.com/artifactory/cloudera-repos/</url>

</repository>

</repositories>

<groupId>org.scala-lang</groupId>

<artifactId>scala-library</artifactId>

<version>${scala.version}</version>

</dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_${scala.binary.version}</artifactId>

<version>${spark.version}</version>

<scope>provided</scope>

</dependency>

</dependencies>

</project>

Learn about the Hadoop eco system with examples

Learn HDFS, Spark, HBase, Hive, AVro, Parquet by examples.

About
Latest Posts

Arulkumaran Kumaraswamipillai

Mechanical Engineer to self-taught Java engineer within 2 years & a freelancer within 3 years. Freelancing since 2003. Preparation empowered me to attend 190+ job interviews & choose from 150+ job offers with sought-after contract rates. Author of the book "Java/J2EE job interview companion", which sold 35K+ copies & superseded by this site with 2,050+ registered users. Amazon.com profile | Reviews | LinkedIn | LinkedIn Group | YouTube

Contact us: java-interview@hotmail.com

Latest posts by Arulkumaran Kumaraswamipillai (see all)

300+ Big Data Interview Questions & Answers - July 11, 2022
800+ Java interview questions & answers - July 11, 2022
What does this Unix code snippet do? - July 2, 2022

The contents in this Java-Success are copyrighted and from EmpoweringTech pty ltd. The EmpoweringTech pty ltd has the right to correct or enhance the current content without any prior notice. These are general advice only, and one needs to take his/her own circumstances into consideration. The EmpoweringTech pty ltd will not be held liable for any damages caused or alleged to be caused either directly or indirectly by these materials and resources. Any trademarked names or labels used in this blog remain the property of their respective trademark owners. Links to external sites do not imply endorsement of the linked-to sites. Privacy Policy

Responsive WordPress Theme powered by CyberChimps

Top

01: Learn Hadoop API by examples in Java

List files in HDFS

List files in a local Unix file system

Append contents to a file in HDFS

How do you find the name node URI?

What libraries do you need in the classpath?

Learn about the Hadoop eco system with examples

Java Developer Interview Q&As

Java Architect Interview Q&As

Big Data Interview Q&As

800+ Java Interview Q&As

Disclaimer