04: Create new or append to an existing AVRO file tutorial

Posted on August 8, 2016 by

This extends Create or append a file to HDFS – Hadoop API tutorial to write an AVRO file to HDFS. Step 1: Include the AVRO library files in the pom.xml file.


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mytutorial</groupId>
    <artifactId>simple-hadoop</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>simple-hadoop</name>
    <url>http://maven.apache.org</url>

    <properties>
        <hadoop.version>2.7.2</hadoop.version>
        <avro.version>1.8.1</avro.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <!-- Hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <!-- Required for the  FsInput class  -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-mapred</artifactId>
            <version>${avro.version}</version>
            <classifier>hadoop2</classifier>
        </dependency>

    </dependencies>
    <build>
        <plugins>
            <!-- build an uber jar with hadoop dependencies included -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<groupId>com.mytutorial</groupId>

<artifactId>simple-hadoop</artifactId>

<version>1.0-SNAPSHOT</version>

<name>simple-hadoop</name>

<url>http://maven.apache.org</url>

<hadoop.version>2.7.2</hadoop.version>

<avro.version>1.8.1</avro.version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-hdfs</artifactId>

<version>${hadoop.version}</version>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>

<version>${hadoop.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<version>${avro.version}</version>

</dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro-mapred</artifactId>

<version>${avro.version}</version>

<classifier>hadoop2</classifier>

</dependency>

</dependencies>

<build>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>

Step 2: The AVRO files are schema based….

Members Only Content

Why & What are the benefits

Arulkumaran

Mechanical Engineer to self-taught Java engineer within 2 years & a freelancer within 3 years. Freelancing since 2003. Preparation empowered me to attend 190+ job interviews & choose from 150+ job offers with sought-after contract rates. Author of the book “Java/J2EE job interview companion“, which sold 35K+ copies & superseded by this site with1900+ registered users. Amazon.com profile | Reviews | LinkedIn | LinkedIn Group | YouTube

Contact us: java-interview@hotmail.com

🎯 Why java-success.com?

🎯 What are the benefits of Q&As approach?

Learn by categories such as FAQs – Core Java, Key Area – Low Latency, Core Java – Java 8, JEE – Microservices, Big Data – NoSQL, Architecture – Distributed, Big Data – Spark, etc. Some posts belong to multiple categories.

Going places as a Java/Big Data Engineer?

🎯 Why & How to choose from 2-6 job offers?

🎯 How to increase your earning potential?

🎯 What can you do as a techie to open more doors?

🎯 Freelancing with confidence to find work in any job market

🎯 100+ Resume phrases & resume writing for more job interview calls

The contents in this Java-Success are copyrighted and from EmpoweringTech pty ltd. The EmpoweringTech pty ltd has the right to correct or enhance the current content without any prior notice. These are general advice only, and one needs to take his/her own circumstances into consideration. The EmpoweringTech pty ltd will not be held liable for any damages caused or alleged to be caused either directly or indirectly by these materials and resources. Any trademarked names or labels used in this blog remain the property of their respective trademark owners. Links to external sites do not imply endorsement of the linked-to sites. Privacy Policy

Responsive WordPress Theme powered by CyberChimps

Top

04: Create new or append to an existing AVRO file tutorial

Members Only Content

800+ Java & Big Data Q&As Menu

Java Q&As Courses

Don’t be overwhelmed….

Disclaimer