Java String & Array limitations and OutOfMemoryError

Posted on November 25, 2016 by

Q1. What is the maximum possible length of a Java String & how much heap space do you need to store the maximum possible String object?
A1. A Java String internally uses a char array (i.e. char[]) and the indices of an array is an integer. The maximum value of an integer is Integer.MAX_VALUE, which is 2^31 – 1 (or approximately 2 billion). So, you can store a file up to 2 GB in size as a String, and for which you need at least 4 GB memory to store as each char is 2 bytes in Java, plus additional ~4 GB memory for creating the String object, so, in total around 8GB of heap space. So, it is not a good design to read a large file ~2GB into a string for further processing.

Q2. What will be the output if you try to read a 2 GB file into a StringBuilder?

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
public class ReadLargeFileUsingStringBuilder {
	public static void main(String[] args) throws IOException {
		File file = new File("c:/temp/large-2gb-file.xml"); // ~2 GB
		FileInputStream fis = new FileInputStream(file);
		InputStreamReader isr = new InputStreamReader(fis);
		BufferedReader br = new BufferedReader(isr);
		
		String line = null;		
		StringBuilder sb = new StringBuilder();
		
		while((line = br.readLine()) != null) {
			sb.append(line);
		}
		
		br.close();
	}
}

import java.io.BufferedReader;

import java.io.File;

import java.io.FileInputStream;

import java.io.IOException;

import java.io.InputStreamReader;

public class ReadLargeFileUsingStringBuilder {

public static void main(String[] args) throws IOException {

File file = new File("c:/temp/large-2gb-file.xml"); // ~2 GB

FileInputStream fis = new FileInputStream(file);

InputStreamReader isr = new InputStreamReader(fis);

BufferedReader br = new BufferedReader(isr);

String line = null;

StringBuilder sb = new StringBuilder();

while((line = br.readLine()) != null) {

sb.append(line);

}

br.close();

}

A2. If you run as it is you will get “java.lang.OutOfMemoryError: Java heap space”

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at com.util.ReadLargeFileUsingStringBuilder.main(ReadLargeFileUsingStringBuilder.java:21)

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOf(Arrays.java:3332)

at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)

at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)

at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)

at java.lang.StringBuilder.append(StringBuilder.java:136)

at com.util.ReadLargeFileUsingStringBuilder.main(ReadLargeFileUsingStringBuilder.java:21)

Now, if you run it with an 8 GB heap memory, 4 GB to store 2 GB file, and another 4 GB for reading the file into a string.

-d64 -Xms8000m -Xmx8000m

-d64 -Xms8000m -Xmx8000m

You will get a slightly different “OutOfMemoryError – Requested array size exceeds VM limit”

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at com.util.ReadLargeFileUsingStringBuilder.main(ReadLargeFileUsingStringBuilder.java:21)

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

at java.util.Arrays.copyOf(Arrays.java:3332)

at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)

at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)

at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)

at java.lang.StringBuilder.append(StringBuilder.java:136)

at com.util.ReadLargeFileUsingStringBuilder.main(ReadLargeFileUsingStringBuilder.java:21)

When you do “sb.append(line);”, if the underlying array char[] knows that it needs more capacity to satisfy a specific append, it doubles its capacity. This happens each time it runs out of room. The above error indicates that it has gone past the Integer.MAX_VALUE, which is 2^31 – 1.

Q3. How will you fix the above error?
A3. Firstly, it is not a good design to read a 2 GB file into memory. You must read as a stream, and write out as a stream without storing the whole file in memory. You 1) process the contents as you read in chunks 2) split file into smaller sizes, or 3) use a byte array as shown below

The following code will run without throwing “OutOfMemoryError” if run with 3 GB or more heap memory. The byte array will consume 2 GB memory and an additional 1 GB for processing.

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
public class ReadLargeFileUsingByteArray {
	public static void main(String[] args) throws IOException {
		File file = new File("c:/temp/large-2gb-file.xml");
		FileInputStream fis = new FileInputStream(file);
	
		int total = 0;
	
		int size = (int)file.length();
		byte[] bytes = new byte[size]; //~2GB
		
		while(fis.available() > 0) {
			int result = fis.read(bytes, total, size - total );
			total = total + result;
		}
	
		fis.close();
	}
}

import java.io.File;

import java.io.FileInputStream;

import java.io.IOException;

public class ReadLargeFileUsingByteArray {

public static void main(String[] args) throws IOException {

File file = new File("c:/temp/large-2gb-file.xml");

FileInputStream fis = new FileInputStream(file);

int total = 0;

int size = (int)file.length();

byte[] bytes = new byte[size]; //~2GB

while(fis.available() > 0) {

int result = fis.read(bytes, total, size - total );

total = total + result;

}

fis.close();

}

Runs if you execute with a 3 GB heap size.

-d64 -Xms3000m -Xmx3000m

-d64 -Xms3000m -Xmx3000m

Q4. What happens to the following code?

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
public class ReadLargeFileUsingByteArray {
	public static void main(String[] args) throws IOException {
		File file = new File("c:/temp/large-2gb-file.xml");
		
		FileInputStream fis = new FileInputStream(file);
	
		int total = 0;
	
		int size = (int)file.length();
		byte[] bytes = new byte[size]; //1.8GB
			
		while(fis.available() > 0) {
			int result = fis.read(bytes, total, size - total );
			total = total + result;
			System.out.println("****" + result);
		}
		
		String str = new String(bytes, "UTF-8");
		System.out.println(str);
	
		fis.close();
	}
}

import java.io.File;

import java.io.FileInputStream;

import java.io.IOException;

public class ReadLargeFileUsingByteArray {

public static void main(String[] args) throws IOException {

File file = new File("c:/temp/large-2gb-file.xml");

FileInputStream fis = new FileInputStream(file);

int total = 0;

int size = (int)file.length();

byte[] bytes = new byte[size]; //1.8GB

while(fis.available() > 0) {

int result = fis.read(bytes, total, size - total );

total = total + result;

System.out.println("****" + result);

}

String str = new String(bytes, "UTF-8");

System.out.println(str);

fis.close();

}

A4. The above code will throw “Exception in thread “main” java.lang.OutOfMemoryError: Java heap space”. But if you increase the heap memory to 8 GB.

-d64 -Xms8000m -Xmx8000m

-d64 -Xms8000m -Xmx8000m

It runs because you need 4 GB to store as a String (i.e. 2 GB * 2 bytes per char) and an additional 4 GB for processing.

Q5. How will you prevent the following code from throwing “java.lang.OutOfMemoryError: Requested array size exceeds VM limit” when the bytes are closer to 2 GB?

org.apache.hadoop.io.Text input = new org.apache.hadoop.io.Text(new String(bytes))

org.apache.hadoop.io.Text input = new org.apache.hadoop.io.Text(new String(bytes))

A5. The above code can throw the following exception.

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:795)
        at org.apache.hadoop.io.Text.encode(Text.java:450)
        at org.apache.hadoop.io.Text.set(Text.java:198)
        at org.apache.hadoop.io.Text.<init>(Text.java:88)

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)

at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)

at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:795)

at org.apache.hadoop.io.Text.encode(Text.java:450)

at org.apache.hadoop.io.Text.set(Text.java:198)

at org.apache.hadoop.io.Text.<init>(Text.java:88)

It can be prevented by just passing the “bytes” directly as shown below.

org.apache.hadoop.io.Text input = new org.apache.hadoop.io.Text(bytes)

org.apache.hadoop.io.Text input = new org.apache.hadoop.io.Text(bytes)

Relevant links

1) Java primitives & objects – memory consumption interview Q&As

2) 8 Java FAQ memory management interview Q&As.

3) jvisualvm to sample Java heap memory.

Java String & Array limitations and OutOfMemoryError

Relevant links

300+ Java & Big Data Interview FAQs

16+ Java Key Areas Interview Q&As

800+ Java Interview Q&As

300+ Java & Big Data Tutorials

Reviews

Disclaimer