What happens when there's insufficient memory to throw an OutOfMemoryError?

The JVM never really runs out of memory. It does memory computation of the heap stack in advance.

The Structure of the JVM, Chapter 3, section 3.5.2 states:

If Java virtual machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory can be made available to effect the expansion, or if insufficient memory can be made available to create the initial Java virtual machine stack for a new thread, the Java virtual machine throws an OutOfMemoryError.

For Heap, Section 3.5.3.

If a computation requires more heap than can be made available by the automatic storage management system, the Java virtual machine throws an OutOfMemoryError.

So, it does a computation in advance before doing allocation of the object.

What happens is that the JVM tries to allocate memory for an object in the memory called Permanent Generation region (or PermSpace). If allocation fails (even after the JVM invokes the Garbage Collector to try & allocate free space), it throws an OutOfMemoryError. Even exceptions requires a memory space so the error will be thrown indefinitely.

Further reading.? Furthermore, OutOfMemoryError can occur in different JVM structure.

Graham Borland seems to be right: at least my JVM apparently re-uses OutOfMemoryErrors. To test this, I wrote a simple test program:

class OOMTest {
    private static void test (OutOfMemoryError o) {
        try {
            for (int n = 1; true; n += n) {
                int[] foo = new int[n];
            }
        } catch (OutOfMemoryError e) {
            if (e == o)
                System.out.println("Got the same OutOfMemoryError twice: " + e);
            else test(e);
        }
    }
    public static void main (String[] args) {
        test(null);
    }
}

Running it produces this output:

$ javac OOMTest.java && java -Xmx10m OOMTest 
Got the same OutOfMemoryError twice: java.lang.OutOfMemoryError: Java heap space

BTW, the JVM I'm running (on Ubuntu 10.04) is this:

$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Edit: I tried to see what would happen if I forced the JVM to run completely out of memory using the following program:

class OOMTest2 {
    private static void test (int n) {
        int[] foo;
        try {
            foo = new int[n];
            test(n * 2);
        }
        catch (OutOfMemoryError e) {
            test((n+1) / 2);
        }
    }
    public static void main (String[] args) {
        test(1);
    }
}

As it turns out, it seems to loop forever. However, curiously, trying to terminate the program with Ctrl+C doesn't work, but only gives the following message:

Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated

Most runtime environments will pre-allocate on startup, or otherwise reserve, enough memory to deal with memory starvation situations. I imagine most sane JVM implementations would do this.

Last time I was working in Java and using a debugger, the heap inspector showed that the JVM allocated an instance of OutOfMemoryError on startup. In other words, it allocates the object before your program has a chance to start consuming, let alone run out of, memory.

From the JVM Spec, Chapter 3.5.2:

If Java virtual machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory can be made available to effect the expansion, or if insufficient memory can be made available to create the initial Java virtual machine stack for a new thread, the Java virtual machine throws an OutOfMemoryError.

Every Java Virtual Machine has to guarantee that it will throw an OutOfMemoryError. That implies, that it has to be capable of creating an instance of OutOfMemoryError (or haveing on created in advance) even if there's no heap space left.

Although it does not have to guarantee, that there's enough memory left to catch it and print a nice stacktrace...

Addition

You added some code to show, that the JVM might run out of heap space if it had to throw more than one OutOfMemoryError. But such an implementation would violate the requirement from above.

There is no requirement that the thrown instances of OutOfMemoryError are unique or created on demand. A JVM could prepare exactly one instance of OutOfMemoryError during startup and throw this whenever it runs out of heap space - which is once, in normal environment. In other words: the instance of OutOfMemoryError that we see could be a singleton.

Interesting question :-). While the others have given good explanations of the theoretical aspects, I decided to try it out. This is on Oracle JDK 1.6.0_26, Windows 7 64 bit.

Test setup

I wrote a simple program to exhaust memory (see below).

The program just creates a static java.util.List, and keeps stuffing fresh strings into it, until OOM is thrown. It then catches it and continues stuffing in an endless loop (poor JVM...).

Test result

As one can see from the output, the first four times OOME is thrown, it comes with a stack trace. After that, subsequent OOMEs only print java.lang.OutOfMemoryError: Java heap space if printStackTrace() is invoked.

So apparently the JVM makes an effort to print a stack trace if it can, but if memory is really tight, it just omits the trace, just like the other answers suggest.

Also interesting is the hash code of the OOME. Note that the first few OOME all have different hashes. Once the JVM starts omitting stack traces, the hash is always the same. This suggests that the JVM will use fresh (preallocated?) OOME instances as long as possible, but if push comes to shove, it will just reuse the same instance rather than having nothing to throw.

Output

Note: I truncated some stack traces to make the output easier to read ("[...]").

iteration 0
iteration 100000
iteration 200000
iteration 300000
iteration 400000
iteration 500000
iteration 600000
iteration 700000
iteration 800000
iteration 900000
iteration 1000000
iteration 1100000
iteration 1200000
iteration 1300000
iteration 1400000
iteration 1500000
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 1069480624
Keep on trying...
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.ArrayList.ensureCapacity(Unknown Source)
    at java.util.ArrayList.add(Unknown Source)
    at testsl.Div.gobbleUpMemory(Div.java:23)
    at testsl.Div.exhaustMemory(Div.java:12)
    at testsl.Div.main(Div.java:7)
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
[...]
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 616699029
Keep on trying...
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
[...]
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 2136955031
Keep on trying...
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
[...]
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 1535562945
Keep on trying...
java.lang.OutOfMemoryError: Java heap space
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 1734048134
Keep on trying...
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 1734048134
Keep on trying...
java.lang.OutOfMemoryError: Java heap space
Ouch: java.lang.OutOfMemoryError: Java heap space; hash: 1734048134
Keep on trying...
[...]

The program

public class Div{
    static java.util.List<String> list = new java.util.ArrayList<String>();

    public static void main(String[] args) {
        exhaustMemory();
    }

    private static void exhaustMemory() {
        try {
            gobbleUpMemory();
        } catch (OutOfMemoryError e) {
            System.out.println("Ouch: " + e+"; hash: "+e.hashCode());
            e.printStackTrace();
            System.out.println("Keep on trying...");
            exhaustMemory();
        }
    }

    private static void gobbleUpMemory() {
        for (int i = 0; i < 10000000; i++) {
            list.add(new String("some random long string; use constructor to force new instance"));
            if (i % 10000000== 0) {
                System.out.println("iteration "+i);
            }
        }

    }
}

I am pretty sure, the JVM will make absolutely sure that it has at least enough memory to throw an exception before it runs out of memory.

Exceptions indicating an attempt to violate the boundaries of a managed-memory environment are handled by the runtime of said environment, in this case the JVM. The JVM is its own process, which is running your application's IL. Should a program attempt to make a call that extends the call stack beyond the limits, or allocate more memory than the JVM can reserve, the runtime itself will inject an exception, which will cause the call stack to be unwound. Regardless of the amount of memory your program currently needs, or how deep its call stack, the JVM will have allocated enough memory within its own process bounds to create said exception and inject it into your code.

You seem to be confusing the virtual memory reserved by the JVM in which the JVM runs Java programs with the host OS's native memory in which the JVM is run as a native process. The JVM on your machine is running in the memory managed by the OS, not in the memory the JVM has reserved to run Java programs.

Further reading:

http://java.sys-con.com/node/1229281

http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?topic=%2Fcom.ibm.java.doc.igaa%2F_1vg000121410cbe-1195c23a635-7ffa_1001.html

And as a final note, trying to catch a java.lang.Error (and its descendant classes) in order to print a stacktrace may not give you any useful information. You want a heap dump instead.

To further clarify @Graham Borland's answer, functionally, the JVM does this at startup:

private static final OutOfMemoryError OOME = new OutOfMemoryError();

Later, the JVM executes one of the following Java bytecodes: 'new', 'anewarray', or 'multianewarray'. This instruction causes the JVM to perform a number of steps in an out of memory condition:

Invoke a native function, say allocate(). allocate() attempts to allocate memory for some a new instance of a particular class or array. That allocation request fails, so the JVM invokes another native function, say doGC(), which attempts to do garbage collection. When that function returns, allocate() tries to allocate memory for the instance once again. If that fails(*), then the JVM, within allocate(), simply does a throw OOME;, referring to the OOME that it instantiated at startup. Note that it did not have to allocate that OOME, it just refers to it.

Obviously, these are not literal steps; they'll vary from JVM to JVM in implementation, but this is the high-level idea.

(*) A significant amount of work happens here before failing. The JVM will attempt to clear SoftReference objects, attempt allocation directly into the tenured generation when using a generational collector, and possibly other things, like finalization.

The answers that say that the JVM will pre-allocate OutOfMemoryErrors are indeed correct.
In addition to testing this by provoking an out-of-memory situation we can just check the heap of any JVM (I used a small program that just does a sleep, running it using Oracle's Hotspot JVM from Java 8 update 31).

Using jmap we see that there seems to be 9 instances of OutOfMemoryError (even though we have plenty of memory):

> jmap -histo 12103 | grep OutOfMemoryError
 71:             9            288  java.lang.OutOfMemoryError
170:             1             32  [Ljava.lang.OutOfMemoryError;

We can then generate a heap dump:

> jmap -dump:format=b,file=heap.hprof 12315

and open it using Eclipse Memory Analyzer, where an OQL query shows that the JVM actually seems to pre-allocate OutOfMemoryErrors for all possible messages:

https://i.stack.imgur.com/mD2at.png

The code for the Java 8 Hotspot JVM that actually preallocates these can be found here, and looks like this (with some parts omitted):

...
// Setup preallocated OutOfMemoryError errors
k = SystemDictionary::resolve_or_fail(vmSymbols::java_lang_OutOfMemoryError(), true, CHECK_false);
k_h = instanceKlassHandle(THREAD, k);
Universe::_out_of_memory_error_java_heap = k_h->allocate_instance(CHECK_false);
Universe::_out_of_memory_error_metaspace = k_h->allocate_instance(CHECK_false);
Universe::_out_of_memory_error_class_metaspace = k_h->allocate_instance(CHECK_false);
Universe::_out_of_memory_error_array_size = k_h->allocate_instance(CHECK_false);
Universe::_out_of_memory_error_gc_overhead_limit =
  k_h->allocate_instance(CHECK_false);

...

if (!DumpSharedSpaces) {
  // These are the only Java fields that are currently set during shared space dumping.
  // We prefer to not handle this generally, so we always reinitialize these detail messages.
  Handle msg = java_lang_String::create_from_str("Java heap space", CHECK_false);
  java_lang_Throwable::set_message(Universe::_out_of_memory_error_java_heap, msg());

  msg = java_lang_String::create_from_str("Metaspace", CHECK_false);
  java_lang_Throwable::set_message(Universe::_out_of_memory_error_metaspace, msg());
  msg = java_lang_String::create_from_str("Compressed class space", CHECK_false);
  java_lang_Throwable::set_message(Universe::_out_of_memory_error_class_metaspace, msg());

  msg = java_lang_String::create_from_str("Requested array size exceeds VM limit", CHECK_false);
  java_lang_Throwable::set_message(Universe::_out_of_memory_error_array_size, msg());

  msg = java_lang_String::create_from_str("GC overhead limit exceeded", CHECK_false);
  java_lang_Throwable::set_message(Universe::_out_of_memory_error_gc_overhead_limit, msg());

  msg = java_lang_String::create_from_str("/ by zero", CHECK_false);
  java_lang_Throwable::set_message(Universe::_arithmetic_exception_instance, msg());

  // Setup the array of errors that have preallocated backtrace
  k = Universe::_out_of_memory_error_java_heap->klass();
  assert(k->name() == vmSymbols::java_lang_OutOfMemoryError(), "should be out of memory error");
  k_h = instanceKlassHandle(THREAD, k);

  int len = (StackTraceInThrowable) ? (int)PreallocatedOutOfMemoryErrorCount : 0;
  Universe::_preallocated_out_of_memory_error_array = oopFactory::new_objArray(k_h(), len, CHECK_false);
  for (int i=0; i<len; i++) {
    oop err = k_h->allocate_instance(CHECK_false);
    Handle err_h = Handle(THREAD, err);
    java_lang_Throwable::allocate_backtrace(err_h, CHECK_false);
    Universe::preallocated_out_of_memory_errors()->obj_at_put(i, err_h());
  }
  Universe::_preallocated_out_of_memory_error_avail_count = (jint)len;
}
...

and this code shows that the JVM will first try to use one of the pre-allocated errors with space for a stack trace, and then fall back to one without a stack trace:

oop Universe::gen_out_of_memory_error(oop default_err) {
  // generate an out of memory error:
  // - if there is a preallocated error with backtrace available then return it wth
  //   a filled in stack trace.
  // - if there are no preallocated errors with backtrace available then return
  //   an error without backtrace.
  int next;
  if (_preallocated_out_of_memory_error_avail_count > 0) {
    next = (int)Atomic::add(-1, &_preallocated_out_of_memory_error_avail_count);
    assert(next < (int)PreallocatedOutOfMemoryErrorCount, "avail count is corrupt");
  } else {
    next = -1;
  }
  if (next < 0) {
    // all preallocated errors have been used.
    // return default
    return default_err;
  } else {
    // get the error object at the slot and set set it to NULL so that the
    // array isn't keeping it alive anymore.
    oop exc = preallocated_out_of_memory_errors()->obj_at(next);
    assert(exc != NULL, "slot has been used already");
    preallocated_out_of_memory_errors()->obj_at_put(next, NULL);

    // use the message from the default error
    oop msg = java_lang_Throwable::message(default_err);
    assert(msg != NULL, "no message");
    java_lang_Throwable::set_message(exc, msg);

    // populate the stack trace and return it.
    java_lang_Throwable::fill_in_stack_trace_of_preallocated_backtrace(exc);
    return exc;
  }
}