So we have a 12c RMAN Recovery Catalog in a database called rman12c running on a Windows 2012 R2 server.  We noticed the rman12c database was reporting “ORA-04030: out of process memory”:

Wed Jul 05 22:47:45 2017

Errors in file E:\APP\ORACLE\diag\rdbms\rman12c\rman12c\trace\rman12c_m000_9188.trc  (incident=76891):

ORA-04030: out of process memory when trying to allocate 8048 bytes (pga heap,KGLSPF_P ARRAY)

Incident details in: E:\APP\ORACLE\diag\rdbms\rman12c\rman12c\incident\incdir_76891\rman12c_m000_9188_i76891.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Process m000 died, see its trace file

This was rather strange, seems as though the server has 8Gb of RAM and 24Gb Virtual Memory.  But from Task Manager we could see 30.5Gb of 32Gb was committed:

Investigating further using Resource Monitor we can see the process oravssw.exe has nearly 26Gb committed memory! 😐 :

Even when Task Manager is only showing 77.1Mb memory used:

A search in My Oracle Support (MOS – formally Metalink), we found the following MOS notes:
Bug 19835650 – Windows: VSS oravssw.exe memory leak (Doc ID 19835650.8)

“Description

VSS Writer leaks memory when performing volume based shadow copy.

Work Around:
Periodically stop/restart the VSS service to free the allocated memory.

Packaging Issue:
Technically and in theory this fix is included in the source of the 11.2.0.4.16 Windows Bundle.  Due build issues the 11.2.0.4.18 Windows Bundle is needed to use VSS without the memory leak.

In 12c the fix of this bug is included in 12.1.0.2.160419 Windows Bundle through the fix of bug 21634629.

Affects:

Product (Component):
Oracle Server (Rdbms)

Range of versions believed to be affected:
Versions BELOW 12.2

Versions confirmed as being affected:
12.1.0.1 (Base Release)
11.2.0.4

Platforms affected:
Windows 64bit (AMD64/EM64T)

Fixed:

The fix for 19835650 is first included in:
12.2.0.1 (Base Release)
12.1.0.2.160419 (Apr 2016) Bundle Patch for Windows Platforms
12.1.0.1 Patch 20 on Windows Platforms
11.2.0.4 Patch 16 on Windows Platforms”

Bug 19835650 : VSS ORAVSSW.EXE MEMORY LEAK

“DETAILED PROBLEM DESCRIPTION
===============================
Using Oracle VSS Writer service to backup to EMC VSS backup.  The first backup the backup memory grows from 8M to 77M and grows again and does not release the memory until the oravss service is restarted.

DIAGNOSTIC ANALYSIS
====================
oravssw.exe memory 8336k to 8500k per service running idle.  After backup this climbs to 67,192k + memory used by oravssw.exe.  The next backup it increases to 77,900k consumed and not released.  The memory returns to idle size only after bouncing the service.

WORKAROUND?
==============
No

TECHNICAL IMPACT
=================
Customer has custom script now to stop/restart the service to free the memory. This is unacceptable as a long term solution.

RELATED ISSUES (bugs, forums, RFAs)
=================================
Bug 9063341 : MEMORY LEAK OCCURS FROM ORAVSSW.EXE EVERY TIME BACKUP JOB HAS DONE”

So we restart the Oracle VSS Writer:

We instantly see the committed memory drop from 30.5Gb to 4.9Gb 🙂 :

Also the oravssw.exe drops from under 26Gb to 4Mb in Resource Monitor 🙂 :

And no more “ORA-04030: out of process memory” 🙂

The long term solution would be to apply the Bundle Patch for the correct version mentioned above.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)