How to Checksum Downloads

As a DBA, in your time, you will need to download files such as Oracle binaries, patches, etc.  As a rule of thumb, you should always checksum you downloads, to ensure they are free of corruption.  In my example, I needed to download:
Patch 25440397: QUARTERLY FULL STACK DOWNLOAD PATCH FOR EXADATA (Apr 2017 – 12.1.0.2)

This is the Quarterly Full Stack Download Patch for Exadata, which is 10 zip files that total 13Gb.  That’s a lot of bytes and although highly unlikely, it’s possible for one of them to become corrupt.  Therefore, to avoid downstream pain, check the downloads’ md5 checksum on your machine.  The Microsoft File Checksum Integrity Verifier (FCIV), can be found here:
How to compute the MD5 or SHA-1 cryptographic hash values for a file

Below shows the downloads’ MD5 checksum using FCIV:

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_1of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
855c3d128adb7f87eee04d7615f20e2b c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_1of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_2of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
93cea0dddd371e148f6cb3e35f9faa6a c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_2of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_3of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
22318acb7fa36154f0bb6d02115daba4 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_3of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_4of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
28957a7ac3f2b05625cf3058fc97d981 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_4of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_5of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
a1cdad9db470bb10eb14d85c3240fa64 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_5of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_6of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
f358ebbf9e2fc222795176091939054c c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_6of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_7of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
3d20be91084b10fcb6fe3aa045346bf1 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_7of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_8of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
7cebaca222e384957b8e65df0de6e6b1 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_8of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_9of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
5a79f025564427a14ab7adce31d1a8bb c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_9of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_10of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
34b8bfe3792fa9dbd0cd13ba61bc5696 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_10of10.zip

C:\Users\anwarz>

The MD5 checksums then can verified with what Oracle Support list under the patch, click ‘Download’, then ‘View Digest Details’:

Patch_MD5

Once checked on your machine and then you transfer to server (assuming you didn’t download directly to server using WGET or something), then check again using md5sum:

[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_1of10.zip
 855c3d128adb7f87eee04d7615f20e2b p25440397_121020_linux-x86-64_1of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_2of10.zip
93cea0dddd371e148f6cb3e35f9faa6a p25440397_121020_linux-x86-64_2of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_3of10.zip
22318acb7fa36154f0bb6d02115daba4 p25440397_121020_linux-x86-64_3of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_4of10.zip
28957a7ac3f2b05625cf3058fc97d981 p25440397_121020_linux-x86-64_4of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_5of10.zip
a1cdad9db470bb10eb14d85c3240fa64 p25440397_121020_linux-x86-64_5of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_6of10.zip
f358ebbf9e2fc222795176091939054c p25440397_121020_linux-x86-64_6of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_7of10.zip
3d20be91084b10fcb6fe3aa045346bf1 p25440397_121020_linux-x86-64_7of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_8of10.zip
7cebaca222e384957b8e65df0de6e6b1 p25440397_121020_linux-x86-64_8of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_9of10.zip
5a79f025564427a14ab7adce31d1a8bb p25440397_121020_linux-x86-64_9of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_10of10.zip
34b8bfe3792fa9dbd0cd13ba61bc5696 p25440397_121020_linux-x86-64_10of10.zip
[oracle@v1oex1dbadm01 25440397]$

Once all checked, you’re free to unzip, knowing you’re free from corruption 🙂

Mined you, that doesn’t guarantee you wouldn’t get corruption from I/O but that would be extremely unlucky and you’ll have bigger problems.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to exit out of ed in SQL*Plus

I, from time to time, get stuck in ‘ed‘ on SQL*Plus because the editor not defined like this:

SQL> select * from duall;
select * from duall
 *
ERROR at line 1:
ORA-00942: table or view does not exist

SQL> ed
Wrote file afiedt.buf
22
^C
?
^Z
?

The Ctrl-C and Ctrl-Z, wouldn’t get you out.

The solution is simple, type ‘q‘ and press ‘Enter‘, then set editor using ‘define_editor=vi‘ and try again 🙂 :

q

1* select * from duall
SQL> define_editor=vi
SQL> ed
Wrote file afiedt.buf

1* select * from dual
SQL> /

D
-
X

SQL>

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

VSS oravssw.exe Memory Leak on Windows

So we have a 12c RMAN Recovery Catalog in a database called rman12c running on a Windows 2012 R2 server.  We noticed the rman12c database was reporting “ORA-04030: out of process memory”:

Wed Jul 05 22:47:45 2017

Errors in file E:\APP\ORACLE\diag\rdbms\rman12c\rman12c\trace\rman12c_m000_9188.trc  (incident=76891):

ORA-04030: out of process memory when trying to allocate 8048 bytes (pga heap,KGLSPF_P ARRAY)

Incident details in: E:\APP\ORACLE\diag\rdbms\rman12c\rman12c\incident\incdir_76891\rman12c_m000_9188_i76891.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Process m000 died, see its trace file

This was rather strange, seems as though the server has 8Gb of RAM and 24Gb Virtual Memory.  But from Task Manager we could see 30.5Gb of 32Gb was committed:

VSS_Server_Memory_before_restart

Investigating further using Resource Monitor we can see the process oravssw.exe has nearly 26Gb committed memory! 😐 :

VSS_oravssw_leak_before_restart

Even when Task Manager is only showing 77.1Mb memory used:

VSS_oravssw_leak_process

A search in My Oracle Support (MOS – formally Metalink), we found the following MOS notes:
Bug 19835650 – Windows: VSS oravssw.exe memory leak (Doc ID 19835650.8)

“Description

VSS Writer leaks memory when performing volume based shadow copy.

Work Around:
Periodically stop/restart the VSS service to free the allocated memory.

Packaging Issue:
Technically and in theory this fix is included in the source of the 11.2.0.4.16 Windows Bundle.  Due build issues the 11.2.0.4.18 Windows Bundle is needed to use VSS without the memory leak.

In 12c the fix of this bug is included in 12.1.0.2.160419 Windows Bundle through the fix of bug 21634629.

Affects:

Product (Component):
Oracle Server (Rdbms)

Range of versions believed to be affected:
Versions BELOW 12.2

Versions confirmed as being affected:
12.1.0.1 (Base Release)
11.2.0.4

Platforms affected:
Windows 64bit (AMD64/EM64T)

Fixed:

The fix for 19835650 is first included in:
12.2.0.1 (Base Release)
12.1.0.2.160419 (Apr 2016) Bundle Patch for Windows Platforms
12.1.0.1 Patch 20 on Windows Platforms
11.2.0.4 Patch 16 on Windows Platforms”

Bug 19835650 : VSS ORAVSSW.EXE MEMORY LEAK

“DETAILED PROBLEM DESCRIPTION
===============================
Using Oracle VSS Writer service to backup to EMC VSS backup.  The first backup the backup memory grows from 8M to 77M and grows again and does not release the memory until the oravss service is restarted.

DIAGNOSTIC ANALYSIS
====================
oravssw.exe memory 8336k to 8500k per service running idle.  After backup this climbs to 67,192k + memory used by oravssw.exe.  The next backup it increases to 77,900k consumed and not released.  The memory returns to idle size only after bouncing the service.

WORKAROUND?
==============
No

TECHNICAL IMPACT
=================
Customer has custom script now to stop/restart the service to free the memory. This is unacceptable as a long term solution.

RELATED ISSUES (bugs, forums, RFAs)
=================================
Bug 9063341 : MEMORY LEAK OCCURS FROM ORAVSSW.EXE EVERY TIME BACKUP JOB HAS DONE”

So we restart the Oracle VSS Writer:
Vss_oravssw_restart

We instantly see the committed memory drop from 30.5Gb to 4.9Gb 🙂 :

VSS_Server_Memory_after_restart

Also the oravssw.exe drops from under 26Gb to 4Mb in Resource Monitor 🙂 :

VSS_oravssw_leak_after_restart

And no more “ORA-04030: out of process memory” 🙂

The long term solution would be to apply the Bundle Patch for the correct version mentioned above.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Extending a Non-root LVM Partition on Exadata

On an Oracle Exadata Database Machine, the ‘/u01’ is defaulted to a size of 100Gb, which can easily fill up.  Luckily this is just a Logical Volume and there’s normally lots of space available on the Logical Volume Group which is usually untapped.

 

Extending ‘/u01’

Identify how much space is used and free on ‘/u01’ using df:

[root@v1ex1dbadm01 ~]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1
99G 78G 17G 83% /u01
[root@v1ex1dbadm01 ~]#

Display the current logical volume configuration using the lvs command:

[root@v1ex1dbadm01 ~]# lvs -o lv_name,lv_path,vg_name,lv_size
LV Path VG LSize
LVDbOra1 /dev/VGExaDb/LVDbOra1 VGExaDb 100.00g
LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb 24.00g
LVDbSys1 /dev/VGExaDb/LVDbSys1 VGExaDb 30.00g
LVDbSys2 /dev/VGExaDb/LVDbSys2 VGExaDb 30.00g
LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb 1.00g
[root@v1ex1dbadm01 ~]#

Check the online resize option is available using the tune2fs command:

[root@v1ex1dbadm01 ~]# tune2fs -l /dev/VGExaDb/LVDbOra1 | grep resize_inode
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
[root@v1ex1dbadm01 ~]#

If not available then the file system needs to be unmounted before resizing.  Refer to documentation:

Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

Verify there’s space available in the Logical Volume Group using vgdisplay command:

[root@v1ex1dbadm01 ~]# vgdisplay -s
"VGExaDb" 1.63 TiB [185.00 GiB used / 1.45 TiB free]
[root@v1ex1dbadm01 ~]#

Finally if there’s enough space, then extend the Logical Volume using lvextend command:

[root@v1ex1dbadm01 ~]# lvextend -L +100G /dev/VGExaDb/LVDbOra1
Size of logical volume VGExaDb/LVDbOra1 changed from 100.00 GiB (25600 extents) to 200.00 GiB (51200 extents).
Logical volume LVDbOra1 successfully resized.
[root@v1ex1dbadm01 ~]#

Followed by a resize of the file system using resize2fs command:

[root@v1ex1dbadm01 ~]# resize2fs /dev/VGExaDb/LVDbOra1
resize2fs 1.43-WIP (20-Jun-2013)
Filesystem at /dev/VGExaDb/LVDbOra1 is mounted on /u01; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 13
Performing an on-line resize of /dev/VGExaDb/LVDbOra1 to 52428800 (4k) blocks.
The filesystem on /dev/VGExaDb/LVDbOra1 is now 52428800 blocks long.
[root@v1ex1dbadm01 ~]#

You should now see ‘/u01’ with additional 100Gb less formatting:

[root@v1ex1dbadm01 ~]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1
197G 78G 110G 42% /u01
[root@v1ex1dbadm01 ~]#

Also see the Logical Volume is now 200Gb from 100Gb:

[root@v1ex1dbadm01 ~]# lvs -o lv_name,lv_path,vg_name,lv_size
LV Path VG LSize
LVDbOra1 /dev/VGExaDb/LVDbOra1 VGExaDb 200.00g
LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb 24.00g
LVDbSys1 /dev/VGExaDb/LVDbSys1 VGExaDb 30.00g
LVDbSys2 /dev/VGExaDb/LVDbSys2 VGExaDb 30.00g
LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb 1.00g
[root@v1ex1dbadm01 ~]#

 

Documentation for reference:

Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Cloning an Oracle Home

You may wish to clone an Oracle Home, for example you have all your databases on a single Oracle Home but you want to separate Development from Test.  This could be so you can soak test Patch Set Updates (PSU) on Development before applying to Test and then Production.  Or you might wish to have 2 Oracle Homes, so you can patch one and then switch all databases to the patched Oracle Home for minimal downtime.

 

Copying the Oracle Home

First you need to copy the Oracle Home at file level using cp as user root as shown below:

[root@v1ex2dbadm01 ~]# cp -Rp /u01/app/oracle/product/12.1.0.2/dbhome_1 /u01/app/oracle/product/12.1.0.2/dbhome_2

Then check the Oracle Home and the Cloned Oracle Home are the same size:

[root@v1ex2dbadm01 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_1 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_1
[root@v1ex2dbadm01 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_2 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_2

Then repeat on all the other nodes:

[root@v1ex2dbadm02 ~]# cp -Rp /u01/app/oracle/product/12.1.0.2/dbhome_1 /u01/app/oracle/product/12.1.0.2/dbhome_2
[root@v1ex2dbadm02 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_1 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_1
[root@v1ex2dbadm02 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_2 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_2

 

Cloning the Oracle Home

Now we have a copy of the Oracle Home, we next need to clone using the clone.pl perl script as shown below:

/usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
'-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
'-O"LOCAL_NODE=v1ex2dbadm01"' ORACLE_BASE=/u01/app/oracle \
ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'

CLUSTER_NODES = all the nodes in the cluster
LOCAL_NODE = The node you are running the command on
ORACLE_BASE = The Oracle Base defined on the Server
ORACLE_HOME = The Cloned Oracle Home, already exported
ORACLE_HOME_NAME = The name you wish to give to the Cloned Oracle Home

[root@v1ex2dbadm01 ~]# su - oracle
[oracle@v1ex2dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ /usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
> '-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
> '-O"LOCAL_NODE=v1ex2dbadm01"' ORACLE_BASE=/u01/app/oracle \
> ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'
./runInstaller -clone -waitForCompletion "CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}" "LOCAL_NODE=v1ex2dbadm01" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2" "ORACLE_HOME_NAME=OraDB12Home2" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_2/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 5392 MB Passed
Checking swap space: must be greater than 500 MB. Actual 24532 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-06-21_02-57-14PM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2017-06-21_02-57-14PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 ..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12Home2 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2017-06-21_02-57-14PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
 .................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh on the following nodes:
[v1ex2dbadm01]

.................................................. 100% Done.

[oracle@v1ex2dbadm01 ~]$

Next check the re-linking is RDS not UDP:

[oracle@v1ex2dbadm01 ~]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/skgxpinfo
rds
[oracle@v1ex2dbadm01 ~]$

If UDP, then relink using command below:

cd $ORACLE_HOME/rdbms/lib; ORACLE_HOME=$ORACLE_HOME make -f ins_rdbms.mk ipc_rds ioracle

Then repeat on all the other nodes, remember to change LOCAL_NODE:

[root@v1ex2dbadm02 ~]# su - oracle
[oracle@v1ex2dbadm02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm02 ~]$ /usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
> '-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
> '-O"LOCAL_NODE=v1ex2dbadm02"' ORACLE_BASE=/u01/app/oracle \
> ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'
./runInstaller -clone -waitForCompletion "CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}" "LOCAL_NODE=v1ex2dbadm02" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2" "ORACLE_HOME_NAME=OraDB12Home2" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_2/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 10725 MB Passed
Checking swap space: must be greater than 500 MB. Actual 24565 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-06-21_03-05-37PM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2017-06-21_03-05-37PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 ..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12Home2 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2017-06-21_03-05-37PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
 .................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh on the following nodes:
[v1ex2dbadm02]

.................................................. 100% Done.

[oracle@v1ex2dbadm02 ~]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/skgxpinfo
rds
[oracle@v1ex2dbadm02 ~]$

Next need to run root.sh as shown below:

[oracle@v1ex2dbadm01 ~]$ exit
logout
[root@v1ex2dbadm01 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_2/install/root_v1ex2dbadm01.v1.com_2017-06-21_15-04-16.log for the output of root script
[root@v1ex2dbadm01 ~]#

Then repeat on all the other nodes:

[oracle@v1ex2dbadm02 ~]$ exit
logout
[root@v1ex2dbadm02 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_2/install/root_v1ex2dbadm02.v1.com_2017-06-21_15-06-49.log for the output of root script
[root@v1ex2dbadm02 ~]#

Verify that the Cloned Oracle Home comprises of all the nodes in the cluster:

[root@v1ex2dbadm01 ~]# su - oracle
[oracle@v1ex2dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
Local node = v1ex2dbadm01
Remote node = v1ex2dbadm02
[oracle@v1ex2dbadm01 ~]$

Then repeat on all the other nodes:

[root@v1ex2dbadm02 ~]# su - oracle
[oracle@v1ex2dbadm02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm02 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
Local node = v1ex2dbadm02
Remote node = v1ex2dbadm01
[oracle@v1ex2dbadm02 ~]$

 

Switching Databases to Cloned Oracle Home

Change the Oracle Home using server control:

[oracle@v1ex2dbadm01 ~]$ srvctl config database -d V1DEV -a
Database unique name: V1DEV
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATAC1/V1DEV/PARAMETERFILE/spfileV1DEV.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: v1jobservice
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dba
Database instances: V1DEV1,V1DEV2
Configured nodes: v1ex2dbadm01,v1ex2dbadm02
Database is administrator managed
[oracle@v1ex2dbadm01 ~]$ srvctl modify database -d V1DEV -o /u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ srvctl config database -d V1DEV -a
Database unique name: V1DEV
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2
Oracle user: oracle
Spfile: +DATAC1/V1DEV/PARAMETERFILE/spfileV1DEV.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: v1jobservice
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dba
Database instances: V1DEV1,V1DEV2
Configured nodes: v1ex2dbadm01,v1ex2dbadm02
Database is administrator managed
[oracle@v1ex2dbadm01 ~]$

Next change /etc/oratab to reflect the new Oracle Home:

[oracle@v1ex2dbadm01 ~]$ vi /etc/oratab
[oracle@v1ex2dbadm01 ~]$ more /etc/oratab | grep dbhome_2
V1DEV1:/u01/app/oracle/product/12.1.0.2/dbhome_2:N # line added by Agent

Then repeat on all the other nodes:

[oracle@v1ex2dbadm02 ~]$ vi /etc/oratab
[oracle@v1ex2dbadm02 ~]$ more /etc/oratab | grep dbhome_2
V1DEV2:/u01/app/oracle/product/12.1.0.2/dbhome_2:N # line added by Agent

Now we rolling bounce the database:

[oracle@v1ex2dbadm01 ~]$ srvctl status database -d V1DEV -v
Instance V1DEV1 is running on node v1ex2dbadm01. Instance status: Open,Running from Old Oracle Home.
Instance V1DEV2 is running on node v1ex2dbadm02 with online services v1jobservice. Instance status: Open,Running from Old Oracle Home.
[oracle@v1ex2dbadm01 ~]$ srvctl stop instance -d V1DEV -i V1DEV1 -f
[oracle@v1ex2dbadm01 ~]$ srvctl start instance -d V1DEV -i V1DEV1
[oracle@v1ex2dbadm01 ~]$ srvctl stop instance -d V1DEV -i V1DEV2 -f
[oracle@v1ex2dbadm01 ~]$ srvctl start instance -d V1DEV -i V1DEV2
[oracle@v1ex2dbadm01 ~]$ srvctl status database -d V1DEV -v
Instance V1DEV1 is running on node v1ex2dbadm01. Instance status: Open.
Instance V1DEV2 is running on node v1ex2dbadm02 with online services v1jobservice. Instance status: Open.
[oracle@v1ex2dbadm01 ~]$

 

Related My Oracle Support (MOS) notes:

Master Note For Cloning Oracle Database Server ORACLE_HOME’s Using the Oracle Universal Installer (OUI) (Doc ID 1154613.1)

Cloning An Existing Oracle Database 12c Release 1 (12.1.0.x) RDBMS Installation Using OUI (Doc ID 1493677.1)

Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories (Doc ID 1136544.1)

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

SNMP unresponsive on Storage Cell after Exadata Patching

After a round of Exadata patching, the Storage Cells become unresponsive to SNMP for monitoring tools like OpsViews:

check_snmp_sysinfo CRITICAL - Agent not responding, tried SNMP v1 and v2c

This is because the Storage Cells get re-imaged as part of the Exadata patching, confirmed by imagehistory:

[root@v1ex1celadm01 ~]# imagehistory
Version : 12.1.2.1.1.150316.2
Image activation date : 2015-05-01 16:10:10 -0700
Imaging mode : fresh
Imaging status : success

Version : 12.1.2.1.3.151021
Image activation date : 2015-12-16 05:25:21 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.2.1.160330
Image activation date : 2016-05-12 14:02:43 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.2.160721
Image activation date : 2016-10-05 01:45:42 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.3.161208
Image activation date : 2017-01-18 02:53:10 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.4.170111
Image activation date : 2017-05-23 10:58:24 +0100
Imaging mode : out of partition upgrade
Imaging status : success

[root@v1oex1celadm01 ~]#

Which knocks out the configuration in iptables and snmpd.conf.

To resolve add back in the lines for your primary and secondary monitoring servers, for example:

rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32

So looks like this:

[root@v1ex1celadm01 ~]# more /etc/snmp/snmpd.conf
trapcommunity public
trapsink 127.0.0.1 public
rocommunity public 127.0.0.1
rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32
rwcommunity public 127.0.0.1

access RWGroup "" any noauth exact all all all
com2sec snmpclient 127.0.0.1 public
group RWGroup v1 snmpclient

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
syslocation Unknown (edit /etc/snmp/snmpd.conf)

view all included .1 80

[root@v1ex1celadm01 ~]#

Then reload snmp using the user root:

[root@v1ex2celadm01 ~]# service snmpd reload
Reloading snmpd: [ OK ]
[root@v1ex2celadm01 ~]#

Next add the monitoring servers primary and secondary to iptables as follows:

[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.31 -p udp --dport 161 -j ACCEPT
[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.32 -p udp --dport 161 -j ACCEPT

And make permanent by saving to firewall rules:

[root@v1ex1dbadm01 ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@v1ex1dbadm01 ~]#

Now your monitoring should return as SNMP connectivity is restored 🙂 :

SNMP OK - "Linux v1ex1celadm01.v1.com 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64"

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

ORA-00322: log 41 of thread 2 is not current copy

On a Data Guard Physical Standby, I noticed the following in the alert log:

Fri Jan 13 08:12:39 2017 
Errors in file /u01/app/oracle/diag/rdbms/v1s/V1S1/trace/V1S1_arc1_399861.trc: 
ORA-00322: log 41 of thread 2 is not current copy 
ORA-00312: online log 41 thread 2: '+RECOC1/V1S/ONLINELOG/group_41.3099.912775569' 
ORA-00322: log 41 of thread 2 is not current copy 
ORA-00312: online log 41 thread 2: '+DATAC1/V1S/ONLINELOG/group_41.807.912775567'

If you read metalink note:
Physical standby – alert*log shows ORA-322, ORA-312 intermittently (Doc ID 1592057.1)

SYMPTOMS

alert.log shows intermittently ORA-322 and ORA-312 for SRLs (standby redo logs).

CAUSE

These errors are written due to a very small timing window, when the SRL header
is being updated to reflect new content from the primary and in the same way
the ARCH process is doing its activity and seeing a CF flag entry that reflects an earlier
state.

SOLUTION

check the alert.log for any further issues. As long as ORA-322, ORA-312 is not combined with other issues, you can ignore this error.

Related Issue:
ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn’t match 37361

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)