Cloning an Oracle Home

You may wish to clone an Oracle Home, for example you have all your databases on a single Oracle Home but you want to separate Development from Test.  This could be so you can soak test Patch Set Updates (PSU) on Development before applying to Test and then Production.  Or you might wish to have 2 Oracle Homes, so you can patch one and then switch all databases to the patched Oracle Home for minimal downtime.

 

Copying the Oracle Home

First you need to copy the Oracle Home at file level using cp as user root as shown below:

[root@v1ex2dbadm01 ~]# cp -Rp /u01/app/oracle/product/12.1.0.2/dbhome_1 /u01/app/oracle/product/12.1.0.2/dbhome_2

Then check the Oracle Home and the Cloned Oracle Home are the same size:

[root@v1ex2dbadm01 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_1 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_1
[root@v1ex2dbadm01 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_2 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_2

Then repeat on all the other nodes:

[root@v1ex2dbadm02 ~]# cp -Rp /u01/app/oracle/product/12.1.0.2/dbhome_1 /u01/app/oracle/product/12.1.0.2/dbhome_2
[root@v1ex2dbadm02 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_1 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_1
[root@v1ex2dbadm02 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_2 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_2

 

Cloning the Oracle Home

Now we have a copy of the Oracle Home, we next need to clone using the clone.pl perl script as shown below:

/usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
'-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
'-O"LOCAL_NODE=v1ex2dbadm01"' ORACLE_BASE=/u01/app/oracle \
ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'

CLUSTER_NODES = all the nodes in the cluster
LOCAL_NODE = The node you are running the command on
ORACLE_BASE = The Oracle Base defined on the Server
ORACLE_HOME = The Cloned Oracle Home, already exported
ORACLE_HOME_NAME = The name you wish to give to the Cloned Oracle Home

[root@v1ex2dbadm01 ~]# su - oracle
[oracle@v1ex2dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ /usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
> '-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
> '-O"LOCAL_NODE=v1ex2dbadm01"' ORACLE_BASE=/u01/app/oracle \
> ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'
./runInstaller -clone -waitForCompletion "CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}" "LOCAL_NODE=v1ex2dbadm01" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2" "ORACLE_HOME_NAME=OraDB12Home2" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_2/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 5392 MB Passed
Checking swap space: must be greater than 500 MB. Actual 24532 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-06-21_02-57-14PM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2017-06-21_02-57-14PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 ..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12Home2 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2017-06-21_02-57-14PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
 .................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh on the following nodes:
[v1ex2dbadm01]

.................................................. 100% Done.

[oracle@v1ex2dbadm01 ~]$

Next check the re-linking is RDS not UDP:

[oracle@v1ex2dbadm01 ~]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/skgxpinfo
rds
[oracle@v1ex2dbadm01 ~]$

If UDP, then relink using command below:

cd $ORACLE_HOME/rdbms/lib; ORACLE_HOME=$ORACLE_HOME make -f ins_rdbms.mk ipc_rds ioracle

Then repeat on all the other nodes, remember to change LOCAL_NODE:

[root@v1ex2dbadm02 ~]# su - oracle
[oracle@v1ex2dbadm02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm02 ~]$ /usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
> '-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
> '-O"LOCAL_NODE=v1ex2dbadm02"' ORACLE_BASE=/u01/app/oracle \
> ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'
./runInstaller -clone -waitForCompletion "CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}" "LOCAL_NODE=v1ex2dbadm02" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2" "ORACLE_HOME_NAME=OraDB12Home2" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_2/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 10725 MB Passed
Checking swap space: must be greater than 500 MB. Actual 24565 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-06-21_03-05-37PM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2017-06-21_03-05-37PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 ..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12Home2 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2017-06-21_03-05-37PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
 .................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh on the following nodes:
[v1ex2dbadm02]

.................................................. 100% Done.

[oracle@v1ex2dbadm02 ~]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/skgxpinfo
rds
[oracle@v1ex2dbadm02 ~]$

Next need to run root.sh as shown below:

[oracle@v1ex2dbadm01 ~]$ exit
logout
[root@v1ex2dbadm01 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_2/install/root_v1ex2dbadm01.v1.com_2017-06-21_15-04-16.log for the output of root script
[root@v1ex2dbadm01 ~]#

Then repeat on all the other nodes:

[oracle@v1ex2dbadm02 ~]$ exit
logout
[root@v1ex2dbadm02 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_2/install/root_v1ex2dbadm02.v1.com_2017-06-21_15-06-49.log for the output of root script
[root@v1ex2dbadm02 ~]#

Verify that the Cloned Oracle Home comprises of all the nodes in the cluster:

[root@v1ex2dbadm01 ~]# su - oracle
[oracle@v1ex2dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
Local node = v1ex2dbadm01
Remote node = v1ex2dbadm02
[oracle@v1ex2dbadm01 ~]$

Then repeat on all the other nodes:

[root@v1ex2dbadm02 ~]# su - oracle
[oracle@v1ex2dbadm02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm02 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
Local node = v1ex2dbadm02
Remote node = v1ex2dbadm01
[oracle@v1ex2dbadm02 ~]$

 

Switching Databases to Cloned Oracle Home

Change the Oracle Home using server control:

[oracle@v1ex2dbadm01 ~]$ srvctl config database -d V1DEV -a
Database unique name: V1DEV
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATAC1/V1DEV/PARAMETERFILE/spfileV1DEV.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: v1jobservice
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dba
Database instances: V1DEV1,V1DEV2
Configured nodes: v1ex2dbadm01,v1ex2dbadm02
Database is administrator managed
[oracle@v1ex2dbadm01 ~]$ srvctl modify database -d V1DEV -o /u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ srvctl config database -d V1DEV -a
Database unique name: V1DEV
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2
Oracle user: oracle
Spfile: +DATAC1/V1DEV/PARAMETERFILE/spfileV1DEV.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: v1jobservice
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dba
Database instances: V1DEV1,V1DEV2
Configured nodes: v1ex2dbadm01,v1ex2dbadm02
Database is administrator managed
[oracle@v1ex2dbadm01 ~]$

Next change /etc/oratab to reflect the new Oracle Home:

[oracle@v1ex2dbadm01 ~]$ vi /etc/oratab
[oracle@v1ex2dbadm01 ~]$ more /etc/oratab | grep dbhome_2
V1DEV1:/u01/app/oracle/product/12.1.0.2/dbhome_2:N # line added by Agent

Then repeat on all the other nodes:

[oracle@v1ex2dbadm02 ~]$ vi /etc/oratab
[oracle@v1ex2dbadm02 ~]$ more /etc/oratab | grep dbhome_2
V1DEV2:/u01/app/oracle/product/12.1.0.2/dbhome_2:N # line added by Agent

Now we rolling bounce the database:

[oracle@v1ex2dbadm01 ~]$ srvctl status database -d V1DEV -v
Instance V1DEV1 is running on node v1ex2dbadm01. Instance status: Open,Running from Old Oracle Home.
Instance V1DEV2 is running on node v1ex2dbadm02 with online services v1jobservice. Instance status: Open,Running from Old Oracle Home.
[oracle@v1ex2dbadm01 ~]$ srvctl stop instance -d V1DEV -i V1DEV1 -f
[oracle@v1ex2dbadm01 ~]$ srvctl start instance -d V1DEV -i V1DEV1
[oracle@v1ex2dbadm01 ~]$ srvctl stop instance -d V1DEV -i V1DEV2 -f
[oracle@v1ex2dbadm01 ~]$ srvctl start instance -d V1DEV -i V1DEV2
[oracle@v1ex2dbadm01 ~]$ srvctl status database -d V1DEV -v
Instance V1DEV1 is running on node v1ex2dbadm01. Instance status: Open.
Instance V1DEV2 is running on node v1ex2dbadm02 with online services v1jobservice. Instance status: Open.
[oracle@v1ex2dbadm01 ~]$

 

Related My Oracle Support (MOS) notes:

Master Note For Cloning Oracle Database Server ORACLE_HOME’s Using the Oracle Universal Installer (OUI) (Doc ID 1154613.1)

Cloning An Existing Oracle Database 12c Release 1 (12.1.0.x) RDBMS Installation Using OUI (Doc ID 1493677.1)

Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories (Doc ID 1136544.1)

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

SNMP unresponsive on Storage Cell after Exadata Patching

After a round of Exadata patching, the Storage Cells become unresponsive to SNMP for monitoring tools like OpsViews:

check_snmp_sysinfo CRITICAL - Agent not responding, tried SNMP v1 and v2c

This is because the Storage Cells get re-imaged as part of the Exadata patching, confirmed by imagehistory:

[root@v1ex1celadm01 ~]# imagehistory
Version : 12.1.2.1.1.150316.2
Image activation date : 2015-05-01 16:10:10 -0700
Imaging mode : fresh
Imaging status : success

Version : 12.1.2.1.3.151021
Image activation date : 2015-12-16 05:25:21 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.2.1.160330
Image activation date : 2016-05-12 14:02:43 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.2.160721
Image activation date : 2016-10-05 01:45:42 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.3.161208
Image activation date : 2017-01-18 02:53:10 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.4.170111
Image activation date : 2017-05-23 10:58:24 +0100
Imaging mode : out of partition upgrade
Imaging status : success

[root@v1oex1celadm01 ~]#

Which knocks out the configuration in iptables and snmpd.conf.

To resolve add back in the lines for your primary and secondary monitoring servers, for example:

rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32

So looks like this:

[root@v1ex1celadm01 ~]# more /etc/snmp/snmpd.conf
trapcommunity public
trapsink 127.0.0.1 public
rocommunity public 127.0.0.1
rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32
rwcommunity public 127.0.0.1

access RWGroup "" any noauth exact all all all
com2sec snmpclient 127.0.0.1 public
group RWGroup v1 snmpclient

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
syslocation Unknown (edit /etc/snmp/snmpd.conf)

view all included .1 80

[root@v1ex1celadm01 ~]#

Then reload snmp using the user root:

[root@v1ex2celadm01 ~]# service snmpd reload
Reloading snmpd: [ OK ]
[root@v1ex2celadm01 ~]#

Next add the monitoring servers primary and secondary to iptables as follows:

[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.31 -p udp --dport 161 -j ACCEPT
[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.32 -p udp --dport 161 -j ACCEPT

And make permanent by saving to firewall rules:

[root@v1ex1dbadm01 ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@v1ex1dbadm01 ~]#

Now your monitoring should return as SNMP connectivity is restored 🙂 :

SNMP OK - "Linux v1ex1celadm01.v1.com 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64"

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

ORA-00322: log 41 of thread 2 is not current copy

On a Data Guard Physical Standby, I noticed the following in the alert log:

Fri Jan 13 08:12:39 2017 
Errors in file /u01/app/oracle/diag/rdbms/v1s/V1S1/trace/V1S1_arc1_399861.trc: 
ORA-00322: log 41 of thread 2 is not current copy 
ORA-00312: online log 41 thread 2: '+RECOC1/V1S/ONLINELOG/group_41.3099.912775569' 
ORA-00322: log 41 of thread 2 is not current copy 
ORA-00312: online log 41 thread 2: '+DATAC1/V1S/ONLINELOG/group_41.807.912775567'

If you read metalink note:

Physical standby – alert*log shows ORA-322, ORA-312 intermittently (Doc ID 1592057.1)

SYMPTOMS

alert<SID>.log shows intermittently ORA-322 and ORA-312 for SRLs (standby redo logs).

CAUSE

These errors are written due to a very small timing window, when the SRL header
is being updated to reflect new content from the primary and in the same way
the ARCH process is doing its activity and seeing a CF flag entry that reflects an earlier
state.

SOLUTION

check the alert<SID>.log for any further issues. As long as ORA-322, ORA-312 is not combined with other issues, you can ignore this error.

Related Issue:

ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn’t match 37361

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn’t match 37361

On a Data Guard Physical Standby, I noticed the following in the alert log:

Mon Jan 30 12:37:22 2017
Errors in file /u01/app/oracle/diag/rdbms/v1s/V1S1/trace/V1S1_arc1_105040.trc:
ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn't match 37361
ORA-00312: online log 42 thread 2: '+RECOC1/V1S/ONLINELOG/group_42.15446.927996415'
ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn't match 37361
ORA-00312: online log 42 thread 2: '+DATAC1/V1S/ONLINELOG/group_42.1051.927996411'

If you read metalink note:

ORA-00314: LOG 404 OF THREAD 4, EXPECTED SEQUENCE# 33808 DOESN’T MATCH 33543 (Doc ID 1077564.1)

It explains that the standby redo has a corrupt entry, due to a problem (instance crash, network problem) that happened, it was receiving sequence 33808 and was corrupted in the middle.  So the header has info on sequence 33808 but current archive sequence # being transferred is 33543.

This was confirmed when I checked the trace file:

*** 2017-01-30 12:37:17.760
----- START Event Driven Actions Dump ----
---- END Event Driven Actions Dump ----
----- START DDE Actions Dump -----
Executing SYNC actions
----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----
Successfully dispatched
----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----
Executing ASYNC actions
----- END DDE Actions Dump (total 0 csec) -----
Reading mirrors of log member '+DATAC1/V1S/ONLINELOG/group_42.1051.927996411' (thread 2 seq 37363 block 1)
ksfdrfms:Mirror Read file=+DATAC1/V1S/ONLINELOG/group_42.1051.927996411 fob=0x248d0ba68 bufp=0x7f4139596000 blkno=1 nbytes=512
ksfdafReadMirror: Read success from mirror side=1 logical extent number=0 disk=DATAC1_CD_01_V1OEX2CELADM01 path=o/10.1.11.14;10.1.11.15/DATAC1_CD_01_v1oex2celadm01
Mirror I/O done from ASM disk o/10.1.11.14;10.1.11.15/DATAC1_CD_01_v1oex2celadm01
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 42 thread 2: '+DATAC1/V1S/ONLINELOG/group_42.1051.927996411'
Read from log mirror 'DATAC1_CD_01_V1OEX2CELADM01' (thread 2 seq 37363 block 1) found corrupt block
ksfdrnms:Mirror Read file=+DATAC1/V1S/ONLINELOG/group_42.1051.927996411 fob=0x248d0ba68 bufp=0x7f4139596000 nbytes=512

Where we can see the ‘found corrupt block’.

Luckily for me, Oracle resolved this itself by doing a RFS call for the archive redo log it required and then continued media recovery 🙂 :

Mon Jan 30 12:37:23 2017
RFS[2]: Selected log 42 for thread 2 sequence 37363 dbid 284911310 branch 891349070
Mon Jan 30 12:38:09 2017
Media Recovery Waiting for thread 2 sequence 37363 (in transit)
Mon Jan 30 12:38:09 2017
Recovery of Online Redo Log: Thread 2 Group 42 Seq 37363 Reading mem 0
 Mem# 0: +DATAC1/V1S/ONLINELOG/group_42.1051.927996411
 Mem# 1: +RECOC1/V1S/ONLINELOG/group_42.15446.927996415

If you’re not so lucky you can resolve following the steps in the metalink note:

ORA-00314: LOG 404 OF THREAD 4, EXPECTED SEQUENCE# 33808 DOESN’T MATCH 33543 (Doc ID 1077564.1)

Which are:
Stopping recovery
Clear standby redo log
Restart recovery

Related Issue:

ORA-00322: log 41 of thread 2 is not current copy

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Oracle Database Backup Service Fails with: ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’

I discovered an Oracle Cloud Database Backup failing with:

Starting backup at 2017/01/20 20:00:04
current log archived
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 01/20/2017 20:00:08
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27023: skgfqsbi: media manager protocol error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
 KBHS-00715: HTTP error occurred 'oracle-error'
KBHS-00712: ORA-28750 received from local HTTP service

Recovery Manager complete.

Upon investigation I found the following metalink note:
Oracle Database Backup Service Fails with: ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’ (Doc ID 2093475.1)

The issue is caused by Bug 22481516 – ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’

The fix for Bug 22481516  has been  included in the latest Cloud Backup Module.  The Linux 64 bit SBT Library has been updated with the fix as well.

So the solution is to install the latest Cloud Backup Module as follows:

1. Download the latest opc_installer.zip from:
http://www.oracle.com/technetwork/database/availability/oracle-cloud-backup-2162729.html
2. Install the latest Cloud Backup Module:

 [oracle@V1LOEM ~]$ cd /u01/oracle_stage/cloud/
 [oracle@V1LOEM cloud]$ ls -ltrh
 total 4.9M
 -rw-r--r--. 1 oracle oinstall 9.8K Nov 28 2014 opc_readme.txt
 -rw-r--r--. 1 oracle oinstall 2.5M May 12 2015 opc_install.jar
 -rw-r--r--. 1 oracle oinstall 2.4M Sep 9 2015 opc_installer.zip
 [oracle@V1LOEM cloud]$ mv opc_installer.zip opc_installer.zip.old

scp the latest opc_installer.zip to the server.

 [oracle@V1LOEM cloud]$ unzip opc_installer.zip
 Archive: opc_installer.zip
 replace opc_install.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 inflating: opc_install.jar
 replace opc_readme.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 inflating: opc_readme.txt
 [oracle@V1LOEM cloud]$ ls -l
 total 7596
 -rw-r--r--. 1 root root 2591749 Jan 25 14:16 opc_installer.zip
 -rw-r--r--. 1 oracle oinstall 2452163 Sep 9 2015 opc_installer.zip.old
 -rw-r--r--. 1 oracle oinstall 2718012 Oct 7 07:16 opc_install.jar
 -rw-r--r--. 1 oracle oinstall 10414 Oct 6 16:21 opc_readme.txt
 [oracle@V1LOEM cloud]$ . oraenv
 ORACLE_SID = [oracle] ? EMREPOS
 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 is /u01/app/oracle
 [oracle@V1LOEM cloud]$ java -jar opc_install.jar -serviceName v1cloud -identityDomain v1 -opcId 'oraclecloudbackup@version1.com' -opcPass 'xxx' -walletDir /u01/oracle/opc_wallet -libDir $ORACLE_HOME/lib -libPlatform linux64 -debug
 Oracle Database Cloud Backup Module Install Tool, build 2016-10-07
 Debug: os.name = Linux
 Debug: os.arch = amd64
 Debug: os.version = 3.8.13-98.1.2.el6uek.x86_64
 Debug: file.separator = /
 Debug: Platform = PLATFORM_LINUX64
 Debug: content >> oracle-data-v1clou-173484218880202412224280.087677
 Oracle Database Cloud Backup Module credentials are valid.
 Debug: Certificate Success:
 Subject : CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
 Validity : Wed Nov 08 00:00:00 GMT 2006 - Thu Jul 17 00:59:59 IST 2036
 Issuer : CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
 Oracle Database Cloud Backup Module wallet created in directory /u01/oracle/opc_wallet.
 Oracle Database Cloud Backup Module initialization file /u01/app/oracle/product/12.1.0/dbhome_1/dbs/opcEMREPOS.ora created.
 Downloading Oracle Database Cloud Backup Module Software Library from file opc_linux64.zip.
 Debug: Temp zip file = /tmp/opc_linux641742885153833539199.zip
 Downloaded 26528348 bytes in 17 seconds. Transfer rate was 1560491 bytes/second.
 Download complete.
 Debug: Delete RC = true

Now test the latest Cloud Backup Module:

[oracle@V1LOEM cloud]$ rman target /
 Recovery Manager: Release 12.1.0.2.0 - Production on Wed Jan 25 14:21:29 2017

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

connected to target database: EMREPOS (DBID=2214718289)

RMAN> delete obsolete recovery window of 8 days device type sbt;

using target database control file instead of recovery catalog
 allocated channel: ORA_SBT_TAPE_1
 channel ORA_SBT_TAPE_1: SID=818 device type=SBT_TAPE
 channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_2
 channel ORA_SBT_TAPE_2: SID=411 device type=SBT_TAPE
 channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_3
 channel ORA_SBT_TAPE_3: SID=1159 device type=SBT_TAPE
 channel ORA_SBT_TAPE_3: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_4
 channel ORA_SBT_TAPE_4: SID=43 device type=SBT_TAPE
 channel ORA_SBT_TAPE_4: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_5
 channel ORA_SBT_TAPE_5: SID=787 device type=SBT_TAPE
 channel ORA_SBT_TAPE_5: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_6
 channel ORA_SBT_TAPE_6: SID=42 device type=SBT_TAPE
 channel ORA_SBT_TAPE_6: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_7
 channel ORA_SBT_TAPE_7: SID=41 device type=SBT_TAPE
 channel ORA_SBT_TAPE_7: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_8
 channel ORA_SBT_TAPE_8: SID=1163 device type=SBT_TAPE
 channel ORA_SBT_TAPE_8: Oracle Database Backup Service Library VER=3.16.9.21
 Deleting the following obsolete backups and copies:
 Type Key Completion Time Filename/Handle
 -------------------- ------ ------------------ --------------------
Backup Set 7986 16-JAN-17
 Backup Piece 7986 16-JAN-17 qsrq6lep_1_1
 Backup Set 7990 16-JAN-17
 Backup Piece 7990 16-JAN-17 qtrq6lep_1_1
 Backup Set 7987 16-JAN-17
 Backup Piece 7987 16-JAN-17 qurq6lep_1_1
 Backup Set 7989 16-JAN-17
 Backup Piece 7989 16-JAN-17 qvrq6lep_1_1
 Backup Set 7984 16-JAN-17
 Backup Piece 7984 16-JAN-17 r0rq6lep_1_1
 Backup Set 7988 16-JAN-17
 Backup Piece 7988 16-JAN-17 r1rq6lep_1_1
 Backup Set 7985 16-JAN-17
 Backup Piece 7985 16-JAN-17 r2rq6lep_1_1

Do you really want to delete the above objects (enter YES or NO)? no

RMAN> exit
 Recovery Manager complete.
 [oracle@V1LOEM cloud]$

Everything working again 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

DBCA and DBUA no longer function after applying Windows DB Bundle Patch 24922870 – 11.2.0.4.161118

It’s been discovered that after applying 11.2.0.4.161118 Windows DB Bundle Patch 24922870, both DBCA and DBUA no longer function and give the following error:

dbca_error_after_patching

Oracle Support have only filed a bug on Wednesday this week (11th January 2017):

Bug 25369562 : DBCA & DBUA NO LONGER FUNCTION AFTER APPLYING PATCH 24922870

So if you’re planning to use either DBCA or DBUA after applying the patch, you might want to hold off until the bug is fixed 🙂

It’s also discovered NETCA no longer function after applying this patch, it will start but does nothing when you click on ‘Configure Listener’.

Credit to my colleague for sharing this info.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Using Linux Screen to protect against network breaks

When I carried out my first Exadata patching, I came across ‘Linux Screen‘ in the documentation, which Oracle recommended to use when patching:

“It’s recommended to run the dbnodeupdate.sh session using the Linux ‘screen’ or ‘vnc’ utility such that when a network problem breaks the connection to the server the patching session continues.”

Basically ‘Linux Screen‘ allows you to:

  1. Have a shell that can be reconnected to if you lose your connection due to network break, accidental close of terminal, the machine running the terminal crashes, etc. We’ve all been there, but now you can have the ability to protect yourself from these issues 🙂
  2. Disconnect from your screen session and reconnect to the same session at a later point in time, i.e. disconnect at the office, go home, reconnect when back home or kick off a long batch, disconnect, reconnect later to see if it’s completed.
  3. Open multiple windows within the one screen session.

Install Linux Screen

Linux screen is available from Oracle and can be downloaded via Oracle Public-yum:

[root@v1ex1dbadm01 ~]# yum install screen
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package screen.x86_64 0:4.0.3-19.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================
 Package Arch Version Repository Size
==============================================================================================================================================================================================================================================
Installing:
 screen x86_64 4.0.3-19.el6 ol6_latest 494 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install 1 Package(s)

Total download size: 494 k
Installed size: 795 k
Is this ok [y/N]: y
Downloading Packages:
screen-4.0.3-19.el6.x86_64.rpm | 494 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
 Installing : screen-4.0.3-19.el6.x86_64 1/1
 Verifying : screen-4.0.3-19.el6.x86_64 1/1

Installed:
 screen.x86_64 0:4.0.3-19.el6

Complete!
[root@v1ex1dbadm01 ~]#

To manually install:

[root@v1ex1dbadm01 patches]# rpm -i screen-4.0.3-19.el6.x86_64.rpm
[root@v1ex1dbadm01 patches]# rpm -qa | grep screen
screen-4.0.3-19.el6.x86_64
[root@v1ex1dbadm01 patches]# which screen
/usr/bin/screen
[root@v1ex1dbadm01 patches]#

RPMs are available via Oracle Public-yum:

http://public-yum.oracle.com/oracle-linux-5.html

http://public-yum.oracle.com/oracle-linux-6.html

Examples:

Oracle Linux 5, Update 11:

http://public-yum.oracle.com/repo/OracleLinux/OL5/11/base/x86_64/getPackage/screen-4.0.3-4.el5.x86_64.rpm

Oracle Linux 6, Update 8:

http://public-yum.oracle.com/repo/OracleLinux/OL6/8/base/x86_64/getPackage/screen-4.0.3-19.el6.x86_64.rpm

Starting Linux Screen

To start Linux Screen:

[root@v1ex1dbadm01 ~]# screen

Linux Screen Commands

Linux Screen commands are used by first pressing “CTRL” then “a“, followed by command key:

Example: “CTRL” then “a” then “?” get you the following help screen:

Screen key bindings, page 1 of 1.

Command key: ^A Literal ^A: a

break ^B b displays * help ? lockscreen ^X x number N quit \ screen ^C c title A writebuf >
clear C dumptermcap . history { } log H only Q readbuf < select ' vbell ^G xoff ^S s
colon : fit F info i login L other ^A redisplay ^L l silence _ version v xon ^Q q
copy ^[ [ flow ^F f kill K k meta a pow_break B remove X split S width W
detach ^D d focus ^I lastmsg ^M m monitor M pow_detach D removebuf = suspend ^Z z windows ^W w
digraph ^V hardcopy h license , next ^@ ^N sp n prev ^H ^P p ^? reset Z time ^T t wrap ^R r

^] paste .
" windowlist -b
- select -
0 select 0
1 select 1
2 select 2
3 select 3
4 select 4
5 select 5
6 select 6
7 select 7
8 select 8
9 select 9
I login on
O login off
] paste .


 [Press Space or Return to end.]

Detaching from Linux Screen

To detach from Linux Screen:

CTRL” then “a” then “d

[root@v1ex1dbadm01 ~]# echo "Detaching from Linux Screen, leaving my mark :)"
Detaching from Linux Screen, leaving my mark :)
[detached]

You are at this point returned to your normal shell.

Please Note: network break, accidental close of terminal, the machine running the terminal crashes, etc, automatically detach 🙂

Reattaching to Linux Screen

To reattach to Linux Screen:

[root@v1ex1dbadm01 ~]# screen -r

You’ll now be reattached to you last session:

[root@v1ex1dbadm01 ~]# echo "Detaching from Linux Screen, leaving my mark :)"
Detaching from Linux Screen, leaving my mark :)
[root@v1ex1dbadm01 ~]#

To reattach to Linux Screen, when multiple sessions are detached, use screen -r :

[root@v1ex1dbadm01 ~]# screen -r
There are several suitable screens on:
 15090.pts-0.v1ex1dbadm01 (Detached)
 346219.pts-0.v1ex1dbadm01 (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
[root@v1ex1dbadm01 ~]# screen -r 346219.pts-0.v1ex1dbadm01

Terminating Linux Screen

Once you’re finished, just type ‘exit‘ and you’ll exit back to your normal shell:

[screen is terminating]
[root@v1ex1dbadm01 ~]# screen -r
There is no screen to be resumed.
[root@v1ex1dbadm01 ~]#

You can also use:

CTRL” then “a” then “k“:

Really kill this window [y/n]y
[screen is terminating]

Advance Uses

Creating multiple screen in one session

You can add screen to your current session by:

CTRL” then “a” then “c

You’ll notice your putty session will say something like:

[screen 1:bash]root@v1ex1dbadm01:~

Screen ID starts with 0 and increases by 1 for every new screen created.

Switching between multiple screens in one session

You can switch between multiple screens, by going next and previous:

Next: “CTRL” then “a” then “n

Previous: “CTRL” then “a” then “p

You’ll notice the screen ID change as well as the screen.

Closing is the same as terminating Linux Screen, either type ‘exit‘ or “CTRL” then “a” then “k“.  You’ll need to do this for every screen until all are closed and then you will get:

[screen is terminating]

Known Issues

Cannot open your terminal

If you get the following error message:

[oracle@v1ex1dbadm01 ~]$ screen
Cannot open your terminal '/dev/pts/0' - please check.

Simple run Linux Screen as the user ‘root’ and then su to the desired user:

[root@v1ex1dbadm01 ~]# screen
[root@v1ex1dbadm01 ~]# su - oracle
[oracle@v1ex1dbadm01 ~]$

Scrollbar doesn’t work

You can enter copy mode (not sure why it’s call that, but it lets you navigate):

CTRL” then “a” then “esc” then “up“/”down

Or even better add the follow line to the root ~/.screenrc file:

termcapinfo xterm ti@:te@

If the file doesn’t exist, create it and add the line.

Now you can use scrollbar as normal 🙂

 

There are many more features, which I haven’t yet explored, but now you’ve got the idea, you’ll be able to use ‘Linux Screen’ to your benefit 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)