ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn’t match 37361

On a Data Guard Physical Standby, I noticed the following in the alert log:

Mon Jan 30 12:37:22 2017
Errors in file /u01/app/oracle/diag/rdbms/v1s/V1S1/trace/V1S1_arc1_105040.trc:
ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn't match 37361
ORA-00312: online log 42 thread 2: '+RECOC1/V1S/ONLINELOG/group_42.15446.927996415'
ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn't match 37361
ORA-00312: online log 42 thread 2: '+DATAC1/V1S/ONLINELOG/group_42.1051.927996411'

If you read metalink note:

ORA-00314: LOG 404 OF THREAD 4, EXPECTED SEQUENCE# 33808 DOESN’T MATCH 33543 (Doc ID 1077564.1)

It explains that the standby redo has a corrupt entry, due to a problem (instance crash, network problem) that happened, it was receiving sequence 33808 and was corrupted in the middle.  So the header has info on sequence 33808 but current archive sequence # being transferred is 33543.

This was confirmed when I checked the trace file:

*** 2017-01-30 12:37:17.760
----- START Event Driven Actions Dump ----
---- END Event Driven Actions Dump ----
----- START DDE Actions Dump -----
Executing SYNC actions
----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----
Successfully dispatched
----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----
Executing ASYNC actions
----- END DDE Actions Dump (total 0 csec) -----
Reading mirrors of log member '+DATAC1/V1S/ONLINELOG/group_42.1051.927996411' (thread 2 seq 37363 block 1)
ksfdrfms:Mirror Read file=+DATAC1/V1S/ONLINELOG/group_42.1051.927996411 fob=0x248d0ba68 bufp=0x7f4139596000 blkno=1 nbytes=512
ksfdafReadMirror: Read success from mirror side=1 logical extent number=0 disk=DATAC1_CD_01_V1OEX2CELADM01 path=o/10.1.11.14;10.1.11.15/DATAC1_CD_01_v1oex2celadm01
Mirror I/O done from ASM disk o/10.1.11.14;10.1.11.15/DATAC1_CD_01_v1oex2celadm01
DDE: Problem Key 'ORA 312' was flood controlled (0x1) (no incident)
ORA-00312: online log 42 thread 2: '+DATAC1/V1S/ONLINELOG/group_42.1051.927996411'
Read from log mirror 'DATAC1_CD_01_V1OEX2CELADM01' (thread 2 seq 37363 block 1) found corrupt block
ksfdrnms:Mirror Read file=+DATAC1/V1S/ONLINELOG/group_42.1051.927996411 fob=0x248d0ba68 bufp=0x7f4139596000 nbytes=512

Where we can see the ‘found corrupt block’.

Luckily for me, Oracle resolved this itself by doing a RFS call for the archive redo log it required and then continued media recovery 🙂 :

Mon Jan 30 12:37:23 2017
RFS[2]: Selected log 42 for thread 2 sequence 37363 dbid 284911310 branch 891349070
Mon Jan 30 12:38:09 2017
Media Recovery Waiting for thread 2 sequence 37363 (in transit)
Mon Jan 30 12:38:09 2017
Recovery of Online Redo Log: Thread 2 Group 42 Seq 37363 Reading mem 0
 Mem# 0: +DATAC1/V1S/ONLINELOG/group_42.1051.927996411
 Mem# 1: +RECOC1/V1S/ONLINELOG/group_42.15446.927996415

If you’re not so lucky you can resolve following the steps in the metalink note:

ORA-00314: LOG 404 OF THREAD 4, EXPECTED SEQUENCE# 33808 DOESN’T MATCH 33543 (Doc ID 1077564.1)

Which are:
Stopping recovery
Clear standby redo log
Restart recovery

Related Issue:

ORA-00322: log 41 of thread 2 is not current copy

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Oracle Database Backup Service Fails with: ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’

I discovered an Oracle Cloud Database Backup failing with:

Starting backup at 2017/01/20 20:00:04
current log archived
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 01/20/2017 20:00:08
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27023: skgfqsbi: media manager protocol error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
 KBHS-00715: HTTP error occurred 'oracle-error'
KBHS-00712: ORA-28750 received from local HTTP service

Recovery Manager complete.

Upon investigation I found the following metalink note:
Oracle Database Backup Service Fails with: ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’ (Doc ID 2093475.1)

The issue is caused by Bug 22481516 – ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’

The fix for Bug 22481516  has been  included in the latest Cloud Backup Module.  The Linux 64 bit SBT Library has been updated with the fix as well.

So the solution is to install the latest Cloud Backup Module as follows:

1. Download the latest opc_installer.zip from:
http://www.oracle.com/technetwork/database/availability/oracle-cloud-backup-2162729.html
2. Install the latest Cloud Backup Module:

 [oracle@V1LOEM ~]$ cd /u01/oracle_stage/cloud/
 [oracle@V1LOEM cloud]$ ls -ltrh
 total 4.9M
 -rw-r--r--. 1 oracle oinstall 9.8K Nov 28 2014 opc_readme.txt
 -rw-r--r--. 1 oracle oinstall 2.5M May 12 2015 opc_install.jar
 -rw-r--r--. 1 oracle oinstall 2.4M Sep 9 2015 opc_installer.zip
 [oracle@V1LOEM cloud]$ mv opc_installer.zip opc_installer.zip.old

scp the latest opc_installer.zip to the server.

 [oracle@V1LOEM cloud]$ unzip opc_installer.zip
 Archive: opc_installer.zip
 replace opc_install.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 inflating: opc_install.jar
 replace opc_readme.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 inflating: opc_readme.txt
 [oracle@V1LOEM cloud]$ ls -l
 total 7596
 -rw-r--r--. 1 root root 2591749 Jan 25 14:16 opc_installer.zip
 -rw-r--r--. 1 oracle oinstall 2452163 Sep 9 2015 opc_installer.zip.old
 -rw-r--r--. 1 oracle oinstall 2718012 Oct 7 07:16 opc_install.jar
 -rw-r--r--. 1 oracle oinstall 10414 Oct 6 16:21 opc_readme.txt
 [oracle@V1LOEM cloud]$ . oraenv
 ORACLE_SID = [oracle] ? EMREPOS
 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 is /u01/app/oracle
 [oracle@V1LOEM cloud]$ java -jar opc_install.jar -serviceName v1cloud -identityDomain v1 -opcId 'oraclecloudbackup@version1.com' -opcPass 'xxx' -walletDir /u01/oracle/opc_wallet -libDir $ORACLE_HOME/lib -libPlatform linux64 -debug
 Oracle Database Cloud Backup Module Install Tool, build 2016-10-07
 Debug: os.name = Linux
 Debug: os.arch = amd64
 Debug: os.version = 3.8.13-98.1.2.el6uek.x86_64
 Debug: file.separator = /
 Debug: Platform = PLATFORM_LINUX64
 Debug: content >> oracle-data-v1clou-173484218880202412224280.087677
 Oracle Database Cloud Backup Module credentials are valid.
 Debug: Certificate Success:
 Subject : CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
 Validity : Wed Nov 08 00:00:00 GMT 2006 - Thu Jul 17 00:59:59 IST 2036
 Issuer : CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
 Oracle Database Cloud Backup Module wallet created in directory /u01/oracle/opc_wallet.
 Oracle Database Cloud Backup Module initialization file /u01/app/oracle/product/12.1.0/dbhome_1/dbs/opcEMREPOS.ora created.
 Downloading Oracle Database Cloud Backup Module Software Library from file opc_linux64.zip.
 Debug: Temp zip file = /tmp/opc_linux641742885153833539199.zip
 Downloaded 26528348 bytes in 17 seconds. Transfer rate was 1560491 bytes/second.
 Download complete.
 Debug: Delete RC = true

Now test the latest Cloud Backup Module:

[oracle@V1LOEM cloud]$ rman target /
 Recovery Manager: Release 12.1.0.2.0 - Production on Wed Jan 25 14:21:29 2017

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

connected to target database: EMREPOS (DBID=2214718289)

RMAN> delete obsolete recovery window of 8 days device type sbt;

using target database control file instead of recovery catalog
 allocated channel: ORA_SBT_TAPE_1
 channel ORA_SBT_TAPE_1: SID=818 device type=SBT_TAPE
 channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_2
 channel ORA_SBT_TAPE_2: SID=411 device type=SBT_TAPE
 channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_3
 channel ORA_SBT_TAPE_3: SID=1159 device type=SBT_TAPE
 channel ORA_SBT_TAPE_3: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_4
 channel ORA_SBT_TAPE_4: SID=43 device type=SBT_TAPE
 channel ORA_SBT_TAPE_4: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_5
 channel ORA_SBT_TAPE_5: SID=787 device type=SBT_TAPE
 channel ORA_SBT_TAPE_5: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_6
 channel ORA_SBT_TAPE_6: SID=42 device type=SBT_TAPE
 channel ORA_SBT_TAPE_6: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_7
 channel ORA_SBT_TAPE_7: SID=41 device type=SBT_TAPE
 channel ORA_SBT_TAPE_7: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_8
 channel ORA_SBT_TAPE_8: SID=1163 device type=SBT_TAPE
 channel ORA_SBT_TAPE_8: Oracle Database Backup Service Library VER=3.16.9.21
 Deleting the following obsolete backups and copies:
 Type Key Completion Time Filename/Handle
 -------------------- ------ ------------------ --------------------
Backup Set 7986 16-JAN-17
 Backup Piece 7986 16-JAN-17 qsrq6lep_1_1
 Backup Set 7990 16-JAN-17
 Backup Piece 7990 16-JAN-17 qtrq6lep_1_1
 Backup Set 7987 16-JAN-17
 Backup Piece 7987 16-JAN-17 qurq6lep_1_1
 Backup Set 7989 16-JAN-17
 Backup Piece 7989 16-JAN-17 qvrq6lep_1_1
 Backup Set 7984 16-JAN-17
 Backup Piece 7984 16-JAN-17 r0rq6lep_1_1
 Backup Set 7988 16-JAN-17
 Backup Piece 7988 16-JAN-17 r1rq6lep_1_1
 Backup Set 7985 16-JAN-17
 Backup Piece 7985 16-JAN-17 r2rq6lep_1_1

Do you really want to delete the above objects (enter YES or NO)? no

RMAN> exit
 Recovery Manager complete.
 [oracle@V1LOEM cloud]$

Everything working again 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

DBCA and DBUA no longer function after applying Windows DB Bundle Patch 24922870 – 11.2.0.4.161118

It’s been discovered that after applying 11.2.0.4.161118 Windows DB Bundle Patch 24922870, both DBCA and DBUA no longer function and give the following error:

dbca_error_after_patching

Oracle Support have only filed a bug on Wednesday this week (11th January 2017):

Bug 25369562 : DBCA & DBUA NO LONGER FUNCTION AFTER APPLYING PATCH 24922870

So if you’re planning to use either DBCA or DBUA after applying the patch, you might want to hold off until the bug is fixed 🙂

It’s also discovered NETCA no longer function after applying this patch, it will start but does nothing when you click on ‘Configure Listener’.

Credit to my colleague for sharing this info.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Using Linux Screen to protect against network breaks

When I carried out my first Exadata patching, I came across ‘Linux Screen‘ in the documentation, which Oracle recommended to use when patching:

“It’s recommended to run the dbnodeupdate.sh session using the Linux ‘screen’ or ‘vnc’ utility such that when a network problem breaks the connection to the server the patching session continues.”

Basically ‘Linux Screen‘ allows you to:

  1. Have a shell that can be reconnected to if you lose your connection due to network break, accidental close of terminal, the machine running the terminal crashes, etc. We’ve all been there, but now you can have the ability to protect yourself from these issues 🙂
  2. Disconnect from your screen session and reconnect to the same session at a later point in time, i.e. disconnect at the office, go home, reconnect when back home or kick off a long batch, disconnect, reconnect later to see if it’s completed.
  3. Open multiple windows within the one screen session.

Install Linux Screen

Linux screen is available from Oracle and can be downloaded via Oracle Public-yum:

[root@v1ex1dbadm01 ~]# yum install screen
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package screen.x86_64 0:4.0.3-19.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================
 Package Arch Version Repository Size
==============================================================================================================================================================================================================================================
Installing:
 screen x86_64 4.0.3-19.el6 ol6_latest 494 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install 1 Package(s)

Total download size: 494 k
Installed size: 795 k
Is this ok [y/N]: y
Downloading Packages:
screen-4.0.3-19.el6.x86_64.rpm | 494 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
 Installing : screen-4.0.3-19.el6.x86_64 1/1
 Verifying : screen-4.0.3-19.el6.x86_64 1/1

Installed:
 screen.x86_64 0:4.0.3-19.el6

Complete!
[root@v1ex1dbadm01 ~]#

To manually install:

[root@v1ex1dbadm01 patches]# rpm -i screen-4.0.3-19.el6.x86_64.rpm
[root@v1ex1dbadm01 patches]# rpm -qa | grep screen
screen-4.0.3-19.el6.x86_64
[root@v1ex1dbadm01 patches]# which screen
/usr/bin/screen
[root@v1ex1dbadm01 patches]#

RPMs are available via Oracle Public-yum:

http://public-yum.oracle.com/oracle-linux-5.html

http://public-yum.oracle.com/oracle-linux-6.html

Examples:

Oracle Linux 5, Update 11:

http://public-yum.oracle.com/repo/OracleLinux/OL5/11/base/x86_64/getPackage/screen-4.0.3-4.el5.x86_64.rpm

Oracle Linux 6, Update 8:

http://public-yum.oracle.com/repo/OracleLinux/OL6/8/base/x86_64/getPackage/screen-4.0.3-19.el6.x86_64.rpm

Starting Linux Screen

To start Linux Screen:

[root@v1ex1dbadm01 ~]# screen

Linux Screen Commands

Linux Screen commands are used by first pressing “CTRL” then “a“, followed by command key:

Example: “CTRL” then “a” then “?” get you the following help screen:

Screen key bindings, page 1 of 1.

Command key: ^A Literal ^A: a

break ^B b displays * help ? lockscreen ^X x number N quit \ screen ^C c title A writebuf >
clear C dumptermcap . history { } log H only Q readbuf < select ' vbell ^G xoff ^S s
colon : fit F info i login L other ^A redisplay ^L l silence _ version v xon ^Q q
copy ^[ [ flow ^F f kill K k meta a pow_break B remove X split S width W
detach ^D d focus ^I lastmsg ^M m monitor M pow_detach D removebuf = suspend ^Z z windows ^W w
digraph ^V hardcopy h license , next ^@ ^N sp n prev ^H ^P p ^? reset Z time ^T t wrap ^R r

^] paste .
" windowlist -b
- select -
0 select 0
1 select 1
2 select 2
3 select 3
4 select 4
5 select 5
6 select 6
7 select 7
8 select 8
9 select 9
I login on
O login off
] paste .


 [Press Space or Return to end.]

Detaching from Linux Screen

To detach from Linux Screen:

CTRL” then “a” then “d

[root@v1ex1dbadm01 ~]# echo "Detaching from Linux Screen, leaving my mark :)"
Detaching from Linux Screen, leaving my mark :)
[detached]

You are at this point returned to your normal shell.

Please Note: network break, accidental close of terminal, the machine running the terminal crashes, etc, automatically detach 🙂

Reattaching to Linux Screen

To reattach to Linux Screen:

[root@v1ex1dbadm01 ~]# screen -r

You’ll now be reattached to you last session:

[root@v1ex1dbadm01 ~]# echo "Detaching from Linux Screen, leaving my mark :)"
Detaching from Linux Screen, leaving my mark :)
[root@v1ex1dbadm01 ~]#

To reattach to Linux Screen, when multiple sessions are detached, use screen -r :

[root@v1ex1dbadm01 ~]# screen -r
There are several suitable screens on:
 15090.pts-0.v1ex1dbadm01 (Detached)
 346219.pts-0.v1ex1dbadm01 (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
[root@v1ex1dbadm01 ~]# screen -r 346219.pts-0.v1ex1dbadm01

Terminating Linux Screen

Once you’re finished, just type ‘exit‘ and you’ll exit back to your normal shell:

[screen is terminating]
[root@v1ex1dbadm01 ~]# screen -r
There is no screen to be resumed.
[root@v1ex1dbadm01 ~]#

You can also use:

CTRL” then “a” then “k“:

Really kill this window [y/n]y
[screen is terminating]

Advance Uses

Creating multiple screen in one session

You can add screen to your current session by:

CTRL” then “a” then “c

You’ll notice your putty session will say something like:

[screen 1:bash]root@v1ex1dbadm01:~

Screen ID starts with 0 and increases by 1 for every new screen created.

Switching between multiple screens in one session

You can switch between multiple screens, by going next and previous:

Next: “CTRL” then “a” then “n

Previous: “CTRL” then “a” then “p

You’ll notice the screen ID change as well as the screen.

Closing is the same as terminating Linux Screen, either type ‘exit‘ or “CTRL” then “a” then “k“.  You’ll need to do this for every screen until all are closed and then you will get:

[screen is terminating]

Known Issues

Cannot open your terminal

If you get the following error message:

[oracle@v1ex1dbadm01 ~]$ screen
Cannot open your terminal '/dev/pts/0' - please check.

Simple run Linux Screen as the user ‘root’ and then su to the desired user:

[root@v1ex1dbadm01 ~]# screen
[root@v1ex1dbadm01 ~]# su - oracle
[oracle@v1ex1dbadm01 ~]$

Scrollbar doesn’t work

You can enter copy mode (not sure why it’s call that, but it lets you navigate):

CTRL” then “a” then “esc” then “up“/”down

Or even better add the follow line to the root ~/.screenrc file:

termcapinfo xterm ti@:te@

If the file doesn’t exist, create it and add the line.

Now you can use scrollbar as normal 🙂

 

There are many more features, which I haven’t yet explored, but now you’ve got the idea, you’ll be able to use ‘Linux Screen’ to your benefit 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)