KBHS-00600: Internal Error, Arguments [1] [kbhshtCreateDataBucket] Error During Backup To Oracle Cloud

When setting up Oracle Cloud Backup Service for first time and the Oracle Database Cloud Backup Module has been installed successfully using:
Installing the Oracle Database Cloud Backup Module

Your RMAN session will hang and eventually give the following error:

Starting backup at 2017/08/10 12:56:58 
current log archived 
RMAN-00571: =========================================================== 
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== 
RMAN-00571: =========================================================== 
RMAN-03002: failure of backup plus archivelog command at 08/10/2017 16:20:05 
ORA-19554: error allocating device, device type: SBT_TAPE, device name: 
ORA-27023: skgfqsbi: media manager protocol error 
ORA-19511: non RMAN, but media manager or vendor specific failure, error text: 
KBHS-00600: internal error, arguments [1] [kbhshtCreateDataBucket] [] [] []

If the replication policy has not been set for this account, as explained in the MOS Note:
KBHS-00600: Internal Error, Arguments [1] [kbhshtCreateDataBucket] Error During Backup To Cloud (Doc ID 2232778.1)

Details of the replication policy can be found in the following documentation:
Selecting a Replication Policy for Your Service Instance

“Policies that have no georeplication:

These policies specify only the primary data center (DC) that hosts your service instance.

All read and write requests go to the primary DC, always. If the primary DC is unavailable, then the requests fail.

 Such a policy may be adequate if you have standard data-durability requirements and if an occasional failure of read requests (when the primary DC is down) is acceptable.

The Georeplication policies:

These policies specify a primary DC that hosts your service instance as well as a geographically distant, georeplication DC.

Write requests that you send to the global namespace URL are routed to the primary DC. Data that you write is replicated automatically, but asynchronously, to the georeplication DC. The primary and secondary DCs are eventually consistent.

 If the primary DC is unavailable, then write requests fail with the 403 – Forbidden error, but read requests are routed to the georeplication DC. When the primary DC is available again, requests to the global namespace URL are routed to the primary DC.

 You’ll be billed for the sum of the capacities used in both DCs and for the data transfer from the primary to the georeplication DC.

 A policy that has a georeplication DC is ideal if you have advanced durability requirements for your data or if read requests must succeed always regardless of the state of the primary DC.”

The first option, there’s a possibility of not being able to reach your backups but has the benefit of 1:1 ratio on storage used.   The second option has the benefit of guarantees being able to reach your backups but double storage usage and you are charged for data transfer between DC.

Once you select your georeplication policy, your backup will work 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Advertisements

Delete Old SBT_TAPE Backups that are No Longer Available in RMAN

When in the situation of having ‘SBT_TAPE’ backups, that are still known by the catalog or control file, but the device type ‘SBT_TAPE’ is no longer available.  If you try to delete the backups you get the following error message:

RMAN> list backup summary;

List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
11208 B 0 A SBT_TAPE 20-JUL-17 1 1 YES TAG20170720T200232
11209 B 0 A SBT_TAPE 20-JUL-17 1 1 YES TAG20170720T200232
11210 B 0 A SBT_TAPE 20-JUL-17 1 1 YES TAG20170720T200232
...
11356 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11357 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11358 B A A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T201725

RMAN> delete backup;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of delete command at 08/10/2017 12:49:55
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2

 

This is because the backups are of device type ‘SBT_TAPE’, which is no longer available.  To remove, you need to allocate channel for maintenance of device type ‘SBT’ but dummy to disk as explained in MOS Note:

Need Help To Remove Old Tape Backups From Catalog (Doc ID 342284.1)

GOAL

How to delete obsolete TAPE backups via RMAN, when the Media Management Layer (MML Software not installed) is NOT available anymore.

So how to delete OLD backups which have been made to tape, while the current backup strategy is to disk and NO Media Manager is available.

SOLUTION

Allocate a maintenance channel with the dummy sbt API and run the DELETE FORCE OBSOLETE.

RMAN> allocate channel for maintenance device type sbt
parms 'SBT_LIBRARY=oracle.disksbt, ENV=(BACKUP_DIR=/tmp)';


RMAN> delete force obsolete;

The DUMMY API (oracle.disksbt) is simulating the callout to the Media Management Layer (MML).  This way RMAN thinks an actual MML is available and can perform the maintenance.”

My scenario:

RMAN> allocate channel for maintenance device type sbt
2> parms 'SBT_LIBRARY=oracle.disksbt, ENV=(BACKUP_DIR=/tmp)';

allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: SID=808 device type=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: WARNING: Oracle Test Disk API

RMAN> delete force obsolete;

RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 8 days
Deleting the following obsolete backups and copies:
Type Key Completion Time Filename/Handle
-------------------- ------ ------------------ --------------------
Backup Set 11217 20-JUL-17
 Backup Piece 11217 20-JUL-17 vks9rkqo_1_1
Backup Set 11214 20-JUL-17
 Backup Piece 11214 20-JUL-17 vls9rkqo_1_1
Backup Set 11216 20-JUL-17
 Backup Piece 11216 20-JUL-17 vms9rkqo_1_1
...
 Backup Piece 11341 28-JUL-17 3tsagnmq_1_1
Backup Set 11346 28-JUL-17
 Backup Piece 11346 28-JUL-17 3usagnmq_1_1
Backup Set 11340 28-JUL-17
 Backup Piece 11340 28-JUL-17 3vsagnmq_1_1

Do you really want to delete the above objects (enter YES or NO)? yes
deleted backup piece
backup piece handle=vks9rkqo_1_1 RECID=11217 STAMP=949867353
deleted backup piece
backup piece handle=vls9rkqo_1_1 RECID=11214 STAMP=949867353
deleted backup piece
backup piece handle=vms9rkqo_1_1 RECID=11216 STAMP=949867353
...
deleted backup piece
backup piece handle=3tsagnmq_1_1 RECID=11341 STAMP=950558426
deleted backup piece
backup piece handle=3usagnmq_1_1 RECID=11346 STAMP=950558426
deleted backup piece
backup piece handle=3vsagnmq_1_1 RECID=11340 STAMP=950558426
Deleted 132 objects

RMAN>

 

You may still have some ‘SBT_TAPE’ backups remaining due to your retention policy and hence the backups are not obsolete, therefore you need to crosscheck as follows:

RMAN> list backup summary;

List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
11348 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11349 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11350 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11351 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11352 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11353 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11354 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11355 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11356 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11357 B 0 A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T200202
11358 B A A SBT_TAPE 28-JUL-17 1 1 YES TAG20170728T201725

RMAN> allocate channel for maintenance device type sbt
2> parms 'SBT_LIBRARY=oracle.disksbt, ENV=(BACKUP_DIR=/tmp)';

allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: SID=17 device type=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: WARNING: Oracle Test Disk API

RMAN> delete backup;

List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
11348 11348 1 1 AVAILABLE SBT_TAPE 46sagnpr_1_1
11349 11349 1 1 AVAILABLE SBT_TAPE 47sagnpr_1_1
11350 11350 1 1 AVAILABLE SBT_TAPE 44sagnpr_1_1
11351 11351 1 1 AVAILABLE SBT_TAPE 48sagnqb_1_1
11352 11352 1 1 AVAILABLE SBT_TAPE 49sagnqb_1_1
11353 11353 1 1 AVAILABLE SBT_TAPE 45sagnpr_1_1
11354 11354 1 1 AVAILABLE SBT_TAPE 41sagnpr_1_1
11355 11355 1 1 AVAILABLE SBT_TAPE 43sagnpr_1_1
11356 11356 1 1 AVAILABLE SBT_TAPE 42sagnpr_1_1
11357 11357 1 1 AVAILABLE SBT_TAPE 40sagnpr_1_1
11358 11358 1 1 AVAILABLE SBT_TAPE 4asagoml_1_1

Do you really want to delete the above objects (enter YES or NO)? yes

RMAN-06207: WARNING: 11 objects could not be deleted for SBT_TAPE channel(s) due
RMAN-06208: to mismatched status. Use CROSSCHECK command to fix status
RMAN-06210: List of Mismatched objects
RMAN-06211: ==========================
RMAN-06212: Object Type Filename/Handle
RMAN-06213: --------------- ---------------------------------------------------
RMAN-06214: Backup Piece 46sagnpr_1_1
RMAN-06214: Backup Piece 47sagnpr_1_1
RMAN-06214: Backup Piece 44sagnpr_1_1
RMAN-06214: Backup Piece 48sagnqb_1_1
RMAN-06214: Backup Piece 49sagnqb_1_1
RMAN-06214: Backup Piece 45sagnpr_1_1
RMAN-06214: Backup Piece 41sagnpr_1_1
RMAN-06214: Backup Piece 43sagnpr_1_1
RMAN-06214: Backup Piece 42sagnpr_1_1
RMAN-06214: Backup Piece 40sagnpr_1_1
RMAN-06214: Backup Piece 4asagoml_1_1

RMAN> crosscheck backup;

crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=46sagnpr_1_1 RECID=11348 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=47sagnpr_1_1 RECID=11349 STAMP=950558524
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=44sagnpr_1_1 RECID=11350 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=48sagnqb_1_1 RECID=11351 STAMP=950558539
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=49sagnqb_1_1 RECID=11352 STAMP=950558539
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=45sagnpr_1_1 RECID=11353 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=41sagnpr_1_1 RECID=11354 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=43sagnpr_1_1 RECID=11355 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=42sagnpr_1_1 RECID=11356 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=40sagnpr_1_1 RECID=11357 STAMP=950558523
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=4asagoml_1_1 RECID=11358 STAMP=950559445
Crosschecked 11 objects

RMAN> delete expired backup;

List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
11348 11348 1 1 EXPIRED SBT_TAPE 46sagnpr_1_1
11349 11349 1 1 EXPIRED SBT_TAPE 47sagnpr_1_1
11350 11350 1 1 EXPIRED SBT_TAPE 44sagnpr_1_1
11351 11351 1 1 EXPIRED SBT_TAPE 48sagnqb_1_1
11352 11352 1 1 EXPIRED SBT_TAPE 49sagnqb_1_1
11353 11353 1 1 EXPIRED SBT_TAPE 45sagnpr_1_1
11354 11354 1 1 EXPIRED SBT_TAPE 41sagnpr_1_1
11355 11355 1 1 EXPIRED SBT_TAPE 43sagnpr_1_1
11356 11356 1 1 EXPIRED SBT_TAPE 42sagnpr_1_1
11357 11357 1 1 EXPIRED SBT_TAPE 40sagnpr_1_1
11358 11358 1 1 EXPIRED SBT_TAPE 4asagoml_1_1

Do you really want to delete the above objects (enter YES or NO)? yes
deleted backup piece
backup piece handle=46sagnpr_1_1 RECID=11348 STAMP=950558523
deleted backup piece
backup piece handle=47sagnpr_1_1 RECID=11349 STAMP=950558524
deleted backup piece
backup piece handle=44sagnpr_1_1 RECID=11350 STAMP=950558523
deleted backup piece
backup piece handle=48sagnqb_1_1 RECID=11351 STAMP=950558539
deleted backup piece
backup piece handle=49sagnqb_1_1 RECID=11352 STAMP=950558539
deleted backup piece
backup piece handle=45sagnpr_1_1 RECID=11353 STAMP=950558523
deleted backup piece
backup piece handle=41sagnpr_1_1 RECID=11354 STAMP=950558523
deleted backup piece
backup piece handle=43sagnpr_1_1 RECID=11355 STAMP=950558523
deleted backup piece
backup piece handle=42sagnpr_1_1 RECID=11356 STAMP=950558523
deleted backup piece
backup piece handle=40sagnpr_1_1 RECID=11357 STAMP=950558523
deleted backup piece
backup piece handle=4asagoml_1_1 RECID=11358 STAMP=950559445
Deleted 11 EXPIRED objects

RMAN>

Now you can continue to use device type ‘DISK’ or configure a new ‘SBT_TAPE’ 🙂

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

RMAN-06403: could not obtain a fully authorized session

An automated process that refreshes a Development database from Production by doing a RMAN duplicate failed with ‘RMAN-06403: could not obtain a fully authorized session’:

Starting Duplicate Db at 25-JUL-2017 17:44
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 07/25/2017 17:44:46
RMAN-05501: aborting duplication of target database
RMAN-06403: could not obtain a fully authorized session
RMAN-04006: error from auxiliary database: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3651
Additional information: 1054940735

 

This was initially puzzling as nothing appeared to have changed.  A ‘My Oracle Support’ (MOS) search, revealed the following MOS note:

Duplicating database using RMAN fails with errors RMAN-06403, RMAN-04006, ORA-01034, ORA-27101 (Doc ID 274233.1)

CAUSE

The errors are occurring because the auxiliary database is not started in nomount state.

SOLUTION

The auxiliary instance must be in a nomount state during duplication of database using RMAN.

Startup the auxiliary instance in nomount state.  Then try to duplicate the database again.

SQL> connect / as sysdba
startup nomount;

Also you have to verify the following:

1. Ensure that the ORACLE_SID and ORACLE_HOME were set correctly when starting the auxiliary instance.  Especially any additional backslash at the end of the ORACLE_HOME can cause mismatches

2. Verify that the ORACLE_HOME directory is not a linked directory.
Make a hard copy of the init.ora in the ORACLE_HOME/dbs directory.

3. Double check the ‘listener.ora’ and ‘tnsnames.ora’ for correctness.”

 

The database was in nomount state started with a basic pfile:

*.db_name=V1DEV
*.db_unique_name=V1DEV

 

But upon checking point 3, I noticed the static listener registration that is required as the database is in nomount and thus doesn’t register with the listener had the wrong home:

SID_LIST_LISTENER =
 (SID_LIST =
 (SID_DESC =
 (SID_NAME = V1DEV1)
 (GLOBAL_DBNAME = V1DEV)
 (ORACLE_HOME = /u01/app/oracle/product/12.1.0.2/dbhome_1)
 )
 )

 

I recently changed the Oracle Home to use the dbhome_2 Oracle Home, so that the Development databases were not locked in with the DR databases using the dbhome_1 Oracle Home.  This allows for independent patching between the Oracle Homes, which allows soak testing of patches in Development before applying to Production:

Cloning an Oracle Home

 

Once this was changed:

SID_LIST_LISTENER =
 (SID_LIST =
 (SID_DESC =
 (SID_NAME = V1DEV1)
 (GLOBAL_DBNAME = V1DEV)
 (ORACLE_HOME = /u01/app/oracle/product/12.1.0.2/dbhome_2)
 )
 )

 

It started working again 🙂 :

Starting Duplicate Db at 26-JUL-2017 11:38
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=515 device type=DISK
allocated channel: ORA_AUX_DISK_2
channel ORA_AUX_DISK_2: SID=1475 device type=DISK
allocated channel: ORA_AUX_DISK_3
channel ORA_AUX_DISK_3: SID=643 device type=DISK
allocated channel: ORA_AUX_DISK_4
channel ORA_AUX_DISK_4: SID=835 device type=DISK
current log archived

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)