KBHS-00600: Internal Error, Arguments [1] [kbhshtCreateDataBucket] Error During Backup To Oracle Cloud

When setting up Oracle Cloud Backup Service for first time and the Oracle Database Cloud Backup Module has been installed successfully using:
Installing the Oracle Database Cloud Backup Module

Your RMAN session will hang and eventually give the following error:

Starting backup at 2017/08/10 12:56:58 
current log archived 
RMAN-00571: =========================================================== 
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== 
RMAN-00571: =========================================================== 
RMAN-03002: failure of backup plus archivelog command at 08/10/2017 16:20:05 
ORA-19554: error allocating device, device type: SBT_TAPE, device name: 
ORA-27023: skgfqsbi: media manager protocol error 
ORA-19511: non RMAN, but media manager or vendor specific failure, error text: 
KBHS-00600: internal error, arguments [1] [kbhshtCreateDataBucket] [] [] []

If the replication policy has not been set for this account, as explained in the MOS Note:
KBHS-00600: Internal Error, Arguments [1] [kbhshtCreateDataBucket] Error During Backup To Cloud (Doc ID 2232778.1)

Details of the replication policy can be found in the following documentation:
Selecting a Replication Policy for Your Service Instance

“Policies that have no georeplication:

These policies specify only the primary data center (DC) that hosts your service instance.

All read and write requests go to the primary DC, always. If the primary DC is unavailable, then the requests fail.

 Such a policy may be adequate if you have standard data-durability requirements and if an occasional failure of read requests (when the primary DC is down) is acceptable.

The Georeplication policies:

These policies specify a primary DC that hosts your service instance as well as a geographically distant, georeplication DC.

Write requests that you send to the global namespace URL are routed to the primary DC. Data that you write is replicated automatically, but asynchronously, to the georeplication DC. The primary and secondary DCs are eventually consistent.

 If the primary DC is unavailable, then write requests fail with the 403 – Forbidden error, but read requests are routed to the georeplication DC. When the primary DC is available again, requests to the global namespace URL are routed to the primary DC.

 You’ll be billed for the sum of the capacities used in both DCs and for the data transfer from the primary to the georeplication DC.

 A policy that has a georeplication DC is ideal if you have advanced durability requirements for your data or if read requests must succeed always regardless of the state of the primary DC.”

The first option, there’s a possibility of not being able to reach your backups but has the benefit of 1:1 ratio on storage used.   The second option has the benefit of guarantees being able to reach your backups but double storage usage and you are charged for data transfer between DC.

Once you select your georeplication policy, your backup will work 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to easily delete files in the Oracle Cloud using CloudBerry Explorer

So you have some Oracle Cloud storage, which was probably thrown in as a freebie initially by Oracle 🙂  Now your freebie is expiring and you decide you want to retain the service but now as metered, so you want to delete all the unnecessary files, which in my case are old Oracle database backups.

As you can see here I have 50k plus files using 897Gb:

Oracle Cloud Web Console Details

Here are the files listed in the Web Console:

Oracle Cloud Web Console List Objects

To delete them one by one isn’t feasible.

So the solution is to use a 3rd party file explorer, in my case CloudBerry Explorer from CloudBerry Labs:

https://www.cloudberrylab.com/explorer/openstack.aspx

The freeware version is perfectly fine, no need to purchase Pro or use the trial.  Just click ‘Cancel’ on the Register Product dialogue and then the application will load.

Once installed, to connect to your Oracle Cloud storage, you can follow this link:

https://www.cloudberrylab.com/blog/how-to-use-cloudberry-explorer-with-oracle-cloud-storage/

However, the key to connecting is to have the username in the format of:

<Your Oracle Cloud Service Instance Name>-<Your Oracle Cloud Identity Domain>:<Your Oracle Cloud User Name>

e.g. zeddbacloud-zeddba:oraclecloudbackup@zeddba.com

Also select the correct ‘Account location’ which will fill the ‘Authentication Service’:

CloudBerry Login

Keystone, set to ‘Do not use’.

When you finally manage to get connected, you’ll see something like this:

CloudBerry Explorer

Now you have the freedom, to browse your files and delete at leisure 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Oracle Database Backup Service Fails with: ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’

I discovered an Oracle Cloud Database Backup failing with:

Starting backup at 2017/01/20 20:00:04
current log archived
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 01/20/2017 20:00:08
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27023: skgfqsbi: media manager protocol error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
 KBHS-00715: HTTP error occurred 'oracle-error'
KBHS-00712: ORA-28750 received from local HTTP service

Recovery Manager complete.

Upon investigation I found the following metalink note:
Oracle Database Backup Service Fails with: ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’ (Doc ID 2093475.1)

The issue is caused by Bug 22481516 – ORA-19511: – KBHS-00715: HTTP error occurred ‘oracle-error’

The fix for Bug 22481516  has been  included in the latest Cloud Backup Module.  The Linux 64 bit SBT Library has been updated with the fix as well.

So the solution is to install the latest Cloud Backup Module as follows:

1. Download the latest opc_installer.zip from:
http://www.oracle.com/technetwork/database/availability/oracle-cloud-backup-2162729.html
2. Install the latest Cloud Backup Module:

 [oracle@V1LOEM ~]$ cd /u01/oracle_stage/cloud/
 [oracle@V1LOEM cloud]$ ls -ltrh
 total 4.9M
 -rw-r--r--. 1 oracle oinstall 9.8K Nov 28 2014 opc_readme.txt
 -rw-r--r--. 1 oracle oinstall 2.5M May 12 2015 opc_install.jar
 -rw-r--r--. 1 oracle oinstall 2.4M Sep 9 2015 opc_installer.zip
 [oracle@V1LOEM cloud]$ mv opc_installer.zip opc_installer.zip.old

scp the latest opc_installer.zip to the server.

 [oracle@V1LOEM cloud]$ unzip opc_installer.zip
 Archive: opc_installer.zip
 replace opc_install.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 inflating: opc_install.jar
 replace opc_readme.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
 inflating: opc_readme.txt
 [oracle@V1LOEM cloud]$ ls -l
 total 7596
 -rw-r--r--. 1 root root 2591749 Jan 25 14:16 opc_installer.zip
 -rw-r--r--. 1 oracle oinstall 2452163 Sep 9 2015 opc_installer.zip.old
 -rw-r--r--. 1 oracle oinstall 2718012 Oct 7 07:16 opc_install.jar
 -rw-r--r--. 1 oracle oinstall 10414 Oct 6 16:21 opc_readme.txt
 [oracle@V1LOEM cloud]$ . oraenv
 ORACLE_SID = [oracle] ? EMREPOS
 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 is /u01/app/oracle
 [oracle@V1LOEM cloud]$ java -jar opc_install.jar -serviceName v1cloud -identityDomain v1 -opcId 'oraclecloudbackup@version1.com' -opcPass 'xxx' -walletDir /u01/oracle/opc_wallet -libDir $ORACLE_HOME/lib -libPlatform linux64 -debug
 Oracle Database Cloud Backup Module Install Tool, build 2016-10-07
 Debug: os.name = Linux
 Debug: os.arch = amd64
 Debug: os.version = 3.8.13-98.1.2.el6uek.x86_64
 Debug: file.separator = /
 Debug: Platform = PLATFORM_LINUX64
 Debug: content >> oracle-data-v1clou-173484218880202412224280.087677
 Oracle Database Cloud Backup Module credentials are valid.
 Debug: Certificate Success:
 Subject : CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
 Validity : Wed Nov 08 00:00:00 GMT 2006 - Thu Jul 17 00:59:59 IST 2036
 Issuer : CN=VeriSign Class 3 Public Primary Certification Authority - G5, OU="(c) 2006 VeriSign, Inc. - For authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
 Oracle Database Cloud Backup Module wallet created in directory /u01/oracle/opc_wallet.
 Oracle Database Cloud Backup Module initialization file /u01/app/oracle/product/12.1.0/dbhome_1/dbs/opcEMREPOS.ora created.
 Downloading Oracle Database Cloud Backup Module Software Library from file opc_linux64.zip.
 Debug: Temp zip file = /tmp/opc_linux641742885153833539199.zip
 Downloaded 26528348 bytes in 17 seconds. Transfer rate was 1560491 bytes/second.
 Download complete.
 Debug: Delete RC = true

Now test the latest Cloud Backup Module:

[oracle@V1LOEM cloud]$ rman target /
 Recovery Manager: Release 12.1.0.2.0 - Production on Wed Jan 25 14:21:29 2017

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

connected to target database: EMREPOS (DBID=2214718289)

RMAN> delete obsolete recovery window of 8 days device type sbt;

using target database control file instead of recovery catalog
 allocated channel: ORA_SBT_TAPE_1
 channel ORA_SBT_TAPE_1: SID=818 device type=SBT_TAPE
 channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_2
 channel ORA_SBT_TAPE_2: SID=411 device type=SBT_TAPE
 channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_3
 channel ORA_SBT_TAPE_3: SID=1159 device type=SBT_TAPE
 channel ORA_SBT_TAPE_3: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_4
 channel ORA_SBT_TAPE_4: SID=43 device type=SBT_TAPE
 channel ORA_SBT_TAPE_4: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_5
 channel ORA_SBT_TAPE_5: SID=787 device type=SBT_TAPE
 channel ORA_SBT_TAPE_5: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_6
 channel ORA_SBT_TAPE_6: SID=42 device type=SBT_TAPE
 channel ORA_SBT_TAPE_6: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_7
 channel ORA_SBT_TAPE_7: SID=41 device type=SBT_TAPE
 channel ORA_SBT_TAPE_7: Oracle Database Backup Service Library VER=3.16.9.21
 allocated channel: ORA_SBT_TAPE_8
 channel ORA_SBT_TAPE_8: SID=1163 device type=SBT_TAPE
 channel ORA_SBT_TAPE_8: Oracle Database Backup Service Library VER=3.16.9.21
 Deleting the following obsolete backups and copies:
 Type Key Completion Time Filename/Handle
 -------------------- ------ ------------------ --------------------
Backup Set 7986 16-JAN-17
 Backup Piece 7986 16-JAN-17 qsrq6lep_1_1
 Backup Set 7990 16-JAN-17
 Backup Piece 7990 16-JAN-17 qtrq6lep_1_1
 Backup Set 7987 16-JAN-17
 Backup Piece 7987 16-JAN-17 qurq6lep_1_1
 Backup Set 7989 16-JAN-17
 Backup Piece 7989 16-JAN-17 qvrq6lep_1_1
 Backup Set 7984 16-JAN-17
 Backup Piece 7984 16-JAN-17 r0rq6lep_1_1
 Backup Set 7988 16-JAN-17
 Backup Piece 7988 16-JAN-17 r1rq6lep_1_1
 Backup Set 7985 16-JAN-17
 Backup Piece 7985 16-JAN-17 r2rq6lep_1_1

Do you really want to delete the above objects (enter YES or NO)? no

RMAN> exit
 Recovery Manager complete.
 [oracle@V1LOEM cloud]$

Everything working again 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)