SNMP unresponsive on Storage Cell after Exadata Patching

Posted on by Standard 3

After a round of Exadata patching, the Storage Cells become unresponsive to SNMP for monitoring tools like OpsViews:

check_snmp_sysinfo CRITICAL - Agent not responding, tried SNMP v1 and v2c

This is because the Storage Cells get re-imaged as part of the Exadata patching, confirmed by imagehistory:

[root@v1ex1celadm01 ~]# imagehistory
Version : 12.1.2.1.1.150316.2
Image activation date : 2015-05-01 16:10:10 -0700
Imaging mode : fresh
Imaging status : success

Version : 12.1.2.1.3.151021
Image activation date : 2015-12-16 05:25:21 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.2.1.160330
Image activation date : 2016-05-12 14:02:43 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.2.160721
Image activation date : 2016-10-05 01:45:42 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.3.161208
Image activation date : 2017-01-18 02:53:10 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.4.170111
Image activation date : 2017-05-23 10:58:24 +0100
Imaging mode : out of partition upgrade
Imaging status : success

[root@v1oex1celadm01 ~]#

Which knocks out the configuration in iptables and snmpd.conf.

To resolve add back in the lines for your primary and secondary monitoring servers, for example:

rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32

So looks like this:

[root@v1ex1celadm01 ~]# more /etc/snmp/snmpd.conf
trapcommunity public
trapsink 127.0.0.1 public
rocommunity public 127.0.0.1
rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32
rwcommunity public 127.0.0.1

access RWGroup "" any noauth exact all all all
com2sec snmpclient 127.0.0.1 public
group RWGroup v1 snmpclient

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
syslocation Unknown (edit /etc/snmp/snmpd.conf)

view all included .1 80

[root@v1ex1celadm01 ~]#

Then reload snmp using the user root:

[root@v1ex2celadm01 ~]# service snmpd reload
Reloading snmpd: [ OK ]
[root@v1ex2celadm01 ~]#

Next add the monitoring servers primary and secondary to iptables as follows:

[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.31 -p udp --dport 161 -j ACCEPT
[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.32 -p udp --dport 161 -j ACCEPT

And make permanent by saving to firewall rules:

[root@v1ex1dbadm01 ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@v1ex1dbadm01 ~]#

Now your monitoring should return as SNMP connectivity is restored 🙂 :

SNMP OK - "Linux v1ex1celadm01.v1.com 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64"

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to easily delete files in the Oracle Cloud using CloudBerry Explorer

Posted on by Standard Reply

So you have some Oracle Cloud storage, which was probably thrown in as a freebie initially by Oracle 🙂  Now your freebie is expiring and you decide you want to retain the service but now as metered, so you want to delete all the unnecessary files, which in my case are old Oracle database backups.

As you can see here I have 50k plus files using 897Gb:

Oracle Cloud Web Console Details

Here are the files listed in the Web Console:

Oracle Cloud Web Console List Objects

To delete them one by one isn’t feasible.

So the solution is to use a 3rd party file explorer, in my case CloudBerry Explorer from CloudBerry Labs:

https://www.cloudberrylab.com/explorer/openstack.aspx

The freeware version is perfectly fine, no need to purchase Pro or use the trial.  Just click ‘Cancel’ on the Register Product dialogue and then the application will load.

Once installed, to connect to your Oracle Cloud storage, you can follow this link:

https://www.cloudberrylab.com/blog/how-to-use-cloudberry-explorer-with-oracle-cloud-storage/

However, the key to connecting is to have the username in the format of:

<Your Oracle Cloud Service Instance Name>-<Your Oracle Cloud Identity Domain>:<Your Oracle Cloud User Name>

e.g. zeddbacloud-zeddba:oraclecloudbackup@zeddba.com

Also select the correct ‘Account location’ which will fill the ‘Authentication Service’:

CloudBerry Login

Keystone, set to ‘Do not use’.

When you finally manage to get connected, you’ll see something like this:

CloudBerry Explorer

Now you have the freedom, to browse your files and delete at leisure 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

UKOUG Northern Technology SIG – Manchester – April 2017

Posted on by Standard 2

So yesterday was the UKOUG Northern Technology SIG in the Renaissance Hotel in Manchester.  This SIG combined the: 1. Database, 2. RAC Cloud, Infrastructure & Availability and 3. Systems streams into one.

I was approached by the committee a couple months to present, so I offered to present about Exadata Patching or Upgrading Grid Infrastructure.  The latter got accepted, so I knew I was definitely going to attend this SIG.

So the morning of the SIG, I took the 15 minutes or so taxi ride to the hotel (Manchester being my home town 🙂 ) and arrived in good time to have a cup of tea before the morning sessions began.

Patrick Hurley kicked off the day with the ‘Welcome & Introduction’ session in Neil Chandler‘s absence due to family emergency (hope everything is OK Neil).  Next Dominic Giles from Oracle gave the Keynotes session, which the take away from me was:
1. Oracle are investing a lot effort into a) REST APIs and b) Docker
2. Quite a few under the radar features in 12c
3. 12.2.0.2 planned release is end of 2017 with features like OLTP fast cache, support for NVRAM, merge partitions online, AD integration and more but I’m not quoting (Oracle Safe Harbour and that 😉 )

Next was the Vendor awareness sessions, good idea UKOUG.  I like this, thumbs up, just need to get Version 1 up there 😉

Next was Michael Hartley from Oracle who gave an Oracle Support session.  Something he mentioned that I didn’t know, you can in MOS analyse your inventory for patch conflict by uploading text output of your inventory and it will check for conflicts for a given patch.  Pretty cool, I will need to try this out and do a blog post on it 🙂

It was then a half hour coffee break before the stream presentations began.  So I took this opportunity to recheck my slides and make some minor amendments.  Before jumping in to my first stream session ‘Execution Plan Stability & How to Get Creative with SQL Profiles & SQL Plan Management’ by Nigel Bayliss from Oracle.  Quite a good session with my take away being the advice of replacing SQL Profiles with SQL Baselines Plans, so it’s not just a temporary fix but one that can withstand time because new better plans can evolved and be accepted.

Next was a pretty decent lunch, which I got the time to catch up with my ex colleague and good friend Mohsin Patel from Auto Trader.  Then it was fast approaching time for me to present my presentation of ‘Upgrading Grid Infrastructure in 5 Simple Steps’.  So I headed off to my room to setup and get ready.  I was worried I might have been presenting to 1 or 2 people because Jonathan Lewis was presenting at the same time about ‘Getting Rid of Indexes’.  Luckily I had what looked like more then 20 people, which was pretty good for a SIG, so I was pleased.  Presentation went well I believe, with the audience engaged and listening.  Just a few questions at the end and then my duty was done and I could then sit back and enjoy the rest of the SIG.  Thanks to those who attended, my slides will be up on this blog in next few days.

I then stayed in the room I just presented in to listen to Pete Finnigan‘s presentation on ‘Oracle Incident Response & Forensic Analysis’.  This session was a bit of an eye opener and made me realise, we all need to look into security a bit more then we do.

The next session was by Chris Buckel aka FlashDBA: Why Your Database Runs Slowly (& How to Fix it).  Quite enjoyed this session especially the demo of the poor guy who had to go round collecting plastic balls from each person in order, then randomly and then finally everyone throwing them at him, sorry I meant the bag to illustrate sequential, random and flash I/O 🙂

The final session of the day I attended was by my colleagues Chris Slattery – Oracle IaaS “Where the Rubber Meets the Road”.  Pretty good session and I was surprised with the comment of Oracle having 116 Oracle Cloud services! 😐

Then it was the social drinks sponsored by Oracle, got to catch up with Joy Marshall and Martin Widlake.  All in all, was a great day.  Many thanks to UKOUG for putting together a great SIG and thanks to all the speakers.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks for reading

Zed DBA (Zahid Anwar)

Oracle licensing in the cloud and what the recent changes mean for you?

Posted on by Standard 1

At the end of January 2017, Oracle updated its Licensing Oracle Software in the Cloud Computing Environment Document, which effectively removed the core factor 0.5 for Intel and instead made it 1:1, doubling the licensing costs overnight!

For reference, the old document can be found here.

You can imagine this caused a major stir on Social Media, where my friend Tim Hall posts on this topic became very hot!  You can read them here:
1. https://oracle-base.com/blog/2017/01/28/oracles-cloud-licensing-change-be-warned/
2. https://oracle-base.com/blog/2017/02/09/oracle-sad-times/

I’m certainly no Cloud license expert, but fortunately my colleagues in the SAM (Software Assess Management) Team have written a great White Paper to add clarity to the matter:
Oracle Authorized Cloud Environments Overview of Policy Changes

Feel free to take a read.

You can follow V1 SAM Twitter here.
And get more info here.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

ORA-00322: log 41 of thread 2 is not current copy

Posted on by Standard 1

On a Data Guard Physical Standby, I noticed the following in the alert log:

Fri Jan 13 08:12:39 2017 
Errors in file /u01/app/oracle/diag/rdbms/v1s/V1S1/trace/V1S1_arc1_399861.trc: 
ORA-00322: log 41 of thread 2 is not current copy 
ORA-00312: online log 41 thread 2: '+RECOC1/V1S/ONLINELOG/group_41.3099.912775569' 
ORA-00322: log 41 of thread 2 is not current copy 
ORA-00312: online log 41 thread 2: '+DATAC1/V1S/ONLINELOG/group_41.807.912775567'

If you read metalink note:
Physical standby – alert*log shows ORA-322, ORA-312 intermittently (Doc ID 1592057.1)

SYMPTOMS

alert.log shows intermittently ORA-322 and ORA-312 for SRLs (standby redo logs).

CAUSE

These errors are written due to a very small timing window, when the SRL header
is being updated to reflect new content from the primary and in the same way
the ARCH process is doing its activity and seeing a CF flag entry that reflects an earlier
state.

SOLUTION

check the alert.log for any further issues. As long as ORA-322, ORA-312 is not combined with other issues, you can ignore this error.

Related Issue:
ORA-00314: log 42 of thread 2, expected sequence# 37363 doesn’t match 37361

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)