After a round of Exadata patching, the Storage Cells become unresponsive to SNMP for monitoring tools like OpsViews:

check_snmp_sysinfo CRITICAL - Agent not responding, tried SNMP v1 and v2c

This is because the Storage Cells get re-imaged as part of the Exadata patching, confirmed by imagehistory:

[root@v1ex1celadm01 ~]# imagehistory
Version : 12.1.2.1.1.150316.2
Image activation date : 2015-05-01 16:10:10 -0700
Imaging mode : fresh
Imaging status : success

Version : 12.1.2.1.3.151021
Image activation date : 2015-12-16 05:25:21 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.2.1.160330
Image activation date : 2016-05-12 14:02:43 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.2.160721
Image activation date : 2016-10-05 01:45:42 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.3.161208
Image activation date : 2017-01-18 02:53:10 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.4.170111
Image activation date : 2017-05-23 10:58:24 +0100
Imaging mode : out of partition upgrade
Imaging status : success

[root@v1oex1celadm01 ~]#

Which knocks out the configuration in iptables and snmpd.conf.

To resolve add back in the lines for your primary and secondary monitoring servers, for example:

rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32

So looks like this:

[root@v1ex1celadm01 ~]# more /etc/snmp/snmpd.conf
trapcommunity public
trapsink 127.0.0.1 public
rocommunity public 127.0.0.1
rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32
rwcommunity public 127.0.0.1

access RWGroup "" any noauth exact all all all
com2sec snmpclient 127.0.0.1 public
group RWGroup v1 snmpclient

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
syslocation Unknown (edit /etc/snmp/snmpd.conf)

view all included .1 80

[root@v1ex1celadm01 ~]#

Then reload snmp using the user root:

[root@v1ex2celadm01 ~]# service snmpd reload
Reloading snmpd: [ OK ]
[root@v1ex2celadm01 ~]#

Next add the monitoring servers primary and secondary to iptables as follows:

[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.31 -p udp --dport 161 -j ACCEPT
[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.32 -p udp --dport 161 -j ACCEPT

And make permanent by saving to firewall rules:

[root@v1ex1dbadm01 ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@v1ex1dbadm01 ~]#

Now your monitoring should return as SNMP connectivity is restored 🙂 :

SNMP OK - "Linux v1ex1celadm01.v1.com 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64"

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)