Extending a Non-root LVM Partition on Exadata

On an Oracle Exadata Database Machine, the ‘/u01’ is defaulted to a size of 100Gb, which can easily fill up.  Luckily this is just a Logical Volume and there’s normally lots of space available on the Logical Volume Group which is usually untapped.

 

Extending ‘/u01’

Identify how much space is used and free on ‘/u01’ using df:

[root@v1ex1dbadm01 ~]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1
99G 78G 17G 83% /u01
[root@v1ex1dbadm01 ~]#

Display the current logical volume configuration using the lvs command:

[root@v1ex1dbadm01 ~]# lvs -o lv_name,lv_path,vg_name,lv_size
LV Path VG LSize
LVDbOra1 /dev/VGExaDb/LVDbOra1 VGExaDb 100.00g
LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb 24.00g
LVDbSys1 /dev/VGExaDb/LVDbSys1 VGExaDb 30.00g
LVDbSys2 /dev/VGExaDb/LVDbSys2 VGExaDb 30.00g
LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb 1.00g
[root@v1ex1dbadm01 ~]#

Check the online resize option is available using the tune2fs command:

[root@v1ex1dbadm01 ~]# tune2fs -l /dev/VGExaDb/LVDbOra1 | grep resize_inode
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
[root@v1ex1dbadm01 ~]#

If not available then the file system needs to be unmounted before resizing.  Refer to documentation:

Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

Verify there’s space available in the Logical Volume Group using vgdisplay command:

[root@v1ex1dbadm01 ~]# vgdisplay -s
"VGExaDb" 1.63 TiB [185.00 GiB used / 1.45 TiB free]
[root@v1ex1dbadm01 ~]#

Finally if there’s enough space, then extend the Logical Volume using lvextend command:

[root@v1ex1dbadm01 ~]# lvextend -L +100G /dev/VGExaDb/LVDbOra1
Size of logical volume VGExaDb/LVDbOra1 changed from 100.00 GiB (25600 extents) to 200.00 GiB (51200 extents).
Logical volume LVDbOra1 successfully resized.
[root@v1ex1dbadm01 ~]#

Followed by a resize of the file system using resize2fs command:

[root@v1ex1dbadm01 ~]# resize2fs /dev/VGExaDb/LVDbOra1
resize2fs 1.43-WIP (20-Jun-2013)
Filesystem at /dev/VGExaDb/LVDbOra1 is mounted on /u01; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 13
Performing an on-line resize of /dev/VGExaDb/LVDbOra1 to 52428800 (4k) blocks.
The filesystem on /dev/VGExaDb/LVDbOra1 is now 52428800 blocks long.
[root@v1ex1dbadm01 ~]#

You should now see ‘/u01’ with additional 100Gb less formatting:

[root@v1ex1dbadm01 ~]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1
197G 78G 110G 42% /u01
[root@v1ex1dbadm01 ~]#

Also see the Logical Volume is now 200Gb from 100Gb:

[root@v1ex1dbadm01 ~]# lvs -o lv_name,lv_path,vg_name,lv_size
LV Path VG LSize
LVDbOra1 /dev/VGExaDb/LVDbOra1 VGExaDb 200.00g
LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb 24.00g
LVDbSys1 /dev/VGExaDb/LVDbSys1 VGExaDb 30.00g
LVDbSys2 /dev/VGExaDb/LVDbSys2 VGExaDb 30.00g
LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb 1.00g
[root@v1ex1dbadm01 ~]#

 

Documentation for reference:

Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Assess Performance using Calibrate on Exadata

For those who are fortunate to have an Oracle Exadata Database Machine, may wonder if their Exadata meets the IOPS/MBPS as per the technical specifications.  Well with the command CALIBRATE in CellCLI, you can run raw performance tests on the cells’ hard disks and flash drives, enabling you to verify the disk/drive performance:

[root@v1ex1celadm01 ~]# cellcli
CellCLI: Release 12.1.2.3.4 - Production on Tue Jun 13 19:02:05 IST 2017

Copyright (c) 2007, 2016, Oracle. All rights reserved.

CellCLI> calibrate force;
Calibration will take a few minutes...
Aggregate random read throughput across all hard disk LUNs: 1823 MBPS
Aggregate random read throughput across all flash disk LUNs: 9973 MBPS
Aggregate random read IOs per second (IOPS) across all hard disk LUNs: 3002
Calibrating hard disks (read only) ...
LUN 0_0 on drive [8:0 ] random read throughput: 152.00 MBPS, and 243 IOPS
LUN 0_1 on drive [8:1 ] random read throughput: 157.00 MBPS, and 246 IOPS
LUN 0_10 on drive [8:10 ] random read throughput: 161.00 MBPS, and 253 IOPS
LUN 0_11 on drive [8:11 ] random read throughput: 157.00 MBPS, and 251 IOPS
LUN 0_2 on drive [8:2 ] random read throughput: 157.00 MBPS, and 244 IOPS
LUN 0_3 on drive [8:3 ] random read throughput: 158.00 MBPS, and 245 IOPS
LUN 0_4 on drive [8:4 ] random read throughput: 156.00 MBPS, and 248 IOPS
LUN 0_5 on drive [8:5 ] random read throughput: 161.00 MBPS, and 250 IOPS
LUN 0_6 on drive [8:6 ] random read throughput: 159.00 MBPS, and 252 IOPS
LUN 0_7 on drive [8:7 ] random read throughput: 158.00 MBPS, and 251 IOPS
LUN 0_8 on drive [8:8 ] random read throughput: 157.00 MBPS, and 251 IOPS
LUN 0_9 on drive [8:9 ] random read throughput: 159.00 MBPS, and 254 IOPS
Calibrating flash disks (read only, note that writes will be significantly slower) ...
LUN 1_1 on drive [FLASH_1_1] random read throughput: 2,157.00 MBPS, and 280525 IOPS
LUN 2_1 on drive [FLASH_2_1] random read throughput: 2,156.00 MBPS, and 274304 IOPS
LUN 4_1 on drive [FLASH_4_1] random read throughput: 2,158.00 MBPS, and 282083 IOPS
LUN 5_1 on drive [FLASH_5_1] random read throughput: 2,160.00 MBPS, and 287786 IOPS
CALIBRATE results are within an acceptable range.
Calibration has finished.

CellCLI>

 

The CALIBRATE FORCE, allows the test to run when CELLSRV is still up, this is acceptable if there is no user workload.  It is therefore recommended to not run during normal operations.  Without the FORCE, the CELLSRV must be shut down.

PLEASE NOTE: This is a single run on a single storage cell, you will need to run on all storage cells in the Exadata Machine to get the total IOPS/MBPS.  You can use dcli to run this across all the cells 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

SNMP unresponsive on Storage Cell after Exadata Patching

After a round of Exadata patching, the Storage Cells become unresponsive to SNMP for monitoring tools like OpsViews:

check_snmp_sysinfo CRITICAL - Agent not responding, tried SNMP v1 and v2c

This is because the Storage Cells get re-imaged as part of the Exadata patching, confirmed by imagehistory:

[root@v1ex1celadm01 ~]# imagehistory
Version : 12.1.2.1.1.150316.2
Image activation date : 2015-05-01 16:10:10 -0700
Imaging mode : fresh
Imaging status : success

Version : 12.1.2.1.3.151021
Image activation date : 2015-12-16 05:25:21 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.2.1.160330
Image activation date : 2016-05-12 14:02:43 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.2.160721
Image activation date : 2016-10-05 01:45:42 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.3.161208
Image activation date : 2017-01-18 02:53:10 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.4.170111
Image activation date : 2017-05-23 10:58:24 +0100
Imaging mode : out of partition upgrade
Imaging status : success

[root@v1oex1celadm01 ~]#

Which knocks out the configuration in iptables and snmpd.conf.

To resolve add back in the lines for your primary and secondary monitoring servers, for example:

rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32

So looks like this:

[root@v1ex1celadm01 ~]# more /etc/snmp/snmpd.conf
trapcommunity public
trapsink 127.0.0.1 public
rocommunity public 127.0.0.1
rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32
rwcommunity public 127.0.0.1

access RWGroup "" any noauth exact all all all
com2sec snmpclient 127.0.0.1 public
group RWGroup v1 snmpclient

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
syslocation Unknown (edit /etc/snmp/snmpd.conf)

view all included .1 80

[root@v1ex1celadm01 ~]#

Then reload snmp using the user root:

[root@v1ex2celadm01 ~]# service snmpd reload
Reloading snmpd: [ OK ]
[root@v1ex2celadm01 ~]#

Next add the monitoring servers primary and secondary to iptables as follows:

[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.31 -p udp --dport 161 -j ACCEPT
[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.32 -p udp --dport 161 -j ACCEPT

And make permanent by saving to firewall rules:

[root@v1ex1dbadm01 ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@v1ex1dbadm01 ~]#

Now your monitoring should return as SNMP connectivity is restored 🙂 :

SNMP OK - "Linux v1ex1celadm01.v1.com 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64"

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Using Linux Screen to protect against network breaks

When I carried out my first Exadata patching, I came across ‘Linux Screen‘ in the documentation, which Oracle recommended to use when patching:

“It’s recommended to run the dbnodeupdate.sh session using the Linux ‘screen’ or ‘vnc’ utility such that when a network problem breaks the connection to the server the patching session continues.”

Basically ‘Linux Screen‘ allows you to:

  1. Have a shell that can be reconnected to if you lose your connection due to network break, accidental close of terminal, the machine running the terminal crashes, etc. We’ve all been there, but now you can have the ability to protect yourself from these issues 🙂
  2. Disconnect from your screen session and reconnect to the same session at a later point in time, i.e. disconnect at the office, go home, reconnect when back home or kick off a long batch, disconnect, reconnect later to see if it’s completed.
  3. Open multiple windows within the one screen session.

Install Linux Screen

Linux screen is available from Oracle and can be downloaded via Oracle Public-yum:

[root@v1ex1dbadm01 ~]# yum install screen
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package screen.x86_64 0:4.0.3-19.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================
 Package Arch Version Repository Size
==============================================================================================================================================================================================================================================
Installing:
 screen x86_64 4.0.3-19.el6 ol6_latest 494 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install 1 Package(s)

Total download size: 494 k
Installed size: 795 k
Is this ok [y/N]: y
Downloading Packages:
screen-4.0.3-19.el6.x86_64.rpm | 494 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
 Installing : screen-4.0.3-19.el6.x86_64 1/1
 Verifying : screen-4.0.3-19.el6.x86_64 1/1

Installed:
 screen.x86_64 0:4.0.3-19.el6

Complete!
[root@v1ex1dbadm01 ~]#

To manually install:

[root@v1ex1dbadm01 patches]# rpm -i screen-4.0.3-19.el6.x86_64.rpm
[root@v1ex1dbadm01 patches]# rpm -qa | grep screen
screen-4.0.3-19.el6.x86_64
[root@v1ex1dbadm01 patches]# which screen
/usr/bin/screen
[root@v1ex1dbadm01 patches]#

RPMs are available via Oracle Public-yum:

http://public-yum.oracle.com/oracle-linux-5.html

http://public-yum.oracle.com/oracle-linux-6.html

Examples:

Oracle Linux 5, Update 11:

http://public-yum.oracle.com/repo/OracleLinux/OL5/11/base/x86_64/getPackage/screen-4.0.3-4.el5.x86_64.rpm

Oracle Linux 6, Update 8:

http://public-yum.oracle.com/repo/OracleLinux/OL6/8/base/x86_64/getPackage/screen-4.0.3-19.el6.x86_64.rpm

Starting Linux Screen

To start Linux Screen:

[root@v1ex1dbadm01 ~]# screen

Linux Screen Commands

Linux Screen commands are used by first pressing “CTRL” then “a“, followed by command key:

Example: “CTRL” then “a” then “?” get you the following help screen:

Screen key bindings, page 1 of 1.

Command key: ^A Literal ^A: a

break ^B b displays * help ? lockscreen ^X x number N quit \ screen ^C c title A writebuf >
clear C dumptermcap . history { } log H only Q readbuf < select ' vbell ^G xoff ^S s
colon : fit F info i login L other ^A redisplay ^L l silence _ version v xon ^Q q
copy ^[ [ flow ^F f kill K k meta a pow_break B remove X split S width W
detach ^D d focus ^I lastmsg ^M m monitor M pow_detach D removebuf = suspend ^Z z windows ^W w
digraph ^V hardcopy h license , next ^@ ^N sp n prev ^H ^P p ^? reset Z time ^T t wrap ^R r

^] paste .
" windowlist -b
- select -
0 select 0
1 select 1
2 select 2
3 select 3
4 select 4
5 select 5
6 select 6
7 select 7
8 select 8
9 select 9
I login on
O login off
] paste .


 [Press Space or Return to end.]

Detaching from Linux Screen

To detach from Linux Screen:

CTRL” then “a” then “d

[root@v1ex1dbadm01 ~]# echo "Detaching from Linux Screen, leaving my mark :)"
Detaching from Linux Screen, leaving my mark :)
[detached]

You are at this point returned to your normal shell.

Please Note: network break, accidental close of terminal, the machine running the terminal crashes, etc, automatically detach 🙂

Reattaching to Linux Screen

To reattach to Linux Screen:

[root@v1ex1dbadm01 ~]# screen -r

You’ll now be reattached to you last session:

[root@v1ex1dbadm01 ~]# echo "Detaching from Linux Screen, leaving my mark :)"
Detaching from Linux Screen, leaving my mark :)
[root@v1ex1dbadm01 ~]#

To reattach to Linux Screen, when multiple sessions are detached, use screen -r :

[root@v1ex1dbadm01 ~]# screen -r
There are several suitable screens on:
 15090.pts-0.v1ex1dbadm01 (Detached)
 346219.pts-0.v1ex1dbadm01 (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
[root@v1ex1dbadm01 ~]# screen -r 346219.pts-0.v1ex1dbadm01

Terminating Linux Screen

Once you’re finished, just type ‘exit‘ and you’ll exit back to your normal shell:

[screen is terminating]
[root@v1ex1dbadm01 ~]# screen -r
There is no screen to be resumed.
[root@v1ex1dbadm01 ~]#

You can also use:

CTRL” then “a” then “k“:

Really kill this window [y/n]y
[screen is terminating]

Advance Uses

Creating multiple screen in one session

You can add screen to your current session by:

CTRL” then “a” then “c

You’ll notice your putty session will say something like:

[screen 1:bash]root@v1ex1dbadm01:~

Screen ID starts with 0 and increases by 1 for every new screen created.

Switching between multiple screens in one session

You can switch between multiple screens, by going next and previous:

Next: “CTRL” then “a” then “n

Previous: “CTRL” then “a” then “p

You’ll notice the screen ID change as well as the screen.

Closing is the same as terminating Linux Screen, either type ‘exit‘ or “CTRL” then “a” then “k“.  You’ll need to do this for every screen until all are closed and then you will get:

[screen is terminating]

Known Issues

Cannot open your terminal

If you get the following error message:

[oracle@v1ex1dbadm01 ~]$ screen
Cannot open your terminal '/dev/pts/0' - please check.

Simple run Linux Screen as the user ‘root’ and then su to the desired user:

[root@v1ex1dbadm01 ~]# screen
[root@v1ex1dbadm01 ~]# su - oracle
[oracle@v1ex1dbadm01 ~]$

Scrollbar doesn’t work

You can enter copy mode (not sure why it’s call that, but it lets you navigate):

CTRL” then “a” then “esc” then “up“/”down

Or even better add the follow line to the root ~/.screenrc file:

termcapinfo xterm ti@:te@

If the file doesn’t exist, create it and add the line.

Now you can use scrollbar as normal 🙂

 

There are many more features, which I haven’t yet explored, but now you’ve got the idea, you’ll be able to use ‘Linux Screen’ to your benefit 🙂

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)