How to use Oracle Exadata Database Machine Exa Check (exachk)

Oracle customers who are fortunate to have an Oracle Exadata Database Machine, will need to run “exachk” from time to time, which is pronounced as Exa-Check.  This tool checks your Oracle Exadata Database Machine configuration, software, critical issue and provides a Maximum Availability Architecture score.  It is required to run before and after Exadata patching and is good practice to run on a frequent basis and review the recommendations.

More info here:

Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)

“exachk holistically evaluates all Oracle Engineered Systems.   

It includes:

  • Configuration checks for Database Servers, Storage Servers and InfiniBand Switches
  • Grid Infrastructure, Database and ASM and operating system software checks
  • MAA Scorecard which conducts an automatic MAA Review
  • Exadata Software Planner, Software prechecks, Exadata and Database Critical Issue alerts

All checks have explanations, recommendations, and manual verification commands so that customers can self-correct all FAIL and WARNING conditions reported. 

Development recommends that the latest exachk be executed with the following frequency:

  • Monthly
  • Week before any planned maintenance activity
  • Day before any planned maintenance activity
  • Immediately after completion of planned maintenance activity or an outage or incident”

The download can be obtained from the above MOS note 1070954.1.  Place the download zip file as root on a compute node, in the Oracle’s Home for example, I use “/home/oracle/version1/exachk”.  Within this folder create another folder called exachk, which allows you to easily replace the tool with the latest version and unzip the zip file to this folder:

[root@v1ex2dbadm01 exachk]# pwd
/home/oracle/version1/exachk
[root@v1ex2dbadm01 exachk]# rm -rf exachk
[root@v1ex2dbadm01 exachk]# mkdir exachk
[root@v1ex2dbadm01 exachk]# ls -lh
total 19M
drwxr-xr-x 2 root root 4.0K Sep 5 10:53 exachk
-rw-r--r-- 1 root root 19M Sep 5 10:51 exachk.zip
[root@v1ex2dbadm01 exachk]# unzip exachk.zip -d ./exachk
Archive: exachk.zip
creating: ./exachk/.cgrep/
inflating: ./exachk/.cgrep/versions.dat
...
inflating: ./exachk/UserGuide.txt
inflating: ./exachk/readme.txt
inflating: ./exachk/doc/ORAchk_and_EXAchk_User_Guide.pdf
[root@v1ex2dbadm01 exachk]#

You’ll now see the tool “exachk” as an executable along with the other files as part of the tool, including the User Guide:

[root@v1ex2dbadm01 exachk]# ls
exachk exachk.zip
[root@v1ex2dbadm01 exachk]# ls -l exachk
total 69792
-r-xr-xr-x 1 root root 8218911 Apr 4 12:12 Apex5_CollectionManager_App.sql
-r-xr-xr-x 1 root root 4816355 Sep 15 2016 CollectionManager_App.sql
-rw-r--r-- 1 root root 47172483 Jul 19 08:38 collections.dat
drwxr-xr-x 2 root root 4096 Sep 5 10:54 doc
-r-xr-xr-x 1 root root 2897015 May 17 20:59 exachk
-rw-r--r-- 1 root root 1976299 Jul 19 09:03 EXAchk_Health_Check_Catalog.html
drwxr-xr-x 2 root root 4096 Jul 19 03:17 exadiscover
-r--r--r-- 1 root root 4896 Mar 31 09:58 readme.txt
-rw-r--r-- 1 root root 6302687 Jul 19 08:38 rules.dat
-r-xr-xr-x 1 root root 40052 Jul 22 2015 sample_user_defined_checks.xml
drwxr-xr-x 2 root root 4096 Jul 19 03:17 templates
-r-xr-xr-x 1 root root 2888 Oct 9 2015 user_defined_checks.xsd
-r--r--r-- 1 root root 234 Apr 1 10:05 UserGuide.txt
[root@v1ex2dbadm01 exachk]# cd exachk
[root@v1ex2dbadm01 exachk]#

To run the tool, simply execute without any switches (there are other options, refer to User Guide):

[root@v1ex2dbadm01 exachk]# ./exachk

Checking for prompts on v1ex2dbadm01 for oracle user...

Checking ssh user equivalency settings on all nodes in cluster

Node v1ex2dbadm02 is configured for ssh user equivalency for root user

Checking for prompts for oracle user on all nodes...

Searching for running databases . . . . .
. . . . . . . . . . . .
List of running databases registered in OCR
1. VERSS
2. VERSDEV
3. VERS24H
4. IONS
5. IONDEV
6. VERSREP
7. All of above
8. None of above

Select databases from list for checking best practices. For multiple databases, select 7 for All or comma separated number like 1,2 etc [1-8][7].7

Searching out ORACLE_HOME for selected databases.

. . . . . . . . . . . . . . . . . . . . . . . .

.
Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . .
-------------------------------------------------------------------------------------------------------
 Oracle Stack Status
-------------------------------------------------------------------------------------------------------
Host Name    CRS Installed  RDBMS Installed  CRS UP  ASM UP  RDBMS UP  DB Instance Name
-------------------------------------------------------------------------------------------------------
v1ex2dbadm01 Yes            Yes              Yes     Yes     Yes       IONDEV1 IONS1 VERS24H1 VERSDEV1 VERSREP1 VERSS1
v1ex2dbadm02 Yes            Yes              Yes     Yes     Yes       IONDEV2 IONS2 VERS24H2 VERSDEV2 VERSREP2 VERSS2
-------------------------------------------------------------------------------------------------------

v1ex2cel01 is configured for ssh user equivalency for root user
.
v1ex2cel02 is configured for ssh user equivalency for root user
.
v1ex2cel03 is configured for ssh user equivalency for root user
. . . . . . . . . . . . . . . . . . .
v1ex2sw-ibb01 is configured for ssh user equivalency for root user

v1ex2sw-iba01 is configured for ssh user equivalency for root user

v1ex2sw-ibb01 is configured for ssh user equivalency for root user

*** Checking Best Practice Recommendations (PASS/WARNING/FAIL) ***

Collections and audit checks log file is
/home/oracle/version1/exachk/exachk/exachk_v1ex2dbadm01_VERSREP_090517_105859/log/exachk.log

Checking for prompts in /root/.bash_profile on v1ex2dbadm01 for root user...

Checking for prompts in /root/.bash_profile on v1ex2dbadm02 for root user...

Starting to run exachk in background on v1ex2dbadm02

. .
=============================================================
 Node name - v1ex2dbadm01
=============================================================
. . . . .

Collecting - ASM Disk Group for Infrastructure Software and Configuration
Collecting - ASM Diskgroup Attributes
Collecting - ASM diskgroup usable free space
Collecting - ASM initialization parameters
Collecting - Database Parameters for IONDEV database
Collecting - Database Parameters for IONS database
Collecting - Database Parameters for VERSDEV database
Collecting - Database Parameters for VERSREP database
Collecting - Database Parameters for VERSS database
Collecting - Database Undocumented Parameters for IONDEV database
Collecting - Database Undocumented Parameters for VERSDEV database
Collecting - Database Undocumented Parameters for VERSREP database
Collecting - RDBMS Feature Usage for IONDEV database
Collecting - RDBMS Feature Usage for VERSDEV database
Collecting - RDBMS Feature Usage for VERSREP database
Collecting - CPU Information
Collecting - Clusterware and RDBMS software version
Collecting - Compute node PCI bus slot speed for infiniband HCAs
Collecting - Kernel parameters
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - OS Packages
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - RDBMS patch inventory
Collecting - Switch Version Information
Collecting - number of semaphore operations per semop system call
Collecting - CRS user limits configuration
Collecting - CRS user time zone check
Collecting - Check alerthistory for non-test open stateless alerts [Database Server]
Collecting - Check alerthistory for stateful alerts not cleared [Database Server]
Collecting - Check alerthistory for test open stateless alerts [Database Server]
Collecting - Clusterware patch inventory
Collecting - Detect duplicate files in /etc/*init* directories
Collecting - Discover switch type(spine or leaf)
Collecting - Enterprise Manager agent targets
Collecting - Exadata Critical Issue DB09
Collecting - Exadata Critical Issue EX30
Collecting - Exadata software version on database server
Collecting - Exadata system model number
Collecting - Exadata version on database server
Collecting - HCA firmware version on database server
Collecting - HCA transfer rate on database server
Collecting - Infrastructure Software and Configuration for compute
Collecting - MaxStartups setting in sshd_config
Collecting - OFED Software version on database server
Collecting - Obtain hardware information
Collecting - Operating system and Kernel version on database server
Collecting - Oracle monitoring agent and/or OS settings on ADR diagnostic directories
Collecting - Raid controller bus link speed
Collecting - System Event Log
Collecting - Validate key sysctl.conf parameters on database servers
Collecting - Verify Ambient Air Temperature [Database Server]
Collecting - Verify Data Network is Separate from Management Network
Collecting - Verify Database Server Disk Controller Configuration
Collecting - Verify Database Server Physical Drive Configuration
Collecting - Verify Database Server Virtual Drive Configuration
Collecting - Verify Disk Cache Policy on database server
Collecting - Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Database Server]
Collecting - Verify ILOM Power Up Configuration for HOST_AUTO_POWER_ON
Collecting - Verify ILOM Power Up Configuration for HOST_LAST_POWER_STATE
Collecting - Verify IP routing configuration on database servers
Collecting - Verify InfiniBand Address Resolution Protocol (ARP) Configuration on Database Servers
Collecting - Verify InfiniBand Fabric Topology (verify-topology)
Collecting - Verify InfiniBand subnet manager is not running on database server
Collecting - Verify InfiniBand subnet manager is running on an InfiniBand switch
Collecting - Verify Master (Rack) Serial Number is Set [Database Server]
Collecting - Verify NTP configuration on database servers
Collecting - Verify Quorum disks configuration
Collecting - Verify RAID Controller Battery Temperature [Database Server]
Collecting - Verify RAID disk controller CacheVault capacitor condition [Database Server]
Collecting - Verify TCP Segmentation Offload (TSO) is set to off
Collecting - Verify active kernel version matches expected version for installed Exadata Image
Collecting - Verify basic Logical Volume(LVM) system devices configuration
Collecting - Verify database server disk controllers use writeback cache
Collecting - Verify database server file systems have Check interval = 0
Collecting - Verify database server file systems have Maximum mount count = -1
Collecting - Verify imageinfo on database server
Collecting - Verify imageinfo on database server to compare systemwide
Collecting - Verify key InfiniBand fabric error counters are not present
Collecting - Verify no database server kernel out of memory errors
Collecting - Verify service exachkcfg autostart status on database server
Collecting - Verify that the SDP over IB option sdp_apm_enable is set to 0
Collecting - Verify the localhost alias is pingable [Database Server]
Collecting - Verify the Name Service Cache Daemon (NSCD) is Running
Collecting - Verify the file /.updfrm_exact does not exist [Database Server]
Collecting - Verify the grid Infrastructure management database (MGMTDB) does not use hugepages
Collecting - Verify the vm.min_free_kbytes configuration
Collecting - root time zone check
Collecting - verify asr exadata configuration check via ASREXACHECK on database server

Starting to run root privileged commands in background on STORAGE SERVER v1ex2cel01 (10.1.11.15)

Starting to run root privileged commands in background on STORAGE SERVER v1ex2cel02 (10.1.11.17)

Starting to run root privileged commands in background on STORAGE SERVER v1ex2cel03 (10.1.11.19)

Starting to run root privileged commands in background on INFINIBAND SWITCH v1ex2sw-ibb01

Starting to run root privileged commands in background on INFINIBAND SWITCH v1ex2sw-iba01

Collections from STORAGE SERVER:
----------------------------------
Collecting - Exadata Critical Issue EX10
Collecting - Exadata Critical Issue EX11
Collecting - Exadata Critical Issue EX22
Collecting - Exadata Critical Issue EX28
Collecting - Exadata Critical Issue EX31
Collecting - Exadata critical issue EX14
Collecting - Exadata critical issue EX16
Collecting - Exadata critical issue EX17
Collecting - Exadata critical issue EX23
Collecting - Exadata critical issue EX24
Collecting - Exadata software version on storage server
Collecting - Exadata software version on storage servers
Collecting - Exadata storage server system model number
Collecting - Infrastructure Software and Configuration for storage
Collecting - RAID controller version on storage servers
Collecting - Verify Ambient Air Temperature [Storage Server]
Collecting - Verify Disk Cache Policy on storage servers
Collecting - Verify Exadata Smart Flash Cache is created
Collecting - Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Storage Server]
Collecting - Verify ILOM Power Up Configuration for HOST_AUTO_POWER_ON on storage servers
Collecting - Verify ILOM Power Up Configuration for HOST_LAST_POWER_STATE on storage servers
Collecting - Verify InfiniBand subnet manager is not running on storage server
Collecting - Verify Master (Rack) Serial Number is Set [Storage Server]
Collecting - Verify NTP configuration on storage servers
Collecting - Verify OSSCONF/cellinit.ora consistency across storage servers
Collecting - Verify RAID Controller Battery Temperature [Storage Server]
Collecting - Verify RAID disk controller CacheVault capacitor condition [Storage Server]
Collecting - Verify Storage Server user CELLDIAG exists
Collecting - Verify active system values match those defined in configuration file cell.conf [Storage Server]
Collecting - Verify data (non-system) disks on Exadata Storage Servers have no partitions
Collecting - Verify imageinfo on storage server
Collecting - Verify imageinfo on storage server to compare systemwide
Collecting - Verify release tracking bug on storage servers
Collecting - Verify service exachkcfg autostart status on storage server
Collecting - Verify storage server disk controllers use writeback cache
Collecting - Verify the localhost alias is pingable [Storage Server]
Collecting - Verify the file /.updfrm_exact does not exist [Storage Server]
Collecting - verify asr exadata configuration check via ASREXACHECK on storage servers
Collecting - Check alerthistory for non-test open stateless alerts [Storage Server]
Collecting - Check alerthistory for stateful alerts not cleared [Storage Server]
Collecting - Check alerthistory for test open stateless alerts [Storage Server]
Collecting - Configure Storage Server alerts to be sent via email
Collecting - Determine storage server type(All Flash/High Capacity)
Collecting - Exadata Celldisk predictive failures
Collecting - Exadata storage server root filesystem free space
Collecting - HCA firmware version on storage server
Collecting - OFED Software version on storage server
Collecting - Operating system and Kernel version on storage server
Collecting - Storage server flash cache mode
Collecting - Storage server make and model
Collecting - Verify Data Network is Separate from Management Network on storage server
Collecting - Verify Datafiles are Placed on Diskgroups consisting of griddisks with correct attributes
Collecting - Verify Ethernet Cable Connection Quality on storage servers
Collecting - Verify ExaWatcher is executing [Storage Server]
Collecting - Verify Exadata Smart Flash Cache is actually in use
Collecting - Verify Exadata Smart Flash Cache status is normal
Collecting - Verify Exadata Smart Flash Log is Created
Collecting - Verify InfiniBand Cable Connection Quality on storage servers
Collecting - Verify average ping times to DNS nameserver [Storage Server]
Collecting - Verify celldisk configuration on disk drives
Collecting - Verify celldisk configuration on flash memory devices
Collecting - Verify griddisk ASM status
Collecting - Verify griddisk count matches across all storage servers where a given prefix name exists
Collecting - Verify storage server metric CD_IO_ST_RQ
Collecting - Verify the percent of available celldisk space used by the griddisks
Collecting - Verify there are no griddisks configured on flash memory devices
Collecting - Verify total number of griddisks with a given prefix name is evenly divisible of celldisks
Collecting - mpt_cmd_retry_count from /etc/modprobe.conf on Storage Servers

Collections from INFINIBAND SWITCH:
------------------------------------
Collecting - Exadata Critical Issue IB5
Collecting - Hostname in /etc/hosts
Collecting - Infiniband Switch NTP configuration
Collecting - Infiniband subnet manager status
Collecting - Infiniband switch HCA status
Collecting - Infiniband switch HOSTNAME configuration
Collecting - Infiniband switch firmware version
Collecting - Infiniband switch health
Collecting - Infiniband switch localtime configuration
Collecting - Infiniband switch module configuration
Collecting - Infiniband switch subnet manager configuration
Collecting - Infiniband switch type(Spine or leaf)
Collecting - Infrastructure Software and Configuration for switch
Collecting - Verify average ping times to DNS nameserver [IB Switch]
Collecting - Verify no IB switch ports disabled due to excessive symbol errors
Collecting - Verify the localhost alias is pingable [IB Switch]
Collecting - sm_priority configuration on Infiniband switch

Data collections completed. Checking best practices on v1ex2dbadm01.
--------------------------------------------------------------------------------------

FAIL => One or more storage servers have stateful alerts that have not been cleared.
FAIL => DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for VERSDEV
FAIL => DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for IONDEV
FAIL => DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for VERSREP
WARNING => Hidden ASM Initialization Parameter usage is not correct
WARNING => ASM parameter AUDIT_SYSLOG_LEVEL should be set to the recommended value
WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on VERSS1 instance
WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on IONS1 instance
FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSDEV1 instance
FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on IONDEV1 instance
FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSREP1 instance
WARNING => ASM parameter ASM_POWER_LIMIT is not set to the default value.
FAIL => One or more database servers have stateful alerts that have not been cleared
INFO => Oracle GoldenGate failure prevention best practices
INFO => One or more non-default AWR baselines should be created for VERSDEV
INFO => One or more non-default AWR baselines should be created for IONDEV
INFO => One or more non-default AWR baselines should be created for VERSREP
WARNING => Local archive destination does not alternate destination configured for VERSDEV
WARNING => Local archive destination does not alternate destination configured for IONDEV
WARNING => Local archive destination does not alternate destination configured for VERSREP
WARNING => System has fewer than five storage servers but does not have quorum disks configured on database servers.
INFO => Validate database security configuration using database security assessment tool for VERSS
INFO => Validate database security configuration using database security assessment tool for VERSDEV
INFO => Validate database security configuration using database security assessment tool for IONS
INFO => Validate database security configuration using database security assessment tool for IONDEV
INFO => Validate database security configuration using database security assessment tool for VERSREP
FAIL => System is exposed to Exadata Critical Issue IB5 on infiniband switch v1ex2sw-ibb01
FAIL => System is exposed to Exadata Critical Issue IB5 on infiniband switch v1ex2sw-iba01
FAIL => Oracle monitoring agent and Operating systems settings on Automatic diagnostic repository directories are not correct or not all targets have been scanned or not all diagnostic directories found
FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on IONDEV1 instance
FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSREP1 instance
FAIL => Storage Server user "CELLDIAG" should exist
FAIL => Downdelay attribute is not set to recommended value on bonded client interface
FAIL => Database control files are not configured as recommended for VERSS
FAIL => Database control files are not configured as recommended for VERSDEV
FAIL => Database control files are not configured as recommended for IONS
FAIL => Database control files are not configured as recommended for IONDEV
FAIL => Database control files are not configured as recommended for VERSREP
FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSDEV
FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for IONDEV
FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSREP
WARNING => SYS or SYSTEM objects were found to be INVALID for VERSDEV
WARNING => SYS or SYSTEM objects were found to be INVALID for VERSREP
FAIL => All disk groups should have compatible.rdbms attribute set to recommended values
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSS1 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONS1 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONDEV1 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSREP1 instance
FAIL => Storage Server alerts are not configured to be sent via email
WARNING => filesystemio_options is not set to recommended value on VERSS1 instance
WARNING => filesystemio_options is not set to recommended value on VERSDEV1 instance
WARNING => filesystemio_options is not set to recommended value on IONS1 instance
WARNING => filesystemio_options is not set to recommended value on IONDEV1 instance
WARNING => filesystemio_options is not set to recommended value on VERSREP1 instance
FAIL => Some data or temp files are not autoextensible for IONDEV
WARNING => Key InfiniBand fabric error counters should not be present
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSS1 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONS1 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONDEV1 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSREP1 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSS1 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONS1 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONDEV1 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSREP1 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSS1 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONS1 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONDEV1 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSREP1 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSS1 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on IONS1 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSREP1 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSS1 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on IONS1 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSREP1 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSS1 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONS1 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONDEV1 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSREP1 instance
FAIL => Database parameter sql92_security is not set to recommended value on VERSS1 instance
FAIL => Database parameter sql92_security is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter sql92_security is not set to recommended value on IONS1 instance
FAIL => Database parameter sql92_security is not set to recommended value on IONDEV1 instance
FAIL => Database parameter sql92_security is not set to recommended value on VERSREP1 instance
FAIL => ASM parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value
FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSDEV
FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSREP
FAIL => ASM processes parameter is not set to recommended value
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSDEV
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for IONDEV
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSREP
FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for VERSS
FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for IONS
FAIL => Flashback on PRIMARY is not configured for IONDEV
INFO => Operational Best Practices
INFO => Database Consolidation Best Practices
INFO => Computer failure prevention best practices
INFO => Data corruption prevention best practices
INFO => Logical corruption prevention best practices
INFO => Database/Cluster/Site failure prevention best practices
INFO => Client failover operational best practices
WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSS1 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSDEV1 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONS1 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONDEV1 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSREP1 instance
FAIL => ASM parameter MEMORY_TARGET should be set according to recommended value
FAIL => ASM parameter PGA_AGGREGATE_TARGET should be set according to recommended value
FAIL => ASM parameter MEMORY_MAX_TARGET should be set according to recommended value
INFO => Verify the percent of available celldisk space used by the griddisks
FAIL => Database control files are not configured as recommended for VERSS
FAIL => Database control files are not configured as recommended for VERSDEV
FAIL => Database control files are not configured as recommended for IONS
FAIL => Database control files are not configured as recommended for IONDEV
FAIL => Database control files are not configured as recommended for VERSREP
INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for VERSS
INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for IONS
FAIL => Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for VERSDEV
FAIL => Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for VERSREP
INFO => Database failure prevention best practices
FAIL => Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for VERSDEV
FAIL => Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for IONDEV
FAIL => Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for VERSREP
FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSDEV1 instance
FAIL => Database parameter LOG_BUFFER is not set to recommended value on IONDEV1 instance
FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSREP1 instance
WARNING => Redo log files should be appropriately sized for IONDEV
WARNING => Redo log files should be appropriately sized for VERSREP
FAIL => No one high redundancy diskgroup configured for VERSDEV
FAIL => No one high redundancy diskgroup configured for IONDEV
FAIL => No one high redundancy diskgroup configured for VERSREP
INFO => Storage failures prevention best practices
INFO => Software maintenance best practices
FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSS1 instance
FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSDEV1 instance
FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSREP1 instance
FAIL => The data files should be recoverable for VERSS
FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for VERSS
FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for IONS
FAIL => FRA space management problem file types are present without an RMAN backup completion within the last 7 days. for VERSDEV
FAIL => FRA space management problem file types are present without an RMAN backup completion within the last 7 days. for IONDEV
INFO => Oracle recovery manager(rman) best practices
WARNING => RMAN controlfile autobackup should be set to ON for VERSDEV
WARNING => RMAN controlfile autobackup should be set to ON for IONDEV
INFO => Exadata Critical Issues (Doc ID 1270094.1):- DB1-DB4,DB6,DB9-DB37, EX1-EX26,EX29-EX36 and IB1-IB3,IB5
FAIL => ASM parameter SGA_TARGET is not set according to recommended value.
WARNING => Multiple Oracle database instances discovered, observe database consolidation best practices
FAIL => There should be enough freespace in all diskgroups to reestablish redundancy after a single disk failure
Collecting patch inventory on CRS HOME /u01/app/12.1.0.2/grid
Collecting patch inventory on ASM HOME /u01/app/12.1.0.2/grid
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_1
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_2

Copying results from v1ex2dbadm02 and generating report. This might take a while. Be patient.

. .
=============================================================
 Node name - v1ex2dbadm02
=============================================================
. . . . .

Collecting - CPU Information
Collecting - Clusterware and RDBMS software version
Collecting - Compute node PCI bus slot speed for infiniband HCAs
Collecting - Kernel parameters
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - OS Packages
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - RDBMS patch inventory
Collecting - number of semaphore operations per semop system call
Collecting - CRS user limits configuration
Collecting - CRS user time zone check
Collecting - Check alerthistory for non-test open stateless alerts [Database Server]
Collecting - Check alerthistory for stateful alerts not cleared [Database Server]
Collecting - Check alerthistory for test open stateless alerts [Database Server]
Collecting - Clusterware patch inventory
Collecting - Detect duplicate files in /etc/*init* directories
Collecting - Enterprise Manager agent targets
Collecting - Exadata Critical Issue DB09
Collecting - Exadata Critical Issue EX30
Collecting - Exadata software version on database server
Collecting - Exadata system model number
Collecting - Exadata version on database server
Collecting - HCA firmware version on database server
Collecting - HCA transfer rate on database server
Collecting - Infrastructure Software and Configuration for compute
Collecting - MaxStartups setting in sshd_config
Collecting - OFED Software version on database server
Collecting - Obtain hardware information
Collecting - Operating system and Kernel version on database server
Collecting - Oracle monitoring agent and/or OS settings on ADR diagnostic directories
Collecting - Raid controller bus link speed
Collecting - System Event Log
Collecting - Validate key sysctl.conf parameters on database servers
Collecting - Verify Ambient Air Temperature [Database Server]
Collecting - Verify Data Network is Separate from Management Network
Collecting - Verify Database Server Disk Controller Configuration
Collecting - Verify Database Server Physical Drive Configuration
Collecting - Verify Database Server Virtual Drive Configuration
Collecting - Verify Disk Cache Policy on database server
Collecting - Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Database Server]
Collecting - Verify ILOM Power Up Configuration for HOST_AUTO_POWER_ON
Collecting - Verify ILOM Power Up Configuration for HOST_LAST_POWER_STATE
Collecting - Verify IP routing configuration on database servers
Collecting - Verify InfiniBand Address Resolution Protocol (ARP) Configuration on Database Servers
Collecting - Verify InfiniBand subnet manager is not running on database server
Collecting - Verify Master (Rack) Serial Number is Set [Database Server]
Collecting - Verify NTP configuration on database servers
Collecting - Verify Quorum disks configuration
Collecting - Verify RAID Controller Battery Temperature [Database Server]
Collecting - Verify RAID disk controller CacheVault capacitor condition [Database Server]
Collecting - Verify TCP Segmentation Offload (TSO) is set to off
Collecting - Verify active kernel version matches expected version for installed Exadata Image
Collecting - Verify basic Logical Volume(LVM) system devices configuration
Collecting - Verify database server disk controllers use writeback cache
Collecting - Verify database server file systems have Check interval = 0
Collecting - Verify database server file systems have Maximum mount count = -1
Collecting - Verify imageinfo on database server
Collecting - Verify imageinfo on database server to compare systemwide
Collecting - Verify no database server kernel out of memory errors
Collecting - Verify service exachkcfg autostart status on database server
Collecting - Verify that the SDP over IB option sdp_apm_enable is set to 0
Collecting - Verify the localhost alias is pingable [Database Server]
Collecting - Verify the Name Service Cache Daemon (NSCD) is Running
Collecting - Verify the file /.updfrm_exact does not exist [Database Server]
Collecting - Verify the grid Infrastructure management database (MGMTDB) does not use hugepages
Collecting - Verify the vm.min_free_kbytes configuration
Collecting - root time zone check
Collecting - verify asr exadata configuration check via ASREXACHECK on database server

Data collections completed. Checking best practices on v1ex2dbadm02.
--------------------------------------------------------------------------------------

WARNING => Hidden ASM Initialization Parameter usage is not correct
WARNING => ASM parameter AUDIT_SYSLOG_LEVEL should be set to the recommended value
WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on VERSS2 instance
WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on IONS2 instance
FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSDEV2 instance
FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on IONDEV2 instance
FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSREP2 instance
WARNING => ASM parameter ASM_POWER_LIMIT is not set to the default value.
INFO => Oracle GoldenGate failure prevention best practices
WARNING => Local archive destination does not alternate destination configured for VERSDEV
WARNING => Local archive destination does not alternate destination configured for IONDEV
WARNING => Local archive destination does not alternate destination configured for VERSREP
WARNING => System has fewer than five storage servers but does not have quorum disks configured on database servers.
INFO => Validate database security configuration using database security assessment tool for VERSS
INFO => Validate database security configuration using database security assessment tool for VERSDEV
INFO => Validate database security configuration using database security assessment tool for IONS
INFO => Validate database security configuration using database security assessment tool for IONDEV
INFO => Validate database security configuration using database security assessment tool for VERSREP
FAIL => Oracle monitoring agent and Operating systems settings on Automatic diagnostic repository directories are not correct or not all targets have been scanned or not all diagnostic directories found
FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on IONDEV2 instance
FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSREP2 instance
FAIL => Downdelay attribute is not set to recommended value on bonded client interface
FAIL => Database control files are not configured as recommended for VERSS
FAIL => Database control files are not configured as recommended for VERSDEV
FAIL => Database control files are not configured as recommended for IONS
FAIL => Database control files are not configured as recommended for IONDEV
FAIL => Database control files are not configured as recommended for VERSREP
FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSDEV
FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for IONDEV
FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSREP
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSS2 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONS2 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONDEV2 instance
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSREP2 instance
WARNING => filesystemio_options is not set to recommended value on VERSS2 instance
WARNING => filesystemio_options is not set to recommended value on VERSDEV2 instance
WARNING => filesystemio_options is not set to recommended value on IONS2 instance
WARNING => filesystemio_options is not set to recommended value on IONDEV2 instance
WARNING => filesystemio_options is not set to recommended value on VERSREP2 instance
FAIL => ASM Audit file destination file count > 100,000
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSS2 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONS2 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONDEV2 instance
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSREP2 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSS2 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONS2 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONDEV2 instance
FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSREP2 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSS2 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONS2 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONDEV2 instance
FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSREP2 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSS2 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on IONS2 instance
FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSREP2 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSS2 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on IONS2 instance
FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSREP2 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSS2 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONS2 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONDEV2 instance
FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSREP2 instance
FAIL => Database parameter sql92_security is not set to recommended value on VERSS2 instance
FAIL => Database parameter sql92_security is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter sql92_security is not set to recommended value on IONS2 instance
FAIL => Database parameter sql92_security is not set to recommended value on IONDEV2 instance
FAIL => Database parameter sql92_security is not set to recommended value on VERSREP2 instance
FAIL => ASM parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value
FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSDEV
FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSREP
FAIL => ASM processes parameter is not set to recommended value
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSDEV
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for IONDEV
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSREP
FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for VERSS
FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for IONS
WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSS2 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSDEV2 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONS2 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONDEV2 instance
WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSREP2 instance
FAIL => ASM parameter MEMORY_TARGET should be set according to recommended value
FAIL => ASM parameter PGA_AGGREGATE_TARGET should be set according to recommended value
FAIL => ASM parameter MEMORY_MAX_TARGET should be set according to recommended value
FAIL => Database control files are not configured as recommended for VERSS
FAIL => Database control files are not configured as recommended for VERSDEV
FAIL => Database control files are not configured as recommended for IONS
FAIL => Database control files are not configured as recommended for IONDEV
FAIL => Database control files are not configured as recommended for VERSREP
INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for VERSS
INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for IONS
FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSDEV2 instance
FAIL => Database parameter LOG_BUFFER is not set to recommended value on IONDEV2 instance
FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSREP2 instance
WARNING => Redo log files should be appropriately sized for IONDEV
WARNING => Redo log files should be appropriately sized for VERSREP
FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSS2 instance
FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSDEV2 instance
FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSREP2 instance
FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for VERSS
FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for IONS
FAIL => ASM parameter SGA_TARGET is not set according to recommended value.
WARNING => Multiple Oracle database instances discovered, observe database consolidation best practices
Collecting patch inventory on CRS HOME /u01/app/12.1.0.2/grid
Collecting patch inventory on ASM HOME /u01/app/12.1.0.2/grid
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_1
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_2

---------------------------------------------------------------------------------
 CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------

Detailed report (html) - /home/oracle/version1/exachk/exachk/exachk_v1ex2dbadm01_VERSREP_090517_105859/exachk_v1ex2dbadm01_VERSREP_090517_105859.html
UPLOAD(if required) - /home/oracle/version1/exachk/exachk/exachk_v1ex2dbadm01_VERSREP_090517_105859.zip

[root@v1ex2dbadm01 exachk]#

Once completed, you’ll see a new folder and zip file, I recommend moving them both up a level to safe keeping:

[root@v1ex2dbadm01 exachk]# ls -ltrh
total 75M
-r-xr-xr-x 1 root root 40K Jul 22 2015 sample_user_defined_checks.xml
-r-xr-xr-x 1 root root 2.9K Oct 9 2015 user_defined_checks.xsd
-r-xr-xr-x 1 root root 4.6M Sep 15 2016 CollectionManager_App.sql
-r--r--r-- 1 root root 4.8K Mar 31 09:58 readme.txt
-r--r--r-- 1 root root 234 Apr 1 10:05 UserGuide.txt
-r-xr-xr-x 1 root root 7.9M Apr 4 12:12 Apex5_CollectionManager_App.sql
-r-xr-xr-x 1 root root 2.8M May 17 20:59 exachk
drwxr-xr-x 2 root root 4.0K Jul 19 03:17 templates
drwxr-xr-x 2 root root 4.0K Jul 19 03:17 exadiscover
-rw-r--r-- 1 root root 45M Jul 19 08:38 collections.dat
-rw-r--r-- 1 root root 6.1M Jul 19 08:38 rules.dat
-rw-r--r-- 1 root root 1.9M Jul 19 09:03 EXAchk_Health_Check_Catalog.html
drwxr-xr-x 2 root root 4.0K Sep 5 10:54 doc
drwxr-xr-x 8 root root 180K Sep 5 11:18 exachk_v1ex2dbadm01_VERSREP_090517_105859
-rw-r--r-- 1 root root 5.9M Sep 5 11:19 exachk_v1ex2dbadm01_VERSREP_090517_105859.zip
[root@v1ex2dbadm01 exachk]# mv exachk_* ..
[root@v1ex2dbadm01 exachk]# cd ..
[root@v1ex2dbadm01 exachk]# ls -ltrh
total 25M
-rw-r--r-- 1 root root 19M Sep 5 10:51 exachk.zip
drwxr-xr-x 8 root root 180K Sep 5 11:18 exachk_v1ex2dbadm01_VERSREP_090517_105859
-rw-r--r-- 1 root root 5.9M Sep 5 11:19 exachk_v1ex2dbadm01_VERSREP_090517_105859.zip
drwxr-xr-x 6 root root 4.0K Sep 5 11:29 exachk
[root@v1ex2dbadm01 exachk]#

In the new folder you’ll see a html file, which is the report produced by exachk:

[root@v1ex2dbadm01 exachk]# cd exachk_v1ex2dbadm01_VERSREP_090517_105859
[root@v1ex2dbadm01 exachk_v1ex2dbadm01_VERSREP_090517_105859]# ls -ltrh
total 6.4M
-rw-r--r-- 1 root root 762 Sep 5 11:02 targets.xsl
drwxr-xr-x 2 root root 4.0K Sep 5 11:18 upload
drwxr-xr-x 4 root root 100K Sep 5 11:18 outfiles
drwxr-xr-x 2 root root 4.0K Sep 5 11:18 reports
-rw-r--r-- 1 root root 6.2M Sep 5 11:18 exachk_v1ex2dbadm01_VERSREP_090517_105859.html
drwxr-xr-x 2 root root 48K Sep 5 11:19 scripts
drwxr-xr-x 2 root root 4.0K Sep 5 11:19 log
[root@v1ex2dbadm01 exachk_v1ex2dbadm01_VERSREP_090517_105859]#

It’s normally easier to copy the zip file off the compute node to your machine and then unzip and view the report local.

It will look like this:

exchk

Known Issues

1. Running ExaChk reset snmpd.conf and iptables on Storage Cells.  There’s a bug currently open with Oracle Support for this:
Bug 26742216 – EXACHK IS RESETTING IPTABLES AND IT IS BLOCKING SNMP CONNECTIONS UNTIL RESTART

Which hasn’t yet been fixed.  See my blog post with discuss how to fix this after Exadata Patching where this also happens:
SNMP unresponsive on Storage Cell after Exadata Patching

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Advertisements

How to update OPatch

When applying patches, such as PSUs or one-offs, you may need to update OPatch to meet the minimum OPatch version.  It is also recommended to update OPatch when applying any patch.

To see your current OPatch version:

[oracle@v1ex1dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
[oracle@v1ex1dbadm01 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.1.0.1.3

OPatch succeeded.
[oracle@v1ex1dbadm01 ~]$

Backup existing OPatch:

[oracle@v1ex1dbadm01 ~]$ cd $ORACLE_HOME
[oracle@v1ex1dbadm01 dbhome_1]$ tar -cvf OPatch_backup.tar OPatch/*
OPatch/datapatch
OPatch/datapatch.bat
OPatch/docs/
...
OPatch/oplan/README.txt
OPatch/oplan/README.html
OPatch/oplan/oplan
[oracle@v1ex1dbadm01 dbhome_1]$

Check the backup of OPatch:

[oracle@v1ex1dbadm01 dbhome_1]$ ls -lh | grep OPatch_backup.tar
-rw-rw-r--. 1 oracle oracle 6.7M Aug 29 11:51 OPatch_backup.tar
[oracle@v1ex1dbadm01 dbhome_1]$

Remove the existing OPatch:

[oracle@v1ex1dbadm01 dbhome_1]$ rm -rf OPatch

Unzip the latest OPatch:

[oracle@v1ex1dbadm01 dbhome_1]$ unzip -d $ORACLE_HOME ~/sw/p6880880_122010_Linux-x86-64.zip
Archive: /home/oracle/sw/p6880880_122010_Linux-x86-64.zip
creating: /u01/app/oracle/product/12.1.0/dbhome_1/OPatch/
inflating: /u01/app/oracle/product/12.1.0/dbhome_1/OPatch/datapatch
...
inflating: /u01/app/oracle/product/12.1.0/dbhome_1/OPatch/docs/cversion.txt
inflating: /u01/app/oracle/product/12.1.0/dbhome_1/OPatch/docs/FAQ
inflating: /u01/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch.bat
[oracle@v1ex1dbadm01 dbhome_1]$

Which can be found here:
OPatch – Where Can I Find the Latest Version of OPatch(6880880)? [Video] (Doc ID 224346.1)
OPATCH PLACEHOLDER Patch 6880880

To see your newOPatch version:

[oracle@v1ex1dbadm01 dbhome_1]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.9

OPatch succeeded.
[oracle@v1ex1dbadm01 dbhome_1]$

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to Checksum Downloads

As a DBA, in your time, you will need to download files such as Oracle binaries, patches, etc.  As a rule of thumb, you should always checksum your downloads, to ensure they are free of corruption.  In my example, I needed to download:
Patch 25440397: QUARTERLY FULL STACK DOWNLOAD PATCH FOR EXADATA (Apr 2017 – 12.1.0.2)

This is the Quarterly Full Stack Download Patch for Exadata, which is 10 zip files that total 13Gb.  That’s a lot of bytes and although highly unlikely, it’s possible for one of them to become corrupt.  Therefore, to avoid downstream pain, check the downloads’ md5 checksum on your machine.  The Microsoft File Checksum Integrity Verifier (FCIV), can be found here:
How to compute the MD5 or SHA-1 cryptographic hash values for a file

Below shows the downloads’ MD5 checksum using FCIV:

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_1of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
855c3d128adb7f87eee04d7615f20e2b c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_1of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_2of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
93cea0dddd371e148f6cb3e35f9faa6a c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_2of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_3of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
22318acb7fa36154f0bb6d02115daba4 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_3of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_4of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
28957a7ac3f2b05625cf3058fc97d981 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_4of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_5of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
a1cdad9db470bb10eb14d85c3240fa64 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_5of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_6of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
f358ebbf9e2fc222795176091939054c c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_6of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_7of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
3d20be91084b10fcb6fe3aa045346bf1 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_7of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_8of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
7cebaca222e384957b8e65df0de6e6b1 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_8of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_9of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
5a79f025564427a14ab7adce31d1a8bb c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_9of10.zip

C:\Users\anwarz>C:\Users\anwarz\Downloads\md5\fciv.exe "C:\Users\anwarz\Exadata Patching 2017\p25440397_121020_Linux-x86-64_10of10.zip"
//
// File Checksum Integrity Verifier version 2.05.
//
34b8bfe3792fa9dbd0cd13ba61bc5696 c:\users\anwarz\exadata patching 2017\p25440397_121020_linux-x86-64_10of10.zip

C:\Users\anwarz>

The MD5 checksums then can be verified with what Oracle Support list under the patch, click ‘Download’, then ‘View Digest Details’:

Patch_MD5

Once checked on your machine and then you transfer to server (assuming you didn’t download directly to server using WGET or something), then check again using md5sum:

[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_1of10.zip
 855c3d128adb7f87eee04d7615f20e2b p25440397_121020_linux-x86-64_1of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_2of10.zip
93cea0dddd371e148f6cb3e35f9faa6a p25440397_121020_linux-x86-64_2of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_3of10.zip
22318acb7fa36154f0bb6d02115daba4 p25440397_121020_linux-x86-64_3of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_4of10.zip
28957a7ac3f2b05625cf3058fc97d981 p25440397_121020_linux-x86-64_4of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_5of10.zip
a1cdad9db470bb10eb14d85c3240fa64 p25440397_121020_linux-x86-64_5of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_6of10.zip
f358ebbf9e2fc222795176091939054c p25440397_121020_linux-x86-64_6of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_7of10.zip
3d20be91084b10fcb6fe3aa045346bf1 p25440397_121020_linux-x86-64_7of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_8of10.zip
7cebaca222e384957b8e65df0de6e6b1 p25440397_121020_linux-x86-64_8of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_9of10.zip
5a79f025564427a14ab7adce31d1a8bb p25440397_121020_linux-x86-64_9of10.zip
[oracle@v1oex1dbadm01 25440397]$ md5sum p25440397_121020_linux-x86-64_10of10.zip
34b8bfe3792fa9dbd0cd13ba61bc5696 p25440397_121020_linux-x86-64_10of10.zip
[oracle@v1oex1dbadm01 25440397]$

Once all checked, you’re free to unzip, knowing you’re free from corruption 🙂

Mind you, that doesn’t guarantee you wouldn’t get corruption from I/O but that would be extremely unlucky and you’ll have bigger problems.

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to exit out of ed in SQL*Plus

I, from time to time, get stuck in ‘ed‘ on SQL*Plus because the editor not defined like this:

SQL> select * from duall;
select * from duall
 *
ERROR at line 1:
ORA-00942: table or view does not exist

SQL> ed
Wrote file afiedt.buf
22
^C
?
^Z
?

The Ctrl-C and Ctrl-Z, wouldn’t get you out.

The solution is simple, type ‘q‘ and press ‘Enter‘, then set editor using ‘define_editor=vi‘ and try again 🙂 :

q

1* select * from duall
SQL> define_editor=vi
SQL> ed
Wrote file afiedt.buf

1* select * from dual
SQL> /

D
-
X

SQL>

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Extending a Non-root LVM Partition on Exadata

On an Oracle Exadata Database Machine, the ‘/u01’ is defaulted to a size of 100Gb, which can easily fill up.  Luckily this is just a Logical Volume and there’s normally lots of space available on the Logical Volume Group which is usually untapped.

 

Extending ‘/u01’

Identify how much space is used and free on ‘/u01’ using df:

[root@v1ex1dbadm01 ~]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1
99G 78G 17G 83% /u01
[root@v1ex1dbadm01 ~]#

Display the current logical volume configuration using the lvs command:

[root@v1ex1dbadm01 ~]# lvs -o lv_name,lv_path,vg_name,lv_size
LV Path VG LSize
LVDbOra1 /dev/VGExaDb/LVDbOra1 VGExaDb 100.00g
LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb 24.00g
LVDbSys1 /dev/VGExaDb/LVDbSys1 VGExaDb 30.00g
LVDbSys2 /dev/VGExaDb/LVDbSys2 VGExaDb 30.00g
LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb 1.00g
[root@v1ex1dbadm01 ~]#

Check the online resize option is available using the tune2fs command:

[root@v1ex1dbadm01 ~]# tune2fs -l /dev/VGExaDb/LVDbOra1 | grep resize_inode
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
[root@v1ex1dbadm01 ~]#

If not available then the file system needs to be unmounted before resizing.  Refer to documentation:

Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

Verify there’s space available in the Logical Volume Group using vgdisplay command:

[root@v1ex1dbadm01 ~]# vgdisplay -s
"VGExaDb" 1.63 TiB [185.00 GiB used / 1.45 TiB free]
[root@v1ex1dbadm01 ~]#

Finally if there’s enough space, then extend the Logical Volume using lvextend command:

[root@v1ex1dbadm01 ~]# lvextend -L +100G /dev/VGExaDb/LVDbOra1
Size of logical volume VGExaDb/LVDbOra1 changed from 100.00 GiB (25600 extents) to 200.00 GiB (51200 extents).
Logical volume LVDbOra1 successfully resized.
[root@v1ex1dbadm01 ~]#

Followed by a resize of the file system using resize2fs command:

[root@v1ex1dbadm01 ~]# resize2fs /dev/VGExaDb/LVDbOra1
resize2fs 1.43-WIP (20-Jun-2013)
Filesystem at /dev/VGExaDb/LVDbOra1 is mounted on /u01; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 13
Performing an on-line resize of /dev/VGExaDb/LVDbOra1 to 52428800 (4k) blocks.
The filesystem on /dev/VGExaDb/LVDbOra1 is now 52428800 blocks long.
[root@v1ex1dbadm01 ~]#

You should now see ‘/u01’ with additional 100Gb less formatting:

[root@v1ex1dbadm01 ~]# df -h /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbOra1
197G 78G 110G 42% /u01
[root@v1ex1dbadm01 ~]#

Also see the Logical Volume is now 200Gb from 100Gb:

[root@v1ex1dbadm01 ~]# lvs -o lv_name,lv_path,vg_name,lv_size
LV Path VG LSize
LVDbOra1 /dev/VGExaDb/LVDbOra1 VGExaDb 200.00g
LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb 24.00g
LVDbSys1 /dev/VGExaDb/LVDbSys1 VGExaDb 30.00g
LVDbSys2 /dev/VGExaDb/LVDbSys2 VGExaDb 30.00g
LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb 1.00g
[root@v1ex1dbadm01 ~]#

 

Documentation for reference:

Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Cloning an Oracle Home

You may wish to clone an Oracle Home, for example you have all your databases on a single Oracle Home but you want to separate Development from Test.  This could be so you can soak test Patch Set Updates (PSU) on Development before applying to Test and then Production.  Or you might wish to have 2 Oracle Homes, so you can patch one and then switch all databases to the patched Oracle Home for minimal downtime.

 

Copying the Oracle Home

First you need to copy the Oracle Home at file level using cp as user root as shown below:

[root@v1ex2dbadm01 ~]# cp -Rp /u01/app/oracle/product/12.1.0.2/dbhome_1 /u01/app/oracle/product/12.1.0.2/dbhome_2

Then check the Oracle Home and the Cloned Oracle Home are the same size:

[root@v1ex2dbadm01 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_1 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_1
[root@v1ex2dbadm01 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_2 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_2

Then repeat on all the other nodes:

[root@v1ex2dbadm02 ~]# cp -Rp /u01/app/oracle/product/12.1.0.2/dbhome_1 /u01/app/oracle/product/12.1.0.2/dbhome_2
[root@v1ex2dbadm02 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_1 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_1
[root@v1ex2dbadm02 ~]# du -h /u01/app/oracle/product/12.1.0.2/dbhome_2 --max-depth=0
12G /u01/app/oracle/product/12.1.0.2/dbhome_2

 

Cloning the Oracle Home

Now we have a copy of the Oracle Home, we next need to clone using the clone.pl perl script as shown below:

/usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
'-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
'-O"LOCAL_NODE=v1ex2dbadm01"' ORACLE_BASE=/u01/app/oracle \
ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'

CLUSTER_NODES = all the nodes in the cluster
LOCAL_NODE = The node you are running the command on
ORACLE_BASE = The Oracle Base defined on the Server
ORACLE_HOME = The Cloned Oracle Home, already exported
ORACLE_HOME_NAME = The name you wish to give to the Cloned Oracle Home

[root@v1ex2dbadm01 ~]# su - oracle
[oracle@v1ex2dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ /usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
> '-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
> '-O"LOCAL_NODE=v1ex2dbadm01"' ORACLE_BASE=/u01/app/oracle \
> ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'
./runInstaller -clone -waitForCompletion "CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}" "LOCAL_NODE=v1ex2dbadm01" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2" "ORACLE_HOME_NAME=OraDB12Home2" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_2/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 5392 MB Passed
Checking swap space: must be greater than 500 MB. Actual 24532 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-06-21_02-57-14PM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2017-06-21_02-57-14PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 ..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12Home2 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2017-06-21_02-57-14PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
 .................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh on the following nodes:
[v1ex2dbadm01]

.................................................. 100% Done.

[oracle@v1ex2dbadm01 ~]$

Next check the re-linking is RDS not UDP:

[oracle@v1ex2dbadm01 ~]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/skgxpinfo
rds
[oracle@v1ex2dbadm01 ~]$

If UDP, then relink using command below:

cd $ORACLE_HOME/rdbms/lib; ORACLE_HOME=$ORACLE_HOME make -f ins_rdbms.mk ipc_rds ioracle

Then repeat on all the other nodes, remember to change LOCAL_NODE:

[root@v1ex2dbadm02 ~]# su - oracle
[oracle@v1ex2dbadm02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm02 ~]$ /usr/bin/perl $ORACLE_HOME/clone/bin/clone.pl \
> '-O"CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}"' \
> '-O"LOCAL_NODE=v1ex2dbadm02"' ORACLE_BASE=/u01/app/oracle \
> ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB12Home2 '-O-noConfig'
./runInstaller -clone -waitForCompletion "CLUSTER_NODES={v1ex2dbadm01,v1ex2dbadm02}" "LOCAL_NODE=v1ex2dbadm02" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2" "ORACLE_HOME_NAME=OraDB12Home2" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0.2/dbhome_2/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 10725 MB Passed
Checking swap space: must be greater than 500 MB. Actual 24565 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-06-21_03-05-37PM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2017-06-21_03-05-37PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 ..........
Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraDB12Home2 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2017-06-21_03-05-37PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
 .................................................. 95% Done.

As a root user, execute the following script(s):
1. /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh

Execute /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh on the following nodes:
[v1ex2dbadm02]

.................................................. 100% Done.

[oracle@v1ex2dbadm02 ~]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/skgxpinfo
rds
[oracle@v1ex2dbadm02 ~]$

Next need to run root.sh as shown below:

[oracle@v1ex2dbadm01 ~]$ exit
logout
[root@v1ex2dbadm01 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_2/install/root_v1ex2dbadm01.v1.com_2017-06-21_15-04-16.log for the output of root script
[root@v1ex2dbadm01 ~]#

Then repeat on all the other nodes:

[oracle@v1ex2dbadm02 ~]$ exit
logout
[root@v1ex2dbadm02 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_2/root.sh
Check /u01/app/oracle/product/12.1.0.2/dbhome_2/install/root_v1ex2dbadm02.v1.com_2017-06-21_15-06-49.log for the output of root script
[root@v1ex2dbadm02 ~]#

Verify that the Cloned Oracle Home comprises of all the nodes in the cluster:

[root@v1ex2dbadm01 ~]# su - oracle
[oracle@v1ex2dbadm01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
Local node = v1ex2dbadm01
Remote node = v1ex2dbadm02
[oracle@v1ex2dbadm01 ~]$

Then repeat on all the other nodes:

[root@v1ex2dbadm02 ~]# su - oracle
[oracle@v1ex2dbadm02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm02 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
Local node = v1ex2dbadm02
Remote node = v1ex2dbadm01
[oracle@v1ex2dbadm02 ~]$

 

Switching Databases to Cloned Oracle Home

Change the Oracle Home using server control:

[oracle@v1ex2dbadm01 ~]$ srvctl config database -d V1DEV -a
Database unique name: V1DEV
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATAC1/V1DEV/PARAMETERFILE/spfileV1DEV.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: v1jobservice
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dba
Database instances: V1DEV1,V1DEV2
Configured nodes: v1ex2dbadm01,v1ex2dbadm02
Database is administrator managed
[oracle@v1ex2dbadm01 ~]$ srvctl modify database -d V1DEV -o /u01/app/oracle/product/12.1.0.2/dbhome_2
[oracle@v1ex2dbadm01 ~]$ srvctl config database -d V1DEV -a
Database unique name: V1DEV
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2
Oracle user: oracle
Spfile: +DATAC1/V1DEV/PARAMETERFILE/spfileV1DEV.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: v1jobservice
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dba
Database instances: V1DEV1,V1DEV2
Configured nodes: v1ex2dbadm01,v1ex2dbadm02
Database is administrator managed
[oracle@v1ex2dbadm01 ~]$

Next change /etc/oratab to reflect the new Oracle Home:

[oracle@v1ex2dbadm01 ~]$ vi /etc/oratab
[oracle@v1ex2dbadm01 ~]$ more /etc/oratab | grep dbhome_2
V1DEV1:/u01/app/oracle/product/12.1.0.2/dbhome_2:N # line added by Agent

Then repeat on all the other nodes:

[oracle@v1ex2dbadm02 ~]$ vi /etc/oratab
[oracle@v1ex2dbadm02 ~]$ more /etc/oratab | grep dbhome_2
V1DEV2:/u01/app/oracle/product/12.1.0.2/dbhome_2:N # line added by Agent

Now we rolling bounce the database:

[oracle@v1ex2dbadm01 ~]$ srvctl status database -d V1DEV -v
Instance V1DEV1 is running on node v1ex2dbadm01. Instance status: Open,Running from Old Oracle Home.
Instance V1DEV2 is running on node v1ex2dbadm02 with online services v1jobservice. Instance status: Open,Running from Old Oracle Home.
[oracle@v1ex2dbadm01 ~]$ srvctl stop instance -d V1DEV -i V1DEV1 -f
[oracle@v1ex2dbadm01 ~]$ srvctl start instance -d V1DEV -i V1DEV1
[oracle@v1ex2dbadm01 ~]$ srvctl stop instance -d V1DEV -i V1DEV2 -f
[oracle@v1ex2dbadm01 ~]$ srvctl start instance -d V1DEV -i V1DEV2
[oracle@v1ex2dbadm01 ~]$ srvctl status database -d V1DEV -v
Instance V1DEV1 is running on node v1ex2dbadm01. Instance status: Open.
Instance V1DEV2 is running on node v1ex2dbadm02 with online services v1jobservice. Instance status: Open.
[oracle@v1ex2dbadm01 ~]$

 

Related My Oracle Support (MOS) notes:

Master Note For Cloning Oracle Database Server ORACLE_HOME’s Using the Oracle Universal Installer (OUI) (Doc ID 1154613.1)

Cloning An Existing Oracle Database 12c Release 1 (12.1.0.x) RDBMS Installation Using OUI (Doc ID 1493677.1)

Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories (Doc ID 1136544.1)

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

SNMP unresponsive on Storage Cell after Exadata Patching

After a round of Exadata patching, the Storage Cells become unresponsive to SNMP for monitoring tools like OpsViews:

check_snmp_sysinfo CRITICAL - Agent not responding, tried SNMP v1 and v2c

This is because the Storage Cells get re-imaged as part of the Exadata patching, confirmed by imagehistory:

[root@v1ex1celadm01 ~]# imagehistory
Version : 12.1.2.1.1.150316.2
Image activation date : 2015-05-01 16:10:10 -0700
Imaging mode : fresh
Imaging status : success

Version : 12.1.2.1.3.151021
Image activation date : 2015-12-16 05:25:21 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.2.1.160330
Image activation date : 2016-05-12 14:02:43 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.2.160721
Image activation date : 2016-10-05 01:45:42 +0100
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.3.161208
Image activation date : 2017-01-18 02:53:10 +0000
Imaging mode : out of partition upgrade
Imaging status : success

Version : 12.1.2.3.4.170111
Image activation date : 2017-05-23 10:58:24 +0100
Imaging mode : out of partition upgrade
Imaging status : success

[root@v1oex1celadm01 ~]#

Which knocks out the configuration in iptables and snmpd.conf.

To resolve add back in the lines for your primary and secondary monitoring servers, for example:

rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32

So looks like this:

[root@v1ex1celadm01 ~]# more /etc/snmp/snmpd.conf
trapcommunity public
trapsink 127.0.0.1 public
rocommunity public 127.0.0.1
rocommunity V12V1 192.168.0.31
rocommunity V12V1 192.168.0.32
rwcommunity public 127.0.0.1

access RWGroup "" any noauth exact all all all
com2sec snmpclient 127.0.0.1 public
group RWGroup v1 snmpclient

pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat
pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
syslocation Unknown (edit /etc/snmp/snmpd.conf)

view all included .1 80

[root@v1ex1celadm01 ~]#

Then reload snmp using the user root:

[root@v1ex2celadm01 ~]# service snmpd reload
Reloading snmpd: [ OK ]
[root@v1ex2celadm01 ~]#

Next add the monitoring servers primary and secondary to iptables as follows:

[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.31 -p udp --dport 161 -j ACCEPT
[root@v1ex1celadm01 ~]# iptables -I INPUT -s 192.168.0.32 -p udp --dport 161 -j ACCEPT

And make permanent by saving to firewall rules:

[root@v1ex1dbadm01 ~]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@v1ex1dbadm01 ~]#

Now your monitoring should return as SNMP connectivity is restored 🙂 :

SNMP OK - "Linux v1ex1celadm01.v1.com 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64"

 

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)