
Oracle customers who are fortunate to have an Oracle Exadata Database Machine, will need to run “exachk” from time to time, which is pronounced as Exa-Check. This tool checks your Oracle Exadata Database Machine configuration, software, critical issue and provides a Maximum Availability Architecture score. It is required to run before and after Exadata patching and is good practice to run on a frequent basis and review the recommendations.
More info here:
Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)
“exachk holistically evaluates all Oracle Engineered Systems.
It includes:
- Configuration checks for Database Servers, Storage Servers and InfiniBand Switches
- Grid Infrastructure, Database and ASM and operating system software checks
- MAA Scorecard which conducts an automatic MAA Review
- Exadata Software Planner, Software prechecks, Exadata and Database Critical Issue alerts
All checks have explanations, recommendations, and manual verification commands so that customers can self-correct all FAIL and WARNING conditions reported.
Development recommends that the latest exachk be executed with the following frequency:
- Monthly
- Week before any planned maintenance activity
- Day before any planned maintenance activity
- Immediately after completion of planned maintenance activity or an outage or incident”
The download can be obtained from the above MOS note 1070954.1. Place the download zip file as root on a compute node, in the Oracle’s Home for example, I use “/home/oracle/version1/exachk”. Within this folder create another folder called exachk, which allows you to easily replace the tool with the latest version and unzip the zip file to this folder:
[root@v1ex2dbadm01 exachk]# pwd /home/oracle/version1/exachk [root@v1ex2dbadm01 exachk]# rm -rf exachk [root@v1ex2dbadm01 exachk]# mkdir exachk [root@v1ex2dbadm01 exachk]# ls -lh total 19M drwxr-xr-x 2 root root 4.0K Sep 5 10:53 exachk -rw-r--r-- 1 root root 19M Sep 5 10:51 exachk.zip [root@v1ex2dbadm01 exachk]# unzip exachk.zip -d ./exachk Archive: exachk.zip creating: ./exachk/.cgrep/ inflating: ./exachk/.cgrep/versions.dat ... inflating: ./exachk/UserGuide.txt inflating: ./exachk/readme.txt inflating: ./exachk/doc/ORAchk_and_EXAchk_User_Guide.pdf [root@v1ex2dbadm01 exachk]#
You’ll now see the tool “exachk” as an executable along with the other files as part of the tool, including the User Guide:
[root@v1ex2dbadm01 exachk]# ls exachk exachk.zip [root@v1ex2dbadm01 exachk]# ls -l exachk total 69792 -r-xr-xr-x 1 root root 8218911 Apr 4 12:12 Apex5_CollectionManager_App.sql -r-xr-xr-x 1 root root 4816355 Sep 15 2016 CollectionManager_App.sql -rw-r--r-- 1 root root 47172483 Jul 19 08:38 collections.dat drwxr-xr-x 2 root root 4096 Sep 5 10:54 doc -r-xr-xr-x 1 root root 2897015 May 17 20:59 exachk -rw-r--r-- 1 root root 1976299 Jul 19 09:03 EXAchk_Health_Check_Catalog.html drwxr-xr-x 2 root root 4096 Jul 19 03:17 exadiscover -r--r--r-- 1 root root 4896 Mar 31 09:58 readme.txt -rw-r--r-- 1 root root 6302687 Jul 19 08:38 rules.dat -r-xr-xr-x 1 root root 40052 Jul 22 2015 sample_user_defined_checks.xml drwxr-xr-x 2 root root 4096 Jul 19 03:17 templates -r-xr-xr-x 1 root root 2888 Oct 9 2015 user_defined_checks.xsd -r--r--r-- 1 root root 234 Apr 1 10:05 UserGuide.txt [root@v1ex2dbadm01 exachk]# cd exachk [root@v1ex2dbadm01 exachk]#
To run the tool, simply execute without any switches (there are other options, refer to User Guide):
[root@v1ex2dbadm01 exachk]# ./exachk Checking for prompts on v1ex2dbadm01 for oracle user... Checking ssh user equivalency settings on all nodes in cluster Node v1ex2dbadm02 is configured for ssh user equivalency for root user Checking for prompts for oracle user on all nodes... Searching for running databases . . . . . . . . . . . . . . . . . List of running databases registered in OCR 1. VERSS 2. VERSDEV 3. VERS24H 4. IONS 5. IONDEV 6. VERSREP 7. All of above 8. None of above Select databases from list for checking best practices. For multiple databases, select 7 for All or comma separated number like 1,2 etc [1-8][7].7 Searching out ORACLE_HOME for selected databases. . . . . . . . . . . . . . . . . . . . . . . . . . Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------- Oracle Stack Status ------------------------------------------------------------------------------------------------------- Host Name CRS Installed RDBMS Installed CRS UP ASM UP RDBMS UP DB Instance Name ------------------------------------------------------------------------------------------------------- v1ex2dbadm01 Yes Yes Yes Yes Yes IONDEV1 IONS1 VERS24H1 VERSDEV1 VERSREP1 VERSS1 v1ex2dbadm02 Yes Yes Yes Yes Yes IONDEV2 IONS2 VERS24H2 VERSDEV2 VERSREP2 VERSS2 ------------------------------------------------------------------------------------------------------- v1ex2cel01 is configured for ssh user equivalency for root user . v1ex2cel02 is configured for ssh user equivalency for root user . v1ex2cel03 is configured for ssh user equivalency for root user . . . . . . . . . . . . . . . . . . . v1ex2sw-ibb01 is configured for ssh user equivalency for root user v1ex2sw-iba01 is configured for ssh user equivalency for root user v1ex2sw-ibb01 is configured for ssh user equivalency for root user *** Checking Best Practice Recommendations (PASS/WARNING/FAIL) *** Collections and audit checks log file is /home/oracle/version1/exachk/exachk/exachk_v1ex2dbadm01_VERSREP_090517_105859/log/exachk.log Checking for prompts in /root/.bash_profile on v1ex2dbadm01 for root user... Checking for prompts in /root/.bash_profile on v1ex2dbadm02 for root user... Starting to run exachk in background on v1ex2dbadm02 . . ============================================================= Node name - v1ex2dbadm01 ============================================================= . . . . . Collecting - ASM Disk Group for Infrastructure Software and Configuration Collecting - ASM Diskgroup Attributes Collecting - ASM diskgroup usable free space Collecting - ASM initialization parameters Collecting - Database Parameters for IONDEV database Collecting - Database Parameters for IONS database Collecting - Database Parameters for VERSDEV database Collecting - Database Parameters for VERSREP database Collecting - Database Parameters for VERSS database Collecting - Database Undocumented Parameters for IONDEV database Collecting - Database Undocumented Parameters for VERSDEV database Collecting - Database Undocumented Parameters for VERSREP database Collecting - RDBMS Feature Usage for IONDEV database Collecting - RDBMS Feature Usage for VERSDEV database Collecting - RDBMS Feature Usage for VERSREP database Collecting - CPU Information Collecting - Clusterware and RDBMS software version Collecting - Compute node PCI bus slot speed for infiniband HCAs Collecting - Kernel parameters Collecting - Maximum number of semaphore sets on system Collecting - Maximum number of semaphores on system Collecting - OS Packages Collecting - Patches for Grid Infrastructure Collecting - Patches for RDBMS Home Collecting - RDBMS patch inventory Collecting - Switch Version Information Collecting - number of semaphore operations per semop system call Collecting - CRS user limits configuration Collecting - CRS user time zone check Collecting - Check alerthistory for non-test open stateless alerts [Database Server] Collecting - Check alerthistory for stateful alerts not cleared [Database Server] Collecting - Check alerthistory for test open stateless alerts [Database Server] Collecting - Clusterware patch inventory Collecting - Detect duplicate files in /etc/*init* directories Collecting - Discover switch type(spine or leaf) Collecting - Enterprise Manager agent targets Collecting - Exadata Critical Issue DB09 Collecting - Exadata Critical Issue EX30 Collecting - Exadata software version on database server Collecting - Exadata system model number Collecting - Exadata version on database server Collecting - HCA firmware version on database server Collecting - HCA transfer rate on database server Collecting - Infrastructure Software and Configuration for compute Collecting - MaxStartups setting in sshd_config Collecting - OFED Software version on database server Collecting - Obtain hardware information Collecting - Operating system and Kernel version on database server Collecting - Oracle monitoring agent and/or OS settings on ADR diagnostic directories Collecting - Raid controller bus link speed Collecting - System Event Log Collecting - Validate key sysctl.conf parameters on database servers Collecting - Verify Ambient Air Temperature [Database Server] Collecting - Verify Data Network is Separate from Management Network Collecting - Verify Database Server Disk Controller Configuration Collecting - Verify Database Server Physical Drive Configuration Collecting - Verify Database Server Virtual Drive Configuration Collecting - Verify Disk Cache Policy on database server Collecting - Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Database Server] Collecting - Verify ILOM Power Up Configuration for HOST_AUTO_POWER_ON Collecting - Verify ILOM Power Up Configuration for HOST_LAST_POWER_STATE Collecting - Verify IP routing configuration on database servers Collecting - Verify InfiniBand Address Resolution Protocol (ARP) Configuration on Database Servers Collecting - Verify InfiniBand Fabric Topology (verify-topology) Collecting - Verify InfiniBand subnet manager is not running on database server Collecting - Verify InfiniBand subnet manager is running on an InfiniBand switch Collecting - Verify Master (Rack) Serial Number is Set [Database Server] Collecting - Verify NTP configuration on database servers Collecting - Verify Quorum disks configuration Collecting - Verify RAID Controller Battery Temperature [Database Server] Collecting - Verify RAID disk controller CacheVault capacitor condition [Database Server] Collecting - Verify TCP Segmentation Offload (TSO) is set to off Collecting - Verify active kernel version matches expected version for installed Exadata Image Collecting - Verify basic Logical Volume(LVM) system devices configuration Collecting - Verify database server disk controllers use writeback cache Collecting - Verify database server file systems have Check interval = 0 Collecting - Verify database server file systems have Maximum mount count = -1 Collecting - Verify imageinfo on database server Collecting - Verify imageinfo on database server to compare systemwide Collecting - Verify key InfiniBand fabric error counters are not present Collecting - Verify no database server kernel out of memory errors Collecting - Verify service exachkcfg autostart status on database server Collecting - Verify that the SDP over IB option sdp_apm_enable is set to 0 Collecting - Verify the localhost alias is pingable [Database Server] Collecting - Verify the Name Service Cache Daemon (NSCD) is Running Collecting - Verify the file /.updfrm_exact does not exist [Database Server] Collecting - Verify the grid Infrastructure management database (MGMTDB) does not use hugepages Collecting - Verify the vm.min_free_kbytes configuration Collecting - root time zone check Collecting - verify asr exadata configuration check via ASREXACHECK on database server Starting to run root privileged commands in background on STORAGE SERVER v1ex2cel01 (10.1.11.15) Starting to run root privileged commands in background on STORAGE SERVER v1ex2cel02 (10.1.11.17) Starting to run root privileged commands in background on STORAGE SERVER v1ex2cel03 (10.1.11.19) Starting to run root privileged commands in background on INFINIBAND SWITCH v1ex2sw-ibb01 Starting to run root privileged commands in background on INFINIBAND SWITCH v1ex2sw-iba01 Collections from STORAGE SERVER: ---------------------------------- Collecting - Exadata Critical Issue EX10 Collecting - Exadata Critical Issue EX11 Collecting - Exadata Critical Issue EX22 Collecting - Exadata Critical Issue EX28 Collecting - Exadata Critical Issue EX31 Collecting - Exadata critical issue EX14 Collecting - Exadata critical issue EX16 Collecting - Exadata critical issue EX17 Collecting - Exadata critical issue EX23 Collecting - Exadata critical issue EX24 Collecting - Exadata software version on storage server Collecting - Exadata software version on storage servers Collecting - Exadata storage server system model number Collecting - Infrastructure Software and Configuration for storage Collecting - RAID controller version on storage servers Collecting - Verify Ambient Air Temperature [Storage Server] Collecting - Verify Disk Cache Policy on storage servers Collecting - Verify Exadata Smart Flash Cache is created Collecting - Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Storage Server] Collecting - Verify ILOM Power Up Configuration for HOST_AUTO_POWER_ON on storage servers Collecting - Verify ILOM Power Up Configuration for HOST_LAST_POWER_STATE on storage servers Collecting - Verify InfiniBand subnet manager is not running on storage server Collecting - Verify Master (Rack) Serial Number is Set [Storage Server] Collecting - Verify NTP configuration on storage servers Collecting - Verify OSSCONF/cellinit.ora consistency across storage servers Collecting - Verify RAID Controller Battery Temperature [Storage Server] Collecting - Verify RAID disk controller CacheVault capacitor condition [Storage Server] Collecting - Verify Storage Server user CELLDIAG exists Collecting - Verify active system values match those defined in configuration file cell.conf [Storage Server] Collecting - Verify data (non-system) disks on Exadata Storage Servers have no partitions Collecting - Verify imageinfo on storage server Collecting - Verify imageinfo on storage server to compare systemwide Collecting - Verify release tracking bug on storage servers Collecting - Verify service exachkcfg autostart status on storage server Collecting - Verify storage server disk controllers use writeback cache Collecting - Verify the localhost alias is pingable [Storage Server] Collecting - Verify the file /.updfrm_exact does not exist [Storage Server] Collecting - verify asr exadata configuration check via ASREXACHECK on storage servers Collecting - Check alerthistory for non-test open stateless alerts [Storage Server] Collecting - Check alerthistory for stateful alerts not cleared [Storage Server] Collecting - Check alerthistory for test open stateless alerts [Storage Server] Collecting - Configure Storage Server alerts to be sent via email Collecting - Determine storage server type(All Flash/High Capacity) Collecting - Exadata Celldisk predictive failures Collecting - Exadata storage server root filesystem free space Collecting - HCA firmware version on storage server Collecting - OFED Software version on storage server Collecting - Operating system and Kernel version on storage server Collecting - Storage server flash cache mode Collecting - Storage server make and model Collecting - Verify Data Network is Separate from Management Network on storage server Collecting - Verify Datafiles are Placed on Diskgroups consisting of griddisks with correct attributes Collecting - Verify Ethernet Cable Connection Quality on storage servers Collecting - Verify ExaWatcher is executing [Storage Server] Collecting - Verify Exadata Smart Flash Cache is actually in use Collecting - Verify Exadata Smart Flash Cache status is normal Collecting - Verify Exadata Smart Flash Log is Created Collecting - Verify InfiniBand Cable Connection Quality on storage servers Collecting - Verify average ping times to DNS nameserver [Storage Server] Collecting - Verify celldisk configuration on disk drives Collecting - Verify celldisk configuration on flash memory devices Collecting - Verify griddisk ASM status Collecting - Verify griddisk count matches across all storage servers where a given prefix name exists Collecting - Verify storage server metric CD_IO_ST_RQ Collecting - Verify the percent of available celldisk space used by the griddisks Collecting - Verify there are no griddisks configured on flash memory devices Collecting - Verify total number of griddisks with a given prefix name is evenly divisible of celldisks Collecting - mpt_cmd_retry_count from /etc/modprobe.conf on Storage Servers Collections from INFINIBAND SWITCH: ------------------------------------ Collecting - Exadata Critical Issue IB5 Collecting - Hostname in /etc/hosts Collecting - Infiniband Switch NTP configuration Collecting - Infiniband subnet manager status Collecting - Infiniband switch HCA status Collecting - Infiniband switch HOSTNAME configuration Collecting - Infiniband switch firmware version Collecting - Infiniband switch health Collecting - Infiniband switch localtime configuration Collecting - Infiniband switch module configuration Collecting - Infiniband switch subnet manager configuration Collecting - Infiniband switch type(Spine or leaf) Collecting - Infrastructure Software and Configuration for switch Collecting - Verify average ping times to DNS nameserver [IB Switch] Collecting - Verify no IB switch ports disabled due to excessive symbol errors Collecting - Verify the localhost alias is pingable [IB Switch] Collecting - sm_priority configuration on Infiniband switch Data collections completed. Checking best practices on v1ex2dbadm01. -------------------------------------------------------------------------------------- FAIL => One or more storage servers have stateful alerts that have not been cleared. FAIL => DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for VERSDEV FAIL => DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for IONDEV FAIL => DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for VERSREP WARNING => Hidden ASM Initialization Parameter usage is not correct WARNING => ASM parameter AUDIT_SYSLOG_LEVEL should be set to the recommended value WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on VERSS1 instance WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on IONS1 instance FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSDEV1 instance FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on IONDEV1 instance FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSREP1 instance WARNING => ASM parameter ASM_POWER_LIMIT is not set to the default value. FAIL => One or more database servers have stateful alerts that have not been cleared INFO => Oracle GoldenGate failure prevention best practices INFO => One or more non-default AWR baselines should be created for VERSDEV INFO => One or more non-default AWR baselines should be created for IONDEV INFO => One or more non-default AWR baselines should be created for VERSREP WARNING => Local archive destination does not alternate destination configured for VERSDEV WARNING => Local archive destination does not alternate destination configured for IONDEV WARNING => Local archive destination does not alternate destination configured for VERSREP WARNING => System has fewer than five storage servers but does not have quorum disks configured on database servers. INFO => Validate database security configuration using database security assessment tool for VERSS INFO => Validate database security configuration using database security assessment tool for VERSDEV INFO => Validate database security configuration using database security assessment tool for IONS INFO => Validate database security configuration using database security assessment tool for IONDEV INFO => Validate database security configuration using database security assessment tool for VERSREP FAIL => System is exposed to Exadata Critical Issue IB5 on infiniband switch v1ex2sw-ibb01 FAIL => System is exposed to Exadata Critical Issue IB5 on infiniband switch v1ex2sw-iba01 FAIL => Oracle monitoring agent and Operating systems settings on Automatic diagnostic repository directories are not correct or not all targets have been scanned or not all diagnostic directories found FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSDEV1 instance FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on IONDEV1 instance FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSREP1 instance FAIL => Storage Server user "CELLDIAG" should exist FAIL => Downdelay attribute is not set to recommended value on bonded client interface FAIL => Database control files are not configured as recommended for VERSS FAIL => Database control files are not configured as recommended for VERSDEV FAIL => Database control files are not configured as recommended for IONS FAIL => Database control files are not configured as recommended for IONDEV FAIL => Database control files are not configured as recommended for VERSREP FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSDEV FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for IONDEV FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSREP WARNING => SYS or SYSTEM objects were found to be INVALID for VERSDEV WARNING => SYS or SYSTEM objects were found to be INVALID for VERSREP FAIL => All disk groups should have compatible.rdbms attribute set to recommended values FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSS1 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSDEV1 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONS1 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONDEV1 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSREP1 instance FAIL => Storage Server alerts are not configured to be sent via email WARNING => filesystemio_options is not set to recommended value on VERSS1 instance WARNING => filesystemio_options is not set to recommended value on VERSDEV1 instance WARNING => filesystemio_options is not set to recommended value on IONS1 instance WARNING => filesystemio_options is not set to recommended value on IONDEV1 instance WARNING => filesystemio_options is not set to recommended value on VERSREP1 instance FAIL => Some data or temp files are not autoextensible for IONDEV WARNING => Key InfiniBand fabric error counters should not be present FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSS1 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSDEV1 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONS1 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONDEV1 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSREP1 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSS1 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSDEV1 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONS1 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONDEV1 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSREP1 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSS1 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSDEV1 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONS1 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONDEV1 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSREP1 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSS1 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSDEV1 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on IONS1 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSREP1 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSS1 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSDEV1 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on IONS1 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSREP1 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSS1 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSDEV1 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONS1 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONDEV1 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSREP1 instance FAIL => Database parameter sql92_security is not set to recommended value on VERSS1 instance FAIL => Database parameter sql92_security is not set to recommended value on VERSDEV1 instance FAIL => Database parameter sql92_security is not set to recommended value on IONS1 instance FAIL => Database parameter sql92_security is not set to recommended value on IONDEV1 instance FAIL => Database parameter sql92_security is not set to recommended value on VERSREP1 instance FAIL => ASM parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSDEV FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSREP FAIL => ASM processes parameter is not set to recommended value WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSDEV WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for IONDEV WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSREP FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for VERSS FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for IONS FAIL => Flashback on PRIMARY is not configured for IONDEV INFO => Operational Best Practices INFO => Database Consolidation Best Practices INFO => Computer failure prevention best practices INFO => Data corruption prevention best practices INFO => Logical corruption prevention best practices INFO => Database/Cluster/Site failure prevention best practices INFO => Client failover operational best practices WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSS1 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSDEV1 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONS1 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONDEV1 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSREP1 instance FAIL => ASM parameter MEMORY_TARGET should be set according to recommended value FAIL => ASM parameter PGA_AGGREGATE_TARGET should be set according to recommended value FAIL => ASM parameter MEMORY_MAX_TARGET should be set according to recommended value INFO => Verify the percent of available celldisk space used by the griddisks FAIL => Database control files are not configured as recommended for VERSS FAIL => Database control files are not configured as recommended for VERSDEV FAIL => Database control files are not configured as recommended for IONS FAIL => Database control files are not configured as recommended for IONDEV FAIL => Database control files are not configured as recommended for VERSREP INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for VERSS INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for IONS FAIL => Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for VERSDEV FAIL => Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for VERSREP INFO => Database failure prevention best practices FAIL => Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for VERSDEV FAIL => Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for IONDEV FAIL => Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for VERSREP FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSDEV1 instance FAIL => Database parameter LOG_BUFFER is not set to recommended value on IONDEV1 instance FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSREP1 instance WARNING => Redo log files should be appropriately sized for IONDEV WARNING => Redo log files should be appropriately sized for VERSREP FAIL => No one high redundancy diskgroup configured for VERSDEV FAIL => No one high redundancy diskgroup configured for IONDEV FAIL => No one high redundancy diskgroup configured for VERSREP INFO => Storage failures prevention best practices INFO => Software maintenance best practices FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSS1 instance FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSDEV1 instance FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSREP1 instance FAIL => The data files should be recoverable for VERSS FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for VERSS FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for IONS FAIL => FRA space management problem file types are present without an RMAN backup completion within the last 7 days. for VERSDEV FAIL => FRA space management problem file types are present without an RMAN backup completion within the last 7 days. for IONDEV INFO => Oracle recovery manager(rman) best practices WARNING => RMAN controlfile autobackup should be set to ON for VERSDEV WARNING => RMAN controlfile autobackup should be set to ON for IONDEV INFO => Exadata Critical Issues (Doc ID 1270094.1):- DB1-DB4,DB6,DB9-DB37, EX1-EX26,EX29-EX36 and IB1-IB3,IB5 FAIL => ASM parameter SGA_TARGET is not set according to recommended value. WARNING => Multiple Oracle database instances discovered, observe database consolidation best practices FAIL => There should be enough freespace in all diskgroups to reestablish redundancy after a single disk failure Collecting patch inventory on CRS HOME /u01/app/12.1.0.2/grid Collecting patch inventory on ASM HOME /u01/app/12.1.0.2/grid Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_1 Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_2 Copying results from v1ex2dbadm02 and generating report. This might take a while. Be patient. . . ============================================================= Node name - v1ex2dbadm02 ============================================================= . . . . . Collecting - CPU Information Collecting - Clusterware and RDBMS software version Collecting - Compute node PCI bus slot speed for infiniband HCAs Collecting - Kernel parameters Collecting - Maximum number of semaphore sets on system Collecting - Maximum number of semaphores on system Collecting - OS Packages Collecting - Patches for Grid Infrastructure Collecting - Patches for RDBMS Home Collecting - RDBMS patch inventory Collecting - number of semaphore operations per semop system call Collecting - CRS user limits configuration Collecting - CRS user time zone check Collecting - Check alerthistory for non-test open stateless alerts [Database Server] Collecting - Check alerthistory for stateful alerts not cleared [Database Server] Collecting - Check alerthistory for test open stateless alerts [Database Server] Collecting - Clusterware patch inventory Collecting - Detect duplicate files in /etc/*init* directories Collecting - Enterprise Manager agent targets Collecting - Exadata Critical Issue DB09 Collecting - Exadata Critical Issue EX30 Collecting - Exadata software version on database server Collecting - Exadata system model number Collecting - Exadata version on database server Collecting - HCA firmware version on database server Collecting - HCA transfer rate on database server Collecting - Infrastructure Software and Configuration for compute Collecting - MaxStartups setting in sshd_config Collecting - OFED Software version on database server Collecting - Obtain hardware information Collecting - Operating system and Kernel version on database server Collecting - Oracle monitoring agent and/or OS settings on ADR diagnostic directories Collecting - Raid controller bus link speed Collecting - System Event Log Collecting - Validate key sysctl.conf parameters on database servers Collecting - Verify Ambient Air Temperature [Database Server] Collecting - Verify Data Network is Separate from Management Network Collecting - Verify Database Server Disk Controller Configuration Collecting - Verify Database Server Physical Drive Configuration Collecting - Verify Database Server Virtual Drive Configuration Collecting - Verify Disk Cache Policy on database server Collecting - Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Database Server] Collecting - Verify ILOM Power Up Configuration for HOST_AUTO_POWER_ON Collecting - Verify ILOM Power Up Configuration for HOST_LAST_POWER_STATE Collecting - Verify IP routing configuration on database servers Collecting - Verify InfiniBand Address Resolution Protocol (ARP) Configuration on Database Servers Collecting - Verify InfiniBand subnet manager is not running on database server Collecting - Verify Master (Rack) Serial Number is Set [Database Server] Collecting - Verify NTP configuration on database servers Collecting - Verify Quorum disks configuration Collecting - Verify RAID Controller Battery Temperature [Database Server] Collecting - Verify RAID disk controller CacheVault capacitor condition [Database Server] Collecting - Verify TCP Segmentation Offload (TSO) is set to off Collecting - Verify active kernel version matches expected version for installed Exadata Image Collecting - Verify basic Logical Volume(LVM) system devices configuration Collecting - Verify database server disk controllers use writeback cache Collecting - Verify database server file systems have Check interval = 0 Collecting - Verify database server file systems have Maximum mount count = -1 Collecting - Verify imageinfo on database server Collecting - Verify imageinfo on database server to compare systemwide Collecting - Verify no database server kernel out of memory errors Collecting - Verify service exachkcfg autostart status on database server Collecting - Verify that the SDP over IB option sdp_apm_enable is set to 0 Collecting - Verify the localhost alias is pingable [Database Server] Collecting - Verify the Name Service Cache Daemon (NSCD) is Running Collecting - Verify the file /.updfrm_exact does not exist [Database Server] Collecting - Verify the grid Infrastructure management database (MGMTDB) does not use hugepages Collecting - Verify the vm.min_free_kbytes configuration Collecting - root time zone check Collecting - verify asr exadata configuration check via ASREXACHECK on database server Data collections completed. Checking best practices on v1ex2dbadm02. -------------------------------------------------------------------------------------- WARNING => Hidden ASM Initialization Parameter usage is not correct WARNING => ASM parameter AUDIT_SYSLOG_LEVEL should be set to the recommended value WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on VERSS2 instance WARNING => Database parameter AUDIT_TRAIL should be set to the recommended value on IONS2 instance FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSDEV2 instance FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on IONDEV2 instance FAIL => Database parameter _smm_auto_max_io_size should be set to the recommended value on VERSREP2 instance WARNING => ASM parameter ASM_POWER_LIMIT is not set to the default value. INFO => Oracle GoldenGate failure prevention best practices WARNING => Local archive destination does not alternate destination configured for VERSDEV WARNING => Local archive destination does not alternate destination configured for IONDEV WARNING => Local archive destination does not alternate destination configured for VERSREP WARNING => System has fewer than five storage servers but does not have quorum disks configured on database servers. INFO => Validate database security configuration using database security assessment tool for VERSS INFO => Validate database security configuration using database security assessment tool for VERSDEV INFO => Validate database security configuration using database security assessment tool for IONS INFO => Validate database security configuration using database security assessment tool for IONDEV INFO => Validate database security configuration using database security assessment tool for VERSREP FAIL => Oracle monitoring agent and Operating systems settings on Automatic diagnostic repository directories are not correct or not all targets have been scanned or not all diagnostic directories found FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSDEV2 instance FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on IONDEV2 instance FAIL => Database parameter _parallel_adaptive_max_users is not set to recommended value on VERSREP2 instance FAIL => Downdelay attribute is not set to recommended value on bonded client interface FAIL => Database control files are not configured as recommended for VERSS FAIL => Database control files are not configured as recommended for VERSDEV FAIL => Database control files are not configured as recommended for IONS FAIL => Database control files are not configured as recommended for IONDEV FAIL => Database control files are not configured as recommended for VERSREP FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSDEV FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for IONDEV FAIL => One or more of SYSTEM, SYSAUX, USERS, TEMP tablespaces are not of type bigfile for VERSREP FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSS2 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSDEV2 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONS2 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on IONDEV2 instance FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value on VERSREP2 instance WARNING => filesystemio_options is not set to recommended value on VERSS2 instance WARNING => filesystemio_options is not set to recommended value on VERSDEV2 instance WARNING => filesystemio_options is not set to recommended value on IONS2 instance WARNING => filesystemio_options is not set to recommended value on IONDEV2 instance WARNING => filesystemio_options is not set to recommended value on VERSREP2 instance FAIL => ASM Audit file destination file count > 100,000 FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSS2 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSDEV2 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONS2 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on IONDEV2 instance FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value on VERSREP2 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSS2 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSDEV2 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONS2 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on IONDEV2 instance FAIL => Database parameter GLOBAL_NAMES is not set to recommended value on VERSREP2 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSS2 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSDEV2 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONS2 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on IONDEV2 instance FAIL => Database parameter PARALLEL_ADAPTIVE_MULTI_USER is not set to recommended value on VERSREP2 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSS2 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSDEV2 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on IONS2 instance FAIL => Database parameter PARALLEL_THREADS_PER_CPU is not set to recommended value on VERSREP2 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSS2 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSDEV2 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on IONS2 instance FAIL => Database parameter OS_AUTHENT_PREFIX is not set to recommended value on VERSREP2 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSS2 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSDEV2 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONS2 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on IONDEV2 instance FAIL => Database parameter USE_LARGE_PAGES is not set to recommended value on VERSREP2 instance FAIL => Database parameter sql92_security is not set to recommended value on VERSS2 instance FAIL => Database parameter sql92_security is not set to recommended value on VERSDEV2 instance FAIL => Database parameter sql92_security is not set to recommended value on IONS2 instance FAIL => Database parameter sql92_security is not set to recommended value on IONDEV2 instance FAIL => Database parameter sql92_security is not set to recommended value on VERSREP2 instance FAIL => ASM parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSDEV FAIL => Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value for VERSREP FAIL => ASM processes parameter is not set to recommended value WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSDEV WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for IONDEV WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for VERSREP FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for VERSS FAIL => Database parameter DB_BLOCK_CHECKING on STANDBY is NOT set to the recommended value. for IONS WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSS2 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSDEV2 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONS2 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on IONDEV2 instance WARNING => fast_start_mttr_target should be greater than or equal to 300. on VERSREP2 instance FAIL => ASM parameter MEMORY_TARGET should be set according to recommended value FAIL => ASM parameter PGA_AGGREGATE_TARGET should be set according to recommended value FAIL => ASM parameter MEMORY_MAX_TARGET should be set according to recommended value FAIL => Database control files are not configured as recommended for VERSS FAIL => Database control files are not configured as recommended for VERSDEV FAIL => Database control files are not configured as recommended for IONS FAIL => Database control files are not configured as recommended for IONDEV FAIL => Database control files are not configured as recommended for VERSREP INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for VERSS INFO => While initialization parameter LOG_ARCHIVE_CONFIG is set it should be verified for your environment on Standby Database for IONS FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSDEV2 instance FAIL => Database parameter LOG_BUFFER is not set to recommended value on IONDEV2 instance FAIL => Database parameter LOG_BUFFER is not set to recommended value on VERSREP2 instance WARNING => Redo log files should be appropriately sized for IONDEV WARNING => Redo log files should be appropriately sized for VERSREP FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSS2 instance FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSDEV2 instance FAIL => Database parameter DB_FILES should be set to a value greater than or equal to the recommended minimum. on VERSREP2 instance FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for VERSS FAIL => Initialization parameter LOG_ARCHIVE_MAX_PROCESSES should be configured as recommended for IONS FAIL => ASM parameter SGA_TARGET is not set according to recommended value. WARNING => Multiple Oracle database instances discovered, observe database consolidation best practices Collecting patch inventory on CRS HOME /u01/app/12.1.0.2/grid Collecting patch inventory on ASM HOME /u01/app/12.1.0.2/grid Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_1 Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/12.1.0.2/dbhome_2 --------------------------------------------------------------------------------- CLUSTERWIDE CHECKS --------------------------------------------------------------------------------- --------------------------------------------------------------------------------- Detailed report (html) - /home/oracle/version1/exachk/exachk/exachk_v1ex2dbadm01_VERSREP_090517_105859/exachk_v1ex2dbadm01_VERSREP_090517_105859.html UPLOAD(if required) - /home/oracle/version1/exachk/exachk/exachk_v1ex2dbadm01_VERSREP_090517_105859.zip [root@v1ex2dbadm01 exachk]#
Once completed, you’ll see a new folder and zip file, I recommend moving them both up a level to safe keeping:
[root@v1ex2dbadm01 exachk]# ls -ltrh total 75M -r-xr-xr-x 1 root root 40K Jul 22 2015 sample_user_defined_checks.xml -r-xr-xr-x 1 root root 2.9K Oct 9 2015 user_defined_checks.xsd -r-xr-xr-x 1 root root 4.6M Sep 15 2016 CollectionManager_App.sql -r--r--r-- 1 root root 4.8K Mar 31 09:58 readme.txt -r--r--r-- 1 root root 234 Apr 1 10:05 UserGuide.txt -r-xr-xr-x 1 root root 7.9M Apr 4 12:12 Apex5_CollectionManager_App.sql -r-xr-xr-x 1 root root 2.8M May 17 20:59 exachk drwxr-xr-x 2 root root 4.0K Jul 19 03:17 templates drwxr-xr-x 2 root root 4.0K Jul 19 03:17 exadiscover -rw-r--r-- 1 root root 45M Jul 19 08:38 collections.dat -rw-r--r-- 1 root root 6.1M Jul 19 08:38 rules.dat -rw-r--r-- 1 root root 1.9M Jul 19 09:03 EXAchk_Health_Check_Catalog.html drwxr-xr-x 2 root root 4.0K Sep 5 10:54 doc drwxr-xr-x 8 root root 180K Sep 5 11:18 exachk_v1ex2dbadm01_VERSREP_090517_105859 -rw-r--r-- 1 root root 5.9M Sep 5 11:19 exachk_v1ex2dbadm01_VERSREP_090517_105859.zip [root@v1ex2dbadm01 exachk]# mv exachk_* .. [root@v1ex2dbadm01 exachk]# cd .. [root@v1ex2dbadm01 exachk]# ls -ltrh total 25M -rw-r--r-- 1 root root 19M Sep 5 10:51 exachk.zip drwxr-xr-x 8 root root 180K Sep 5 11:18 exachk_v1ex2dbadm01_VERSREP_090517_105859 -rw-r--r-- 1 root root 5.9M Sep 5 11:19 exachk_v1ex2dbadm01_VERSREP_090517_105859.zip drwxr-xr-x 6 root root 4.0K Sep 5 11:29 exachk [root@v1ex2dbadm01 exachk]#
In the new folder you’ll see a html file, which is the report produced by exachk:
[root@v1ex2dbadm01 exachk]# cd exachk_v1ex2dbadm01_VERSREP_090517_105859 [root@v1ex2dbadm01 exachk_v1ex2dbadm01_VERSREP_090517_105859]# ls -ltrh total 6.4M -rw-r--r-- 1 root root 762 Sep 5 11:02 targets.xsl drwxr-xr-x 2 root root 4.0K Sep 5 11:18 upload drwxr-xr-x 4 root root 100K Sep 5 11:18 outfiles drwxr-xr-x 2 root root 4.0K Sep 5 11:18 reports -rw-r--r-- 1 root root 6.2M Sep 5 11:18 exachk_v1ex2dbadm01_VERSREP_090517_105859.html drwxr-xr-x 2 root root 48K Sep 5 11:19 scripts drwxr-xr-x 2 root root 4.0K Sep 5 11:19 log [root@v1ex2dbadm01 exachk_v1ex2dbadm01_VERSREP_090517_105859]#
It’s normally easier to copy the zip file off the compute node to your machine and then unzip and view the report local.
It will look like this:
Known Issues
1. Running ExaChk reset snmpd.conf and iptables on Storage Cells. There’s a bug currently open with Oracle Support for this:
Bug 26742216 – EXACHK IS RESETTING IPTABLES AND IT IS BLOCKING SNMP CONNECTIONS UNTIL RESTART
Which hasn’t yet been fixed. See my blog post with discuss how to fix this after Exadata Patching where this also happens:
SNMP unresponsive on Storage Cell after Exadata Patching
2. ExaChk experiences Skipped Checks and Timeouts. Se my blog post which show how to workaround this issue:
Resolving Slow Performance, Skipped Checks and Timeouts on Exa Check (exachk)
If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.
Thanks
Zed DBA (Zahid Anwar)
Pingback: Resolving Slow Performance, Skipped Checks and Timeouts on Exa Check (exachk) | Zed DBA's Oracle Blog
Thanks your detail log help lot , highly appreciate your work
LikeLike
You’re welcome. I’m glad it’s helped.
LikeLike