There maybe times when you are required to fully shutdown an Oracle Exadata Machine, for example for maintenance.
For instructions on how to shutdown an Oracle Exadata Machine, please refer to my blog post:
How to Shutdown an Oracle Exadata Machine
Once shutdown, you will need to be able to re-start which this blog post will detail.
Below is the My Oracle Support note used to carry out the startup:
Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration (Doc ID 1093890.1)
1. Pre-requisites
Ensure you have all the compute nodes and storage cells ILOM addresses and correct passwords. Otherwise you will not be able to remotely power back on and will require a physical power on using the power button on the front panels.
2. Power on first Compute Node
You can power on the first compute node via the ilom via ssh or WebILOM. I prefer the ssh method shown below:
[AnwarZ@v1proxy1 ~]$ ssh root@v1ex1dbadm01-ilom Password: Oracle(R) Integrated Lights Out Manager Version 4.0.4.37 r130617 Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. Warning: HTTPS certificate is set to factory default. Hostname: v1ex1dbadm01-ilom -> show /SYSTEM /System Targets: Open_Problems (0) Processors Memory Power Cooling Storage Networking PCI_Devices Firmware BIOS Log Properties: health = OK health_details = - open_problems_count = 0 type = Rack Mount model = Exadata X5-2 qpart_id = XXXXXX part_number = Exadata X5-2 serial_number = XXXXXXXXXX component_model = ORACLE SERVER X5-2 component_part_number = XXXXXXX component_serial_number = XXXXXXXXXX system_identifier = Exadata Database Machine X5-2 XXXXXXXXXX system_fw_version = 4.0.4.37 primary_operating_system = Not Available primary_operating_system_detail = Comprehensive System monitoring is not available. Ensure the host is running with the Hardware Management Pack. For details go to http://www.oracle.com/goto/ilom-redirect/hmp-osa host_primary_mac_address = xx:xx:xx:xx:xx:xx ilom_address = x.x.x.x ilom_mac_address = xx:xx:xx:xx:xx:xx locator_indicator = Off power_state = Off actual_power_consumption = 22 watts action = (Cannot show property) Commands: cd reset set show start stop -> start /SYSTEM Are you sure you want to start /System (y/n)? y Starting /System -> show /SYSTEM /System Targets: Open_Problems (0) Processors Memory Power Cooling Storage Networking PCI_Devices Firmware BIOS Log Properties: health = OK health_details = - open_problems_count = 0 type = Rack Mount model = Exadata X5-2 qpart_id = XXXXXX part_number = Exadata X5-2 serial_number = XXXXXXXXXX component_model = ORACLE SERVER X5-2 component_part_number = XXXXXXX component_serial_number = XXXXXXXXXX system_identifier = Exadata Database Machine X5-2 XXXXXXXXXX system_fw_version = 4.0.4.37 primary_operating_system = Not Available primary_operating_system_detail = Comprehensive System monitoring is not available. Ensure the host is running with the Hardware Management Pack. For details go to http://www.oracle.com/goto/ilom-redirect/hmp-osa host_primary_mac_address = xx:xx:xx:xx:xx:xx ilom_address = x.x.x.x ilom_mac_address = xx:xx:xx:xx:xx:xx locator_indicator = Off power_state = On actual_power_consumption = 220 watts action = (Cannot show property) Commands: cd reset set show start stop -> exit Connection to v1ex1dbadm01-ilom closed. [AnwarZ@v1proxy1 ~]$
3. Power on all Storage Cells
Login to the first compute node and power on all the storage cells as shown below:
login as: root root@v1ex1dbadm01's password: Last login: Wed Jun 10 09:21:41 IST 2020 from v1ex1dbadm01.v1.com on ssh Last login: Wed Jun 10 17:31:31 2020 from x.x.x.x [root@v1ex1dbadm01 ~]# uptime 17:31:37 up 1 min, 1 user, load average: 3.37, 1.22, 0.44 [root@v1ex1dbadm01 ~]# export HISTIGNORE='*' [root@v1ex1dbadm01 ~]# for host in `cat /opt/oracle.SupportTools/onecommand/cell_group`; do > echo ${host}: `ipmitool -I lanplus -H ${host}-ilom -U root -P XXXXXXXX chassis power on` > done v1ex1celadm01: Chassis Power Control: Up/On v1ex1celadm02: Chassis Power Control: Up/On v1ex1celadm03: Chassis Power Control: Up/On [root@v1ex1dbadm01 ~]# export HISTIGNORE='' [root@v1ex1dbadm01 ~]#
Please Note: the HISTIGNORE is used, so the password isn’t kept in history.
Check the storage cell services are up:
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root 'hostname; uptime' v1ex1celadm01: v1ex1celadm01.v1.com v1ex1celadm01: 18:10:21 up 32 min, 0 users, load average: 1.86, 1.95, 2.03 v1ex1celadm02: v1ex1celadm02.v1.com v1ex1celadm02: 18:10:21 up 32 min, 0 users, load average: 1.47, 1.82, 1.97 v1ex1celadm03: v1ex1celadm03.v1.com v1ex1celadm03: 18:10:22 up 32 min, 0 users, load average: 1.51, 1.85, 2.01 [root@v1ex1dbadm01 ~]#
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e 'list cell detail'" v1ex1celadm01: name: v1ex1celadm01 v1ex1celadm01: accessLevelPerm: remoteLoginEnabled v1ex1celadm01: bbuStatus: normal v1ex1celadm01: cellVersion: OSS_19.2.7.0.0_LINUX.X64_191012 v1ex1celadm01: cpuCount: 32/32 v1ex1celadm01: diagHistoryDays: 7 v1ex1celadm01: eighthRack: FALSE v1ex1celadm01: fanCount: 8/8 v1ex1celadm01: fanStatus: normal v1ex1celadm01: flashCacheMode: WriteBack v1ex1celadm01: httpsAccess: ALL v1ex1celadm01: id: XXXXXXXXXX v1ex1celadm01: interconnectCount: 2 v1ex1celadm01: interconnect1: ib0 v1ex1celadm01: interconnect2: ib1 v1ex1celadm01: iormBoost: 0.0 v1ex1celadm01: ipaddress1: x.x.x.x/22 v1ex1celadm01: ipaddress2: x.x.x.x/22 v1ex1celadm01: kernelVersion: 4.1.12-124.30.1.el7uek.x86_64 v1ex1celadm01: locatorLEDStatus: off v1ex1celadm01: makeModel: Oracle Corporation ORACLE SERVER X5-2L High Capacity v1ex1celadm01: memoryGB: 94 v1ex1celadm01: metricHistoryDays: 7 v1ex1celadm01: notificationMethod: snmp v1ex1celadm01: notificationPolicy: critical,warning,clear v1ex1celadm01: offloadGroupEvents: v1ex1celadm01: powerCount: 2/2 v1ex1celadm01: powerStatus: normal v1ex1celadm01: ramCacheMaxSize: 0 v1ex1celadm01: ramCacheMode: Auto v1ex1celadm01: ramCacheSize: 0 v1ex1celadm01: releaseImageStatus: success v1ex1celadm01: releaseVersion: 19.2.7.0.0.191012 v1ex1celadm01: rpmVersion: cell-19.2.7.0.0_LINUX.X64_191012-1.x86_64 v1ex1celadm01: releaseTrackingBug: 30393131 v1ex1celadm01: rollbackVersion: 18.1.18.0.0.190709 v1ex1celadm01: snmpSubscriber: host=x.x.x.x,port=162,community=public,type=ASR,asrmPort=16161 v1ex1celadm01: host=x.x.x.x,port=161,community=V1 v1ex1celadm01: host=x.x.x.x,port=161,community=V1 v1ex1celadm01: status: online v1ex1celadm01: temperatureReading: 22.0 v1ex1celadm01: temperatureStatus: normal v1ex1celadm01: upTime: 0 days, 0:33 v1ex1celadm01: usbStatus: normal v1ex1celadm01: cellsrvStatus: running v1ex1celadm01: msStatus: running v1ex1celadm01: rsStatus: running v1ex1celadm02: name: v1ex1celadm02 ... v1ex1celadm02: status: online v1ex1celadm02: temperatureReading: 22.0 v1ex1celadm02: temperatureStatus: normal v1ex1celadm02: upTime: 0 days, 0:33 v1ex1celadm02: usbStatus: normal v1ex1celadm02: cellsrvStatus: running v1ex1celadm02: msStatus: running v1ex1celadm02: rsStatus: running v1ex1celadm03: name: v1ex1celadm03 ... v1ex1celadm03: status: online v1ex1celadm03: temperatureReading: 22.0 v1ex1celadm03: temperatureStatus: normal v1ex1celadm03: upTime: 0 days, 0:33 v1ex1celadm03: usbStatus: normal v1ex1celadm03: cellsrvStatus: running v1ex1celadm03: msStatus: running v1ex1celadm03: rsStatus: running [root@v1ex1dbadm01 ~]#
4. Power on remaining Compute Nodes
Power on remaining compute nodes via ipmitool:
[root@v1ex1dbadm01 ~]# export HISTIGNORE='*' [root@v1ex1dbadm01 ~]# ipmitool -I lanplus -H v1ex1dbadm02-ilom -U root -P XXXXXXXX chassis power on Chassis Power Control: Up/On [root@v1ex1dbadm01 ~]# export HISTIGNORE='' [root@v1ex1dbadm01 ~]#
If half or full rack, then the following can be used:
for host in `cat dbs_group_all_but_first`; do echo ${host}: `ipmitool -H ${host}-ilom -U root -P XXXXXXXX chassis power on` done
Check compute nodes are up:
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/dbs_group -l root 'hostname; uptime' v1ex1dbadm01: v1ex1dbadm01.v1.com v1ex1dbadm01: 18:21:55 up 12 min, 1 user, load average: 0.22, 0.89, 1.54 v1ex1dbadm02: v1ex1dbadm02.v1.com v1ex1dbadm02: 18:21:55 up 3 min, 0 users, load average: 2.44, 1.58, 0.66 [root@v1ex1dbadm01 ~]#
5. Re-enable clusterware autostart
Re-enable clusterware autostart via dcli:
[root@v1ex1dbadm01 ~]# . oraenv ORACLE_SID = [root] ? +ASM1 The Oracle base has been set to /u01/app/oracle [root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/dbs_group -l root /u01/app/12.1.0.2/grid/bin/crsctl enable crs v1ex1dbadm01: CRS-4622: Oracle High Availability Services autostart is enabled. v1ex1dbadm02: CRS-4622: Oracle High Availability Services autostart is enabled. [root@v1ex1dbadm01 ~]#
6. Restart Grid Infrastructure on the cluster
Start clusterware on first compute node:
[root@v1ex1dbadm01 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@v1ex1dbadm01 ~]#
Now logon the remaining compute nodes and restart clusterware:
login as: root root@v1ex1dbadm02's password: Last login: Wed Jun 10 18:26:56 IST 2020 from x.x.x.x on ssh Last login: Wed Jun 10 18:30:56 2020 from x.x.x.x [root@v1ex1dbadm02 ~]# . oraenv ORACLE_SID = [root] ? +ASM2 The Oracle base has been set to /u01/app/oracle [root@v1ex1dbadm02 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@v1ex1dbadm02 ~]#
Wait a few minutes and check clusterware is all up as shown below:
[root@v1ex1dbadm02 ~]# crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATAC1.dg ONLINE ONLINE v1ex1dbadm01 STABLE ONLINE ONLINE v1ex1dbadm02 STABLE ... ONLINE ONLINE v1ex1dbadm01 STABLE ONLINE ONLINE v1ex1dbadm02 STABLE ora.RECOC1.dg ONLINE ONLINE v1ex1dbadm01 STABLE ONLINE ONLINE v1ex1dbadm02 STABLE ora.asm ONLINE ONLINE v1ex1dbadm01 Started,STABLE ONLINE ONLINE v1ex1dbadm02 Started,STABLE ... ora.net1.network ONLINE ONLINE v1ex1dbadm01 STABLE ONLINE ONLINE v1ex1dbadm02 STABLE ora.ons ONLINE ONLINE v1ex1dbadm01 STABLE ONLINE ONLINE v1ex1dbadm02 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE v1ex1dbadm02 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE v1ex1dbadm01 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE v1ex1dbadm01 STABLE ... 1 ONLINE ONLINE v1ex1dbadm02 Open,STABLE ora.oc4j 1 ONLINE ONLINE v1ex1dbadm01 STABLE ora.scan1.vip 1 ONLINE ONLINE v1ex1dbadm02 STABLE ora.scan2.vip 1 ONLINE ONLINE v1ex1dbadm01 STABLE ora.scan3.vip 1 ONLINE ONLINE v1ex1dbadm01 STABLE -------------------------------------------------------------------------------- [root@v1ex1dbadm02 ~]#
6. Restart OEM Agent
Optionally if you have an OEM agent (most likely), restart as follows:
[oracle@v1ex1dbadm01 ~]$ cd /u01/app/agent/agent_13.3.0.0.0/bin [oracle@v1ex1dbadm01 bin]$ ./emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent is Not Running [oracle@v1ex1dbadm01 bin]$ ./emctl start agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Starting agent ............................ started. [oracle@v1ex1dbadm01 bin]$ ./emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 13.3.0.0.0 OMS Version : 13.3.0.0.0 Protocol Version : 12.1.0.1.0 Agent Home : /u01/app/agent/agent_inst Agent Log Directory : /u01/app/agent/agent_inst/sysman/log Agent Binaries : /u01/app/agent/agent_13.3.0.0.0 Core JAR Location : /u01/app/agent/agent_13.3.0.0.0/jlib Agent Process ID : 122257 Parent Process ID : 122120 Agent URL : https://v1ex1dbadm01.v1.com:3872/emd/main/ Local Agent URL in NAT : https://v1ex1dbadm01.v1.com:3872/emd/main/ Repository URL : https://v1oem.v1.com:4903/empbs/upload Started at : 2020-06-17 15:40:59 Started by user : oracle Operating System : Linux version 4.1.12-124.30.1.el7uek.x86_64 (amd64) Number of Targets : 43 Last Reload : (none) Last successful upload : 2020-06-17 15:41:23 Last attempted upload : 2020-06-17 15:41:23 Total Megabytes of XML files uploaded so far : 0.1 Number of XML files pending upload : 5 Size of XML files pending upload(MB) : 0.02 Available disk space on upload filesystem : 21.34% Collection Status : Collections enabled Heartbeat Status : Ok Last attempted heartbeat to OMS : 2020-06-17 15:41:18 Last successful heartbeat to OMS : 2020-06-17 15:41:18 Next scheduled heartbeat to OMS : 2020-06-17 15:42:20 --------------------------------------------------------------- Agent is Running and Ready [oracle@v1ex1dbadm01 bin]$
Now on any other compute nodes:
[oracle@v1ex1dbadm01 bin]$ ssh v1ex1dbadm02 Last login: Wed Jun 17 12:37:20 IST 2020 from x.x.x.x on ssh Last login: Wed Jun 17 15:43:07 2020 from x.x.x.x [oracle@v1ex1dbadm02 ~]$ cd /u01/app/agent/agent_13.3.0.0.0/bin [oracle@v1ex1dbadm02 bin]$ ./emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent is Not Running [oracle@v1ex1dbadm02 bin]$ ./emctl start agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Starting agent .................................... started. [oracle@v1ex1dbadm02 bin]$ ./emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 13.3.0.0.0 OMS Version : 13.3.0.0.0 Protocol Version : 12.1.0.1.0 Agent Home : /u01/app/agent/agent_inst Agent Log Directory : /u01/app/agent/agent_inst/sysman/log Agent Binaries : /u01/app/agent/agent_13.3.0.0.0 Core JAR Location : /u01/app/agent/agent_13.3.0.0.0/jlib Agent Process ID : 189737 Parent Process ID : 189513 Agent URL : https://v1ex1dbadm02.v1.com:3872/emd/main/ Local Agent URL in NAT : https://v1ex1dbadm02.v1.com:3872/emd/main/ Repository URL : https://v1oem.v1.com:4903/empbs/upload Started at : 2020-06-17 15:44:01 Started by user : oracle Operating System : Linux version 4.1.12-124.30.1.el7uek.x86_64 (amd64) Number of Targets : 37 Last Reload : (none) Last successful upload : 2020-06-17 15:44:44 Last attempted upload : 2020-06-17 15:44:45 Total Megabytes of XML files uploaded so far : 0.17 Number of XML files pending upload : 1 Size of XML files pending upload(MB) : 0 Available disk space on upload filesystem : 28.51% Collection Status : Collections enabled Heartbeat Status : Ok Last attempted heartbeat to OMS : 2020-06-17 15:44:28 Last successful heartbeat to OMS : 2020-06-17 15:44:28 Next scheduled heartbeat to OMS : 2020-06-17 15:45:28 --------------------------------------------------------------- Agent is Running and Ready [oracle@v1ex1dbadm02 bin]$
If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.
Thanks
Zed DBA (Zahid Anwar)
Zed DBA's Oracle Blog 