How to Startup an Oracle Exadata Machine

There maybe times when you are required to fully shutdown an Oracle Exadata Machine, for example for maintenance.

For instructions on how to shutdown an Oracle Exadata Machine, please refer to my blog post:
How to Shutdown an Oracle Exadata Machine

Once shutdown, you will need to be able to re-start which this blog post will detail.

Below is the My Oracle Support note used to carry out the startup:
Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration (Doc ID 1093890.1)

1. Pre-requisites

Ensure you have all the compute nodes and storage cells ILOM addresses and correct passwords.  Otherwise you will not be able to remotely power back on and will require a physical power on using the power button on the front panels.

2. Power on first Compute Node

You can power on the first compute node via the ilom via ssh or WebILOM.  I prefer the ssh method shown below:

[AnwarZ@v1proxy1 ~]$ ssh root@v1ex1dbadm01-ilom
Password:

Oracle(R) Integrated Lights Out Manager

Version 4.0.4.37 r130617

Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: v1ex1dbadm01-ilom

-> show /SYSTEM

/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
Log

Properties:
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = Exadata X5-2
qpart_id = XXXXXX
part_number = Exadata X5-2
serial_number = XXXXXXXXXX
component_model = ORACLE SERVER X5-2
component_part_number = XXXXXXX
component_serial_number = XXXXXXXXXX
system_identifier = Exadata Database Machine X5-2 XXXXXXXXXX
system_fw_version = 4.0.4.37
primary_operating_system = Not Available
primary_operating_system_detail = Comprehensive System monitoring is not available. Ensure the host is
running with the Hardware Management Pack. For details go to
http://www.oracle.com/goto/ilom-redirect/hmp-osa
host_primary_mac_address = xx:xx:xx:xx:xx:xx
ilom_address = x.x.x.x
ilom_mac_address = xx:xx:xx:xx:xx:xx
locator_indicator = Off
power_state = Off
actual_power_consumption = 22 watts
action = (Cannot show property)

Commands:
cd
reset
set
show
start
stop

-> start /SYSTEM
Are you sure you want to start /System (y/n)? y
Starting /System

-> show /SYSTEM

/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
Log

Properties:
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = Exadata X5-2
qpart_id = XXXXXX
part_number = Exadata X5-2
serial_number = XXXXXXXXXX
component_model = ORACLE SERVER X5-2
component_part_number = XXXXXXX
component_serial_number = XXXXXXXXXX
system_identifier = Exadata Database Machine X5-2 XXXXXXXXXX
system_fw_version = 4.0.4.37
primary_operating_system = Not Available
primary_operating_system_detail = Comprehensive System monitoring is not available. Ensure the host is
running with the Hardware Management Pack. For details go to
http://www.oracle.com/goto/ilom-redirect/hmp-osa
host_primary_mac_address = xx:xx:xx:xx:xx:xx
ilom_address = x.x.x.x
ilom_mac_address = xx:xx:xx:xx:xx:xx
locator_indicator = Off
power_state = On
actual_power_consumption = 220 watts
action = (Cannot show property)

Commands:
cd
reset
set
show
start
stop

-> exit
Connection to v1ex1dbadm01-ilom closed.
[AnwarZ@v1proxy1 ~]$

3. Power on all Storage Cells

Login to the first compute node and power on all the storage cells as shown below:

login as: root
root@v1ex1dbadm01's password:
Last login: Wed Jun 10 09:21:41 IST 2020 from v1ex1dbadm01.v1.com on ssh
Last login: Wed Jun 10 17:31:31 2020 from x.x.x.x
[root@v1ex1dbadm01 ~]# uptime
17:31:37 up 1 min, 1 user, load average: 3.37, 1.22, 0.44
[root@v1ex1dbadm01 ~]# export HISTIGNORE='*'
[root@v1ex1dbadm01 ~]# for host in `cat /opt/oracle.SupportTools/onecommand/cell_group`; do
> echo ${host}: `ipmitool -I lanplus -H ${host}-ilom -U root -P XXXXXXXX chassis power on`
> done
v1ex1celadm01: Chassis Power Control: Up/On
v1ex1celadm02: Chassis Power Control: Up/On
v1ex1celadm03: Chassis Power Control: Up/On
[root@v1ex1dbadm01 ~]# export HISTIGNORE=''
[root@v1ex1dbadm01 ~]#

Please Note: the HISTIGNORE is used, so the password isn’t kept in history.

Check the storage cell services are up:

[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root 'hostname; uptime'
v1ex1celadm01: v1ex1celadm01.v1.com
v1ex1celadm01: 18:10:21 up 32 min, 0 users, load average: 1.86, 1.95, 2.03
v1ex1celadm02: v1ex1celadm02.v1.com
v1ex1celadm02: 18:10:21 up 32 min, 0 users, load average: 1.47, 1.82, 1.97
v1ex1celadm03: v1ex1celadm03.v1.com
v1ex1celadm03: 18:10:22 up 32 min, 0 users, load average: 1.51, 1.85, 2.01
[root@v1ex1dbadm01 ~]#
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e 'list cell detail'"
v1ex1celadm01: name: v1ex1celadm01
v1ex1celadm01: accessLevelPerm: remoteLoginEnabled
v1ex1celadm01: bbuStatus: normal
v1ex1celadm01: cellVersion: OSS_19.2.7.0.0_LINUX.X64_191012
v1ex1celadm01: cpuCount: 32/32
v1ex1celadm01: diagHistoryDays: 7
v1ex1celadm01: eighthRack: FALSE
v1ex1celadm01: fanCount: 8/8
v1ex1celadm01: fanStatus: normal
v1ex1celadm01: flashCacheMode: WriteBack
v1ex1celadm01: httpsAccess: ALL
v1ex1celadm01: id: XXXXXXXXXX
v1ex1celadm01: interconnectCount: 2
v1ex1celadm01: interconnect1: ib0
v1ex1celadm01: interconnect2: ib1
v1ex1celadm01: iormBoost: 0.0
v1ex1celadm01: ipaddress1: x.x.x.x/22
v1ex1celadm01: ipaddress2: x.x.x.x/22
v1ex1celadm01: kernelVersion: 4.1.12-124.30.1.el7uek.x86_64
v1ex1celadm01: locatorLEDStatus: off
v1ex1celadm01: makeModel: Oracle Corporation ORACLE SERVER X5-2L High Capacity
v1ex1celadm01: memoryGB: 94
v1ex1celadm01: metricHistoryDays: 7
v1ex1celadm01: notificationMethod: snmp
v1ex1celadm01: notificationPolicy: critical,warning,clear
v1ex1celadm01: offloadGroupEvents:
v1ex1celadm01: powerCount: 2/2
v1ex1celadm01: powerStatus: normal
v1ex1celadm01: ramCacheMaxSize: 0
v1ex1celadm01: ramCacheMode: Auto
v1ex1celadm01: ramCacheSize: 0
v1ex1celadm01: releaseImageStatus: success
v1ex1celadm01: releaseVersion: 19.2.7.0.0.191012
v1ex1celadm01: rpmVersion: cell-19.2.7.0.0_LINUX.X64_191012-1.x86_64
v1ex1celadm01: releaseTrackingBug: 30393131
v1ex1celadm01: rollbackVersion: 18.1.18.0.0.190709
v1ex1celadm01: snmpSubscriber: host=x.x.x.x,port=162,community=public,type=ASR,asrmPort=16161
v1ex1celadm01: host=x.x.x.x,port=161,community=V1
v1ex1celadm01: host=x.x.x.x,port=161,community=V1
v1ex1celadm01: status: online
v1ex1celadm01: temperatureReading: 22.0
v1ex1celadm01: temperatureStatus: normal
v1ex1celadm01: upTime: 0 days, 0:33
v1ex1celadm01: usbStatus: normal
v1ex1celadm01: cellsrvStatus: running
v1ex1celadm01: msStatus: running
v1ex1celadm01: rsStatus: running
v1ex1celadm02: name: v1ex1celadm02
...
v1ex1celadm02: status: online
v1ex1celadm02: temperatureReading: 22.0
v1ex1celadm02: temperatureStatus: normal
v1ex1celadm02: upTime: 0 days, 0:33
v1ex1celadm02: usbStatus: normal
v1ex1celadm02: cellsrvStatus: running
v1ex1celadm02: msStatus: running
v1ex1celadm02: rsStatus: running
v1ex1celadm03: name: v1ex1celadm03
...
v1ex1celadm03: status: online
v1ex1celadm03: temperatureReading: 22.0
v1ex1celadm03: temperatureStatus: normal
v1ex1celadm03: upTime: 0 days, 0:33
v1ex1celadm03: usbStatus: normal
v1ex1celadm03: cellsrvStatus: running
v1ex1celadm03: msStatus: running
v1ex1celadm03: rsStatus: running
[root@v1ex1dbadm01 ~]#

4. Power on remaining Compute Nodes

Power on remaining compute nodes via ipmitool:

[root@v1ex1dbadm01 ~]# export HISTIGNORE='*'
[root@v1ex1dbadm01 ~]# ipmitool -I lanplus -H v1ex1dbadm02-ilom -U root -P XXXXXXXX chassis power on
Chassis Power Control: Up/On
[root@v1ex1dbadm01 ~]# export HISTIGNORE=''
[root@v1ex1dbadm01 ~]#

If half or full rack, then the following can be used:

for host in `cat dbs_group_all_but_first`; do
echo ${host}: `ipmitool -H ${host}-ilom -U root -P XXXXXXXX chassis power on`
done

Check compute nodes are up:

[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/dbs_group -l root 'hostname; uptime'
v1ex1dbadm01: v1ex1dbadm01.v1.com
v1ex1dbadm01: 18:21:55 up 12 min, 1 user, load average: 0.22, 0.89, 1.54
v1ex1dbadm02: v1ex1dbadm02.v1.com
v1ex1dbadm02: 18:21:55 up 3 min, 0 users, load average: 2.44, 1.58, 0.66
[root@v1ex1dbadm01 ~]#

5. Re-enable clusterware autostart

Re-enable clusterware autostart via dcli:

[root@v1ex1dbadm01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/dbs_group -l root /u01/app/12.1.0.2/grid/bin/crsctl enable crs
v1ex1dbadm01: CRS-4622: Oracle High Availability Services autostart is enabled.
v1ex1dbadm02: CRS-4622: Oracle High Availability Services autostart is enabled.
[root@v1ex1dbadm01 ~]#

6. Restart Grid Infrastructure on the cluster

Start clusterware on first compute node:

[root@v1ex1dbadm01 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@v1ex1dbadm01 ~]#

Now logon the remaining compute nodes and restart clusterware:

login as: root
root@v1ex1dbadm02's password:
Last login: Wed Jun 10 18:26:56 IST 2020 from x.x.x.x on ssh
Last login: Wed Jun 10 18:30:56 2020 from x.x.x.x
[root@v1ex1dbadm02 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/oracle
[root@v1ex1dbadm02 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@v1ex1dbadm02 ~]#

Wait a few minutes and check clusterware is all up as shown below:

[root@v1ex1dbadm02 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATAC1.dg
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
...
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
ora.asm
ONLINE ONLINE v1ex1dbadm01 Started,STABLE
ONLINE ONLINE v1ex1dbadm02 Started,STABLE
...
ora.net1.network
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
ora.ons
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE v1ex1dbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE v1ex1dbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE v1ex1dbadm01 STABLE
...
1 ONLINE ONLINE v1ex1dbadm02 Open,STABLE
ora.oc4j
1 ONLINE ONLINE v1ex1dbadm01 STABLE
ora.scan1.vip
1 ONLINE ONLINE v1ex1dbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE v1ex1dbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE v1ex1dbadm01 STABLE
--------------------------------------------------------------------------------
[root@v1ex1dbadm02 ~]#

6. Restart OEM Agent

Optionally if you have an OEM agent (most likely), restart as follows:

[oracle@v1ex1dbadm01 ~]$ cd /u01/app/agent/agent_13.3.0.0.0/bin
[oracle@v1ex1dbadm01 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent is Not Running
[oracle@v1ex1dbadm01 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Starting agent ............................ started.
[oracle@v1ex1dbadm01 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.3.0.0.0
OMS Version : 13.3.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/agent/agent_inst
Agent Log Directory : /u01/app/agent/agent_inst/sysman/log
Agent Binaries : /u01/app/agent/agent_13.3.0.0.0
Core JAR Location : /u01/app/agent/agent_13.3.0.0.0/jlib
Agent Process ID : 122257
Parent Process ID : 122120
Agent URL : https://v1ex1dbadm01.v1.com:3872/emd/main/
Local Agent URL in NAT : https://v1ex1dbadm01.v1.com:3872/emd/main/
Repository URL : https://v1oem.v1.com:4903/empbs/upload
Started at : 2020-06-17 15:40:59
Started by user : oracle
Operating System : Linux version 4.1.12-124.30.1.el7uek.x86_64 (amd64)
Number of Targets : 43
Last Reload : (none)
Last successful upload : 2020-06-17 15:41:23
Last attempted upload : 2020-06-17 15:41:23
Total Megabytes of XML files uploaded so far : 0.1
Number of XML files pending upload : 5
Size of XML files pending upload(MB) : 0.02
Available disk space on upload filesystem : 21.34%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2020-06-17 15:41:18
Last successful heartbeat to OMS : 2020-06-17 15:41:18
Next scheduled heartbeat to OMS : 2020-06-17 15:42:20

---------------------------------------------------------------
Agent is Running and Ready
[oracle@v1ex1dbadm01 bin]$

Now on any other compute nodes:

[oracle@v1ex1dbadm01 bin]$ ssh v1ex1dbadm02
Last login: Wed Jun 17 12:37:20 IST 2020 from x.x.x.x on ssh
Last login: Wed Jun 17 15:43:07 2020 from x.x.x.x
[oracle@v1ex1dbadm02 ~]$ cd /u01/app/agent/agent_13.3.0.0.0/bin
[oracle@v1ex1dbadm02 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent is Not Running
[oracle@v1ex1dbadm02 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Starting agent .................................... started.
[oracle@v1ex1dbadm02 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.3.0.0.0
OMS Version : 13.3.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/agent/agent_inst
Agent Log Directory : /u01/app/agent/agent_inst/sysman/log
Agent Binaries : /u01/app/agent/agent_13.3.0.0.0
Core JAR Location : /u01/app/agent/agent_13.3.0.0.0/jlib
Agent Process ID : 189737
Parent Process ID : 189513
Agent URL : https://v1ex1dbadm02.v1.com:3872/emd/main/
Local Agent URL in NAT : https://v1ex1dbadm02.v1.com:3872/emd/main/
Repository URL : https://v1oem.v1.com:4903/empbs/upload
Started at : 2020-06-17 15:44:01
Started by user : oracle
Operating System : Linux version 4.1.12-124.30.1.el7uek.x86_64 (amd64)
Number of Targets : 37
Last Reload : (none)
Last successful upload : 2020-06-17 15:44:44
Last attempted upload : 2020-06-17 15:44:45
Total Megabytes of XML files uploaded so far : 0.17
Number of XML files pending upload : 1
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 28.51%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2020-06-17 15:44:28
Last successful heartbeat to OMS : 2020-06-17 15:44:28
Next scheduled heartbeat to OMS : 2020-06-17 15:45:28

---------------------------------------------------------------
Agent is Running and Ready
[oracle@v1ex1dbadm02 bin]$

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

One thought on “How to Startup an Oracle Exadata Machine

  1. Pingback: How to Shutdown an Oracle Exadata Machine | Zed DBA's Oracle Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s