UKOUG Conference 23 Day 2

UKOUG Conference ’23

If you missed my day 1, you can view it here.

Last night was the UKOUG Evening Celebration ’23, and although it finished at approximately 11:30pm, it took the team and board a couple of hours to pack up, debrief and have the traditional late night takeaway as some of us get too busy with the event, we don’t get a moment to eat properly or at all 🙁

Due to the late night, I skipped the first session and arrived for second session, however decided I’d use the hour to go over my presentation as I was presenting later in the day.

I had then planned to attend Kerry Osborne session, however one problem, Kerry wasn’t there! I checked with the team and they said he was delayed getting to the venue. So I looked who was around and saw Nigel Bayliss was around, and asked if he could fill the slot with any material he had already presented recently, as we had 50-100 people in the Ozone waiting for a presentation 🙄. He suggested Dom Giles, who kindly agreed to present minutes before the start time.

It was greatly appreciated, and Dom Giles presented “Oracle Database: What’s New, What’s Coming” which is what he presented at the recent Oracle DatabaseWorld at CloudWorld:

He talked about many new features in 23c and also what was coming. One of these new features that coming, was True Cache, which is similar to Redis which you put cache in front of the application to improve performance for data that doesn’t need to be most current data:

He also spoke about performance enhancements in 23c:

Finally concluded on database release and support timeline, the key being that 19c waived extended support increased by a year and 23c projected release in early 2024:

Whilst Dom was presenting, the UKOUG team had informed me that Kerry had arrived. So I went and spoke to him, luckily the next hour slot in Ozone was free as it was lunch, so I advised the audience that once Dom was finished, that Kerry would delivery his presentation in that free slot and they were free to stay and listen.

Next I attended “How to Improve Your Ability to Solve Complex Performance Problems” by Kerry Osborne, Google:

Has he had issues with his laptop, he decided to abandon his slides but still go through the process of how to solve complex performance problem, broken down into 3 parts:

  • Explain the basic process we must go through to solve a complex performance problem
  • Discuss some of the main factors that can inhibit our efforts
  • Discuss some of the techniques we can apply to improve our chances

It was interesting to hear from his 20 years of experiences, tips and tricks as well as experiences of working with of the best performance troubleshooters in the world.

Next, I went to have a light lunch and head to my next session which is unusual for me to attend, but as I’ve heard so much about CX lately, I thought it was worthwhile attending “Innovation with Oracle Intelligent Advisor” by Leona Chauhan:

Very interesting session, where I came to understand that Oracle Intelligent Advisor is a rules engine technology that is purpose built for automating complex policies and creating tailored personalised applications. For example, a company may have a complex maternity policy, that have large ruleset, that could be converted to rules and an application created off the back of these rules that will be used to question/interview the users and provide tailored response on their entitlement. The impressive part, was the tooling can take policy document and parse them into rules, which then can be refined into the final set of rules which the application is built off.

One of the key features is it automates decision-making, carrying out complex calculations, yet has a low code interface for development. This can save thousand of hours reducing errors in applications by intelligent online interviews, which can respond automatically, giving instant decision in a downloadable letter or form, which mirrors policy or process documents.

It can be embedded within a public webpage or private intranet site and can be integrated into ERP system, which can be on-premises or cloud. It is very flexible and is instrumental in giving the best customer experience.

Next what my own session “Harnessing the power of Oracle Exadata Database Machine“:

Where I talked about the “Oracle Secret Sauce”, well more it’s just simply Brawn Hardware with Smart Software. And it’s the integration of hardware and software that sets apart Exadata from the rest, as Oracle says: “Hardware and Software: Engineered to Work Together”.

I also went through:

  • Why move to Exadata
  • The benefits
  • What’s new in X10M (see blog post here for more info)
  • Migration Challenges
  • Tips and Tricks/Best Practices
  • How to tackle database consolidation
  • Different types of Exadata offerings/Responsibility Model
  • And of course, harnessing the power of Oracle Exadata Database Machine

Once my presentation was done, I was able to relax and attended “A Practical example of implementing TDE and TLS1.2 in an Oracle 19c Data Guard Environment” by Clive Archibald, Virgin Media O2:

This was delivered by one of my clients, Clive, who I have worked for nearly 10 years on their Exadata estate supporting their Smart Metering:

He went through a comprehensive example of how Virgin Media O2 had setup TLS 1.2, using certificates Azure Key Vault to manage them. Also went through example of how TDE was implemented.

Then came the final session of the conference (the best for last 😉), which I attended “Exadata: Strategy and Roadmap for New Technologies, Cloud, and On-Premises” by Alex Blyth, Oracle:

He first spoke about what Exadata is and the vision:

He then went onto discussing Oracle Exadata Exascale:

An exciting new feature, that uses Exadata platform to cater for broad spectrum of requirements, from small configurations to large deployment:

This is similar to how Autonomous database (shared infrastructure) can be consumed from a core and 1TB storage up to thousands of cores and petabytes of storage. It’s great for hyperscale, which Alex said selfishly Oracle needed for Oracle Cloud, but is also great for customer to use in the Oracle Cloud or on-premises, to allow for flexible elastic architecture.

Also, what great is state of the art storage efficient database cloning, meaning no more Exadata Sparse Clones with test master! Actual space efficient redirect on write technology:

Exadata Exascale, will facilitate a new OCI Exadata Database Service namely ExaDB-XS:

This will provide to be very flexible, elastic and cost efficient! It’s a really interesting space, which I look forward to:

So in summary, next gen software architecture, simple, flexible, efficient, foundation for multi-tenant Exadata Database Service:

Once the presentation was over, it was a case of wrapping up the conference, working with the team removing all the branding, pack up and leaving with the conference blues 😕:

Till next time! Hope to see you all soon!

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Oracle switches to AMD for full Exadata X10M lineup, what else is new?

Chipset switch to AMD

So back in September 2022, I noticed the Oracle Exadata Cloud Infrastructure X9M switched to AMD, where as the Exadata X9M On-premises and Exdata X9M Cloud @ Customer remained on Intel. More details in my blog post here.

It come as no surprise the whole lineup has now switched to AMD! Why? Well putting it simply AMD has beaten Intel on the number of cores for sometime now and with the scalability of Exadata it make senses to have the upper limit cores that AMD offers. Oracle is now using the AMD EPYC 9J14 processors.

Where has the Persistent Memory gone?

Since Exadata X8M, Intel Optane Persistent Memory has existed in the storage cells, bringing an additional layer to the multi-tiered storage architecture. Persistent Memory has the benefit of being order of magnitude faster then Flash but at the fraction of the cost of DDR memory resulting in the Exadata Persistent Memory Data Accelerator.

Simply put, you can’t have AMD chipset with Intel Optane Persistent Memory, so Oracle had to come up with a new solution which they glossed over in the announcement. Which they replaced the Persistent Memory with DDR5 memory and address the memory from the compute nodes using the new feature called Exadata RDMA Memory (XRMEM). This extends the existing Remote Direct Memory Access (RDMA), to create a new shared read accelerator. Interesting Oracle compromised on the writes accelerator that Persistent Memory gave but they would have done the analysis and probably Persistent Memory was largely fronting reads and the penalty to flush to Flash instead of Persistent Memory probably wasn’t that significant. There is now 1.5TB DDR5 DRAM in storage cell compared to 256GB in X9M with 1.5TB Persistent Memory.

What else is new?

The other highlights are:

  • Increased memory configuration of 512GB, 1.5TB, 2.25TB and 3TB DDR5 DRAM in the compute nodes
  • PCIe 4 replaced with PCIe 5, to give 2x 100Gb/sec active-active RoCE network for impressive total throughput of 200Gb/sec
  • Extreme Flash storage server now has the introduction of “capacity optimised” flash drive combined with “performance optimised” flash cards to give impressive increase of 2.4x over X9M
  • 22TB disks instead of 18TB on X9M, giving 22% increase storage

Those that know me, will know I have been predicting:

  • Full switch to AMD chipset for whole Exadata lineup
  • Some sort of replacement for Persistent Memory
  • 22TB hard disk drives

I’m glad to say I was right on all accounts 🙂

For more Info

Please refer to the following links:

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

Oracle Switches CPU from Intel to AMD for Exadata Cloud Services

Working on Oracle Exadata Cloud Infrastructure X9M, I noticed the upper OCPU went up pretty high compared to X8M. This made me curious and I discovered the switch from Intel to AMD for the CPU:

Exadata X9M (Exadata Cloud Services)

[root@v1exacs01c1db01 ~]# more /proc/cpuinfo | grep "vendor_id|model name" | sort | uniq -c
8 model name : AMD EPYC 7J13 64-Core Processor
8 vendor_id : AuthenticAMD
[root@v1exacs01c1db01 ~]#

Compare this to on-premises Exadata and they are still Intel processors:

Exadata X9M (on-premises)

[root@v1exadb01 ~]# more /proc/cpuinfo | grep "vendor_id|model name" | sort | uniq -c
76 model name      : Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
76 vendor_id       : GenuineIntel
[root@v1exadb01 ~]#

Comparing with previous Exadata Cloud @ Customer and this is also still Intel processors:

Exadata X8M (Exadata Cloud @ Customer)

[root@v1exacc01c1db01 ~]# more /proc/cpuinfo | grep "vendor_id|model name" | sort | uniq -c
6 model name      : Intel(R) Xeon(R) Platinum 8270CL CPU @ 2.70GHz
6 vendor_id       : GenuineIntel
[root@v1exacc01c1db01 ~]#

This change make sense as the latest Intel processor is the “Intel® Xeon® Platinum 8358 Processor”, which has 32 cores and 64 threads.

Compare this with latest AMD processor “AMD EPYC™ 7713”, which has 64 cores and 128 threads. I believe the AMD EPYC 7J13 in the Exadata Cloud Infrastructure X9M is of the same family.

This allows Oracle to offer 252 OCPU for Quarter Rack, 2 sockets x 64 cores x 2 database servers less 2 cores per database servers for KVM. This is double what could be offered if Intel processors were used. And seems as Oracle charge per a OCPU, it’s a smart move that was quietly done 😉

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to restart InfiniBand Subnet Manager Exadata

You may encounter a scenario where the subnet manager is down on your InfiniBand on your Exadata. To restart is very simple:

[root@v1ex1dbadm01 ~]# ssh v1ex1sw-ibb01
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
[root@v1ex1sw-ibb01 ~]# enablesm
Starting IB Subnet Manager. [ OK ]
Starting partitiond-daemon [ OK ]
[root@v1ex1sw-ibb01 ~]# ps -ef | grep opensm
root 8075 1 0 09:59 ? 00:00:00 /usr/sbin/opensm --daemon
root 8224 7909 0 09:59 pts/0 00:00:00 grep opensm
[root@v1ex1sw-ibb01 ~]#

More information can be found here:
Sun Datacenter InfiniBand Switch 36 – Enable the Subnet Manager

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)

How to Startup an Oracle Exadata Machine

There maybe times when you are required to fully shutdown an Oracle Exadata Machine, for example for maintenance.

For instructions on how to shutdown an Oracle Exadata Machine, please refer to my blog post:
How to Shutdown an Oracle Exadata Machine

Once shutdown, you will need to be able to re-start which this blog post will detail.

Below is the My Oracle Support note used to carry out the startup:
Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration (Doc ID 1093890.1)

1. Pre-requisites

Ensure you have all the compute nodes and storage cells ILOM addresses and correct passwords.  Otherwise you will not be able to remotely power back on and will require a physical power on using the power button on the front panels.

2. Power on first Compute Node

You can power on the first compute node via the ilom via ssh or WebILOM.  I prefer the ssh method shown below:

[AnwarZ@v1proxy1 ~]$ ssh root@v1ex1dbadm01-ilom
Password:

Oracle(R) Integrated Lights Out Manager

Version 4.0.4.37 r130617

Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: v1ex1dbadm01-ilom

-> show /SYSTEM

/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
Log

Properties:
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = Exadata X5-2
qpart_id = XXXXXX
part_number = Exadata X5-2
serial_number = XXXXXXXXXX
component_model = ORACLE SERVER X5-2
component_part_number = XXXXXXX
component_serial_number = XXXXXXXXXX
system_identifier = Exadata Database Machine X5-2 XXXXXXXXXX
system_fw_version = 4.0.4.37
primary_operating_system = Not Available
primary_operating_system_detail = Comprehensive System monitoring is not available. Ensure the host is
running with the Hardware Management Pack. For details go to
http://www.oracle.com/goto/ilom-redirect/hmp-osa
host_primary_mac_address = xx:xx:xx:xx:xx:xx
ilom_address = x.x.x.x
ilom_mac_address = xx:xx:xx:xx:xx:xx
locator_indicator = Off
power_state = Off
actual_power_consumption = 22 watts
action = (Cannot show property)

Commands:
cd
reset
set
show
start
stop

-> start /SYSTEM
Are you sure you want to start /System (y/n)? y
Starting /System

-> show /SYSTEM

/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
Log

Properties:
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = Exadata X5-2
qpart_id = XXXXXX
part_number = Exadata X5-2
serial_number = XXXXXXXXXX
component_model = ORACLE SERVER X5-2
component_part_number = XXXXXXX
component_serial_number = XXXXXXXXXX
system_identifier = Exadata Database Machine X5-2 XXXXXXXXXX
system_fw_version = 4.0.4.37
primary_operating_system = Not Available
primary_operating_system_detail = Comprehensive System monitoring is not available. Ensure the host is
running with the Hardware Management Pack. For details go to
http://www.oracle.com/goto/ilom-redirect/hmp-osa
host_primary_mac_address = xx:xx:xx:xx:xx:xx
ilom_address = x.x.x.x
ilom_mac_address = xx:xx:xx:xx:xx:xx
locator_indicator = Off
power_state = On
actual_power_consumption = 220 watts
action = (Cannot show property)

Commands:
cd
reset
set
show
start
stop

-> exit
Connection to v1ex1dbadm01-ilom closed.
[AnwarZ@v1proxy1 ~]$

3. Power on all Storage Cells

Login to the first compute node and power on all the storage cells as shown below:

login as: root
root@v1ex1dbadm01's password:
Last login: Wed Jun 10 09:21:41 IST 2020 from v1ex1dbadm01.v1.com on ssh
Last login: Wed Jun 10 17:31:31 2020 from x.x.x.x
[root@v1ex1dbadm01 ~]# uptime
17:31:37 up 1 min, 1 user, load average: 3.37, 1.22, 0.44
[root@v1ex1dbadm01 ~]# export HISTIGNORE='*'
[root@v1ex1dbadm01 ~]# for host in `cat /opt/oracle.SupportTools/onecommand/cell_group`; do
> echo ${host}: `ipmitool -I lanplus -H ${host}-ilom -U root -P XXXXXXXX chassis power on`
> done
v1ex1celadm01: Chassis Power Control: Up/On
v1ex1celadm02: Chassis Power Control: Up/On
v1ex1celadm03: Chassis Power Control: Up/On
[root@v1ex1dbadm01 ~]# export HISTIGNORE=''
[root@v1ex1dbadm01 ~]#

Please Note: the HISTIGNORE is used, so the password isn’t kept in history.

Check the storage cell services are up:

[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root 'hostname; uptime'
v1ex1celadm01: v1ex1celadm01.v1.com
v1ex1celadm01: 18:10:21 up 32 min, 0 users, load average: 1.86, 1.95, 2.03
v1ex1celadm02: v1ex1celadm02.v1.com
v1ex1celadm02: 18:10:21 up 32 min, 0 users, load average: 1.47, 1.82, 1.97
v1ex1celadm03: v1ex1celadm03.v1.com
v1ex1celadm03: 18:10:22 up 32 min, 0 users, load average: 1.51, 1.85, 2.01
[root@v1ex1dbadm01 ~]#
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e 'list cell detail'"
v1ex1celadm01: name: v1ex1celadm01
v1ex1celadm01: accessLevelPerm: remoteLoginEnabled
v1ex1celadm01: bbuStatus: normal
v1ex1celadm01: cellVersion: OSS_19.2.7.0.0_LINUX.X64_191012
v1ex1celadm01: cpuCount: 32/32
v1ex1celadm01: diagHistoryDays: 7
v1ex1celadm01: eighthRack: FALSE
v1ex1celadm01: fanCount: 8/8
v1ex1celadm01: fanStatus: normal
v1ex1celadm01: flashCacheMode: WriteBack
v1ex1celadm01: httpsAccess: ALL
v1ex1celadm01: id: XXXXXXXXXX
v1ex1celadm01: interconnectCount: 2
v1ex1celadm01: interconnect1: ib0
v1ex1celadm01: interconnect2: ib1
v1ex1celadm01: iormBoost: 0.0
v1ex1celadm01: ipaddress1: x.x.x.x/22
v1ex1celadm01: ipaddress2: x.x.x.x/22
v1ex1celadm01: kernelVersion: 4.1.12-124.30.1.el7uek.x86_64
v1ex1celadm01: locatorLEDStatus: off
v1ex1celadm01: makeModel: Oracle Corporation ORACLE SERVER X5-2L High Capacity
v1ex1celadm01: memoryGB: 94
v1ex1celadm01: metricHistoryDays: 7
v1ex1celadm01: notificationMethod: snmp
v1ex1celadm01: notificationPolicy: critical,warning,clear
v1ex1celadm01: offloadGroupEvents:
v1ex1celadm01: powerCount: 2/2
v1ex1celadm01: powerStatus: normal
v1ex1celadm01: ramCacheMaxSize: 0
v1ex1celadm01: ramCacheMode: Auto
v1ex1celadm01: ramCacheSize: 0
v1ex1celadm01: releaseImageStatus: success
v1ex1celadm01: releaseVersion: 19.2.7.0.0.191012
v1ex1celadm01: rpmVersion: cell-19.2.7.0.0_LINUX.X64_191012-1.x86_64
v1ex1celadm01: releaseTrackingBug: 30393131
v1ex1celadm01: rollbackVersion: 18.1.18.0.0.190709
v1ex1celadm01: snmpSubscriber: host=x.x.x.x,port=162,community=public,type=ASR,asrmPort=16161
v1ex1celadm01: host=x.x.x.x,port=161,community=V1
v1ex1celadm01: host=x.x.x.x,port=161,community=V1
v1ex1celadm01: status: online
v1ex1celadm01: temperatureReading: 22.0
v1ex1celadm01: temperatureStatus: normal
v1ex1celadm01: upTime: 0 days, 0:33
v1ex1celadm01: usbStatus: normal
v1ex1celadm01: cellsrvStatus: running
v1ex1celadm01: msStatus: running
v1ex1celadm01: rsStatus: running
v1ex1celadm02: name: v1ex1celadm02
...
v1ex1celadm02: status: online
v1ex1celadm02: temperatureReading: 22.0
v1ex1celadm02: temperatureStatus: normal
v1ex1celadm02: upTime: 0 days, 0:33
v1ex1celadm02: usbStatus: normal
v1ex1celadm02: cellsrvStatus: running
v1ex1celadm02: msStatus: running
v1ex1celadm02: rsStatus: running
v1ex1celadm03: name: v1ex1celadm03
...
v1ex1celadm03: status: online
v1ex1celadm03: temperatureReading: 22.0
v1ex1celadm03: temperatureStatus: normal
v1ex1celadm03: upTime: 0 days, 0:33
v1ex1celadm03: usbStatus: normal
v1ex1celadm03: cellsrvStatus: running
v1ex1celadm03: msStatus: running
v1ex1celadm03: rsStatus: running
[root@v1ex1dbadm01 ~]#

4. Power on remaining Compute Nodes

Power on remaining compute nodes via ipmitool:

[root@v1ex1dbadm01 ~]# export HISTIGNORE='*'
[root@v1ex1dbadm01 ~]# ipmitool -I lanplus -H v1ex1dbadm02-ilom -U root -P XXXXXXXX chassis power on
Chassis Power Control: Up/On
[root@v1ex1dbadm01 ~]# export HISTIGNORE=''
[root@v1ex1dbadm01 ~]#

If half or full rack, then the following can be used:

for host in `cat dbs_group_all_but_first`; do
echo ${host}: `ipmitool -H ${host}-ilom -U root -P XXXXXXXX chassis power on`
done

Check compute nodes are up:

[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/dbs_group -l root 'hostname; uptime'
v1ex1dbadm01: v1ex1dbadm01.v1.com
v1ex1dbadm01: 18:21:55 up 12 min, 1 user, load average: 0.22, 0.89, 1.54
v1ex1dbadm02: v1ex1dbadm02.v1.com
v1ex1dbadm02: 18:21:55 up 3 min, 0 users, load average: 2.44, 1.58, 0.66
[root@v1ex1dbadm01 ~]#

5. Re-enable clusterware autostart

Re-enable clusterware autostart via dcli:

[root@v1ex1dbadm01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@v1ex1dbadm01 ~]# dcli -g /opt/oracle.SupportTools/onecommand/dbs_group -l root /u01/app/12.1.0.2/grid/bin/crsctl enable crs
v1ex1dbadm01: CRS-4622: Oracle High Availability Services autostart is enabled.
v1ex1dbadm02: CRS-4622: Oracle High Availability Services autostart is enabled.
[root@v1ex1dbadm01 ~]#

6. Restart Grid Infrastructure on the cluster

Start clusterware on first compute node:

[root@v1ex1dbadm01 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@v1ex1dbadm01 ~]#

Now logon the remaining compute nodes and restart clusterware:

login as: root
root@v1ex1dbadm02's password:
Last login: Wed Jun 10 18:26:56 IST 2020 from x.x.x.x on ssh
Last login: Wed Jun 10 18:30:56 2020 from x.x.x.x
[root@v1ex1dbadm02 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/oracle
[root@v1ex1dbadm02 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@v1ex1dbadm02 ~]#

Wait a few minutes and check clusterware is all up as shown below:

[root@v1ex1dbadm02 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATAC1.dg
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
...
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
ora.asm
ONLINE ONLINE v1ex1dbadm01 Started,STABLE
ONLINE ONLINE v1ex1dbadm02 Started,STABLE
...
ora.net1.network
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
ora.ons
ONLINE ONLINE v1ex1dbadm01 STABLE
ONLINE ONLINE v1ex1dbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE v1ex1dbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE v1ex1dbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE v1ex1dbadm01 STABLE
...
1 ONLINE ONLINE v1ex1dbadm02 Open,STABLE
ora.oc4j
1 ONLINE ONLINE v1ex1dbadm01 STABLE
ora.scan1.vip
1 ONLINE ONLINE v1ex1dbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE v1ex1dbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE v1ex1dbadm01 STABLE
--------------------------------------------------------------------------------
[root@v1ex1dbadm02 ~]#

6. Restart OEM Agent

Optionally if you have an OEM agent (most likely), restart as follows:

[oracle@v1ex1dbadm01 ~]$ cd /u01/app/agent/agent_13.3.0.0.0/bin
[oracle@v1ex1dbadm01 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent is Not Running
[oracle@v1ex1dbadm01 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Starting agent ............................ started.
[oracle@v1ex1dbadm01 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.3.0.0.0
OMS Version : 13.3.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/agent/agent_inst
Agent Log Directory : /u01/app/agent/agent_inst/sysman/log
Agent Binaries : /u01/app/agent/agent_13.3.0.0.0
Core JAR Location : /u01/app/agent/agent_13.3.0.0.0/jlib
Agent Process ID : 122257
Parent Process ID : 122120
Agent URL : https://v1ex1dbadm01.v1.com:3872/emd/main/
Local Agent URL in NAT : https://v1ex1dbadm01.v1.com:3872/emd/main/
Repository URL : https://v1oem.v1.com:4903/empbs/upload
Started at : 2020-06-17 15:40:59
Started by user : oracle
Operating System : Linux version 4.1.12-124.30.1.el7uek.x86_64 (amd64)
Number of Targets : 43
Last Reload : (none)
Last successful upload : 2020-06-17 15:41:23
Last attempted upload : 2020-06-17 15:41:23
Total Megabytes of XML files uploaded so far : 0.1
Number of XML files pending upload : 5
Size of XML files pending upload(MB) : 0.02
Available disk space on upload filesystem : 21.34%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2020-06-17 15:41:18
Last successful heartbeat to OMS : 2020-06-17 15:41:18
Next scheduled heartbeat to OMS : 2020-06-17 15:42:20

---------------------------------------------------------------
Agent is Running and Ready
[oracle@v1ex1dbadm01 bin]$

Now on any other compute nodes:

[oracle@v1ex1dbadm01 bin]$ ssh v1ex1dbadm02
Last login: Wed Jun 17 12:37:20 IST 2020 from x.x.x.x on ssh
Last login: Wed Jun 17 15:43:07 2020 from x.x.x.x
[oracle@v1ex1dbadm02 ~]$ cd /u01/app/agent/agent_13.3.0.0.0/bin
[oracle@v1ex1dbadm02 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent is Not Running
[oracle@v1ex1dbadm02 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Starting agent .................................... started.
[oracle@v1ex1dbadm02 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.3.0.0.0
OMS Version : 13.3.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/agent/agent_inst
Agent Log Directory : /u01/app/agent/agent_inst/sysman/log
Agent Binaries : /u01/app/agent/agent_13.3.0.0.0
Core JAR Location : /u01/app/agent/agent_13.3.0.0.0/jlib
Agent Process ID : 189737
Parent Process ID : 189513
Agent URL : https://v1ex1dbadm02.v1.com:3872/emd/main/
Local Agent URL in NAT : https://v1ex1dbadm02.v1.com:3872/emd/main/
Repository URL : https://v1oem.v1.com:4903/empbs/upload
Started at : 2020-06-17 15:44:01
Started by user : oracle
Operating System : Linux version 4.1.12-124.30.1.el7uek.x86_64 (amd64)
Number of Targets : 37
Last Reload : (none)
Last successful upload : 2020-06-17 15:44:44
Last attempted upload : 2020-06-17 15:44:45
Total Megabytes of XML files uploaded so far : 0.17
Number of XML files pending upload : 1
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 28.51%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2020-06-17 15:44:28
Last successful heartbeat to OMS : 2020-06-17 15:44:28
Next scheduled heartbeat to OMS : 2020-06-17 15:45:28

---------------------------------------------------------------
Agent is Running and Ready
[oracle@v1ex1dbadm02 bin]$

If you found this blog post useful, please like as well as follow me through my various Social Media avenues available on the sidebar and/or subscribe to this oracle blog via WordPress/e-mail.

Thanks

Zed DBA (Zahid Anwar)