Wednesday 18 September 2019

OEL Linux Update - How to relink RAC/Cluster installation after OS update


This entry is based on MOS:
How To Relink The Oracle Grid Infrastructure Standalone (Restart) Installation Or Oracle Grid Infrastructure RAC/Cluster Installation (11.2 or 12c). (Doc ID 1536057.1)


In case OS Linux update, we need to think about relinking Oracle Software. This post show how to perform relinking in Oracle Clusterware(18.4) configuration after OS update from OEL 7.6 to 7.7:


[oracle@rac02 ~]$ cat /etc/oracle-release 
Oracle Linux Server release 7.6
[oracle@rac02 ~]$ uname -a
Linux rac02 4.14.35-1844.0.7.el7uek.x86_64 #2 SMP Wed Dec 12 19:48:02 PST 2018 x86_64 x86_64 x86_64 GNU/Linux


[oracle@rac02 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
28864607;ACFS RELEASE UPDATE 18.5.0.0.0 (28864607)
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)


##Stop and disable clusterware


[root@rac02 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/orabase_grid18
 

[root@rac02 ~]# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.
 

[root@rac02 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac02'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rac02'
CRS-2673: Attempting to stop 'ora.dbprod.db' on 'rac02'
CRS-2673: Attempting to stop 'ora.chad' on 'rac01'
CRS-2673: Attempting to stop 'ora.chad' on 'rac02'
CRS-2677: Stop of 'ora.dbprod.db' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac02'
CRS-2673: Attempting to stop 'ora.MGMT.dg' on 'rac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac02'
CRS-2677: Stop of 'ora.OCR.dg' on 'rac02' succeeded
CRS-2677: Stop of 'ora.MGMT.dg' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac02' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac02'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac02'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac02'
CRS-2677: Stop of 'ora.scan3.vip' on 'rac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rac02' succeeded
CRS-2677: Stop of 'ora.chad' on 'rac01' succeeded
CRS-2677: Stop of 'ora.chad' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rac02'
CRS-2677: Stop of 'ora.qosmserver' on 'rac02' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac02' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rac02'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.rac02.vip' on 'rac02'
CRS-2677: Stop of 'ora.rac02.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rac01'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rac01'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac01'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac01'
CRS-2672: Attempting to start 'ora.cvu' on 'rac01'
CRS-2672: Attempting to start 'ora.rac02.vip' on 'rac01'
CRS-2676: Start of 'ora.cvu' on 'rac01' succeeded
CRS-2676: Start of 'ora.rac02.vip' on 'rac01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rac01'
CRS-2676: Start of 'ora.scan3.vip' on 'rac01' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac01'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac01'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac01' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rac01' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.chad' on 'rac01'
CRS-2676: Start of 'ora.chad' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac02'
CRS-2677: Stop of 'ora.ons' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac02'
CRS-2677: Stop of 'ora.net1.network' on 'rac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac02'
CRS-2673: Attempting to stop 'ora.crf' on 'rac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac02'
CRS-2677: Stop of 'ora.crf' on 'rac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac02'
CRS-2677: Stop of 'ora.ctssd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac02'
CRS-2677: Stop of 'ora.cssd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac02'
CRS-2677: Stop of 'ora.gpnpd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.


##Start update with yum:

[root@rac02 ~]# yum update
Loaded plugins: langpacks, ulninfo
ol7_UEKR5                                                                                                                                                                                   | 2.5 kB  00:00:00    
ol7_latest                                                                                                                                                                                  | 2.7 kB  00:00:00    
(1/2): ol7_latest/x86_64/updateinfo                                                                                                                                                         | 1.1 MB  00:00:01    
(2/2): ol7_latest/x86_64/primary_db                                                                                                                                                         |  20 MB  00:00:06    
Resolving Dependencies
--> Running transaction check
---> Package GeoIP.x86_64 0:1.5.0-13.el7 will be updated
---> Package GeoIP.x86_64 0:1.5.0-14.el7 will be an update
---> Package ModemManager-glib.x86_64 0:1.6.10-3.el7_6 will be an update
---> Package NetworkManager.x86_64 1:1.12.0-8.el7_6 will be updated
---> Package NetworkManager.x86_64 1:1.18.0-5.el7_7.1 will be an update...
---> Package abrt-libs.x86_64 0:2.1.11-52.0.1.el7 will be updated
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.191.b12-1.el7_6 will be updated
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.222.b10-1.el7_7 will be an update
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.191.b12-1.el7_6 will be updated
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.222.b10-1.el7_7 will be an update
--> Processing Dependency: pcsc-lite-libs(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.222.b10-1.el7_7.x86_64
---> Package kernel.x86_64 0:3.10.0-1062.1.1.el7 will be installed
---> Package kernel-headers.x86_64 0:3.10.0-957.1.3.el7 will be updated
---> Package kernel-headers.x86_64 0:3.10.0-1062.1.1.el7 will be an update
---> Package kernel-tools.x86_64 0:3.10.0-957.1.3.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-1062.1.1.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-957.1.3.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-1062.1.1.el7 will be an update
---> Package kernel-uek.x86_64 0:4.14.35-1902.5.2.el7uek will be installed
---> Package kernel-uek-devel.x86_64 0:4.14.35-1902.5.2.el7uek will be installed
---> Package kexec-tools.x86_64 0:2.0.15-21.0.3.el7 will be updated
---> Package kexec-tools.x86_64 0:2.0.15-33.0.1.el7 will be an update
---> Package kmod.x86_64 0:20-23.0.1.el7 will be updated
...

---> Package oracle-logos.noarch 0:70.0.3-4.0.9.el7 will be updated
---> Package oracle-logos.noarch 0:70.7.0-1.0.4.el7 will be an update
---> Package oraclelinux-release.x86_64 7:7.6-1.0.15.el7 will be updated
---> Package oraclelinux-release.x86_64 7:7.7-1.0.5.el7 will be an update
---> Package redhat-release-server.x86_64 1:7.6-4.0.1.el7 will be updated
---> Package redhat-release-server.x86_64 1:7.7-10.0.1.el7 will be an update
---> Package sysstat.x86_64 0:10.1.5-18.el7 will be an update
 Verifying  : gjs-1.52.3-1.el7.x86_64                                                                                                                                                                    976/1010
  Verifying  : glusterfs-cli-3.12.2-18.el7.x86_64                                                                                                                                                         977/1010
...

  Verifying  : 1:dbus-1.10.24-12.0.1.el7.x86_64                                                                                                                                                          1010/1010

Installed:
  kernel.x86_64 0:3.10.0-1062.1.1.el7          kernel-uek.x86_64 0:4.14.35-1902.5.2.el7uek          kernel-uek-devel.x86_64 0:4.14.35-1902.5.2.el7uek          urw-base35-fonts.noarch 0:20170801-10.el7        
...                                
Updated:

...
  usermode-gtk.x86_64 0:1.111-6.el7                                     util-linux.x86_64 0:2.23.2-61.el7                                     vdo.x86_64 0:6.1.2.41-4.el7                                         
  vim-common.x86_64 2:7.4.629-6.el7                                     vim-enhanced.x86_64 2:7.4.629-6.el7                                   vim-filesystem.x86_64 2:7.4.629-6.el7                               
  vim-minimal.x86_64 2:7.4.629-6.el7                                    vinagre.x86_64 0:3.22.0-12.el7                                        volume_key-libs.x86_64 0:0.3.9-9.el7                                
  webkitgtk4.x86_64 0:2.22.7-2.el7                                      webkitgtk4-jsc.x86_64 0:2.22.7-2.el7                                  webkitgtk4-plugin-process-gtk2.x86_64 0:2.22.7-2.el7                
  wget.x86_64 0:1.14-18.el7_6.1                                         xorg-x11-drv-ati.x86_64 0:19.0.1-2.el7                                xorg-x11-drv-qxl.x86_64 0:0.1.5-5.el7                               
  xorg-x11-drv-vesa.x86_64 0:2.4.0-3.el7                                xorg-x11-drv-wacom.x86_64 0:0.36.1-3.el7                              xorg-x11-server-Xorg.x86_64 0:1.20.4-7.el7                          
  ...

Replaced:
  urw-fonts.noarch 0:2.4-16.el7                                                                                                                                                                                   

Complete!


##Server reboot  and OS release check:

[root@rac02 ~]# reboot

[oracle@rac02 ~]$ cat /etc/oracle-release Oracle Linux Server release 7.7
[oracle@rac02 ~]$ uname -a
Linux rac02 4.14.35-1902.5.2.el7uek.x86_64 #2 SMP Wed Aug 28 21:39:33 GMT 2019 x86_64 x86_64 x86_64 GNU/Linux

##Start relinking(Look at the users executing scripts):

[root@rac02 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/orabase_grid18

[root@rac02 ~]# cd $ORACLE_HOME
[root@rac02 grid]# cd crs/install/
[root@rac02 install]# pwd
/u01/app/18/grid/crs/install

[root@rac02 install]# ./rootcrs.sh -unlock
Using configuration parameter file: /u01/app/18/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/orabase_grid18/crsdata/rac02/crsconfig/crsunlock_rac02_2019-09-17_10-32-16AM.log
2019/09/17 10:32:18 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2019/09/17 10:33:59 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2019/09/17 10:33:59 CLSRSC-347: Successfully unlock /u01/app/18/grid

[oracle@rac02 ~]$ . oraenv
ORACLE_SID = [+ASM2] ? 
The Oracle base remains unchanged with value /u01/app/orabase_grid18
[oracle@rac02 ~]$ cd $ORACLE_HOME/bin
[oracle@rac02 bin]$ pwd
/u01/app/18/grid/bin
[oracle@rac02 bin]$ ./relink all
writing relink log to: /u01/app/18/grid/install/relink_2019-09-17_10-40-09AM.log
[oracle@rac02 bin]$ 

[root@rac02 ~]# . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/orabase_grid18
[root@rac02 ~]# cd $ORACLE_HOME
[root@rac02 grid]# cd rdbms/install/
[root@rac02 install]# ./rootadd_
rootadd_filemap.sh  rootadd_rdbms.sh    
[root@rac02 install]# ./rootadd_rdbms.sh 
[root@rac02 install]# cd $ORACLE_HOME
[root@rac02 grid]# cd crs/install/
[root@rac02 install]# ./root
rootcrs.pl  rootcrs.sh  roothas.pl  roothas.sh  
[root@rac02 install]# ./rootcrs.sh -lock
Using configuration parameter file: /u01/app/18/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/orabase_grid18/crsdata/rac02/crsconfig/crslock_rac02_2019-09-17_10-44-15AM.log
2019/09/17 10:44:19 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

##Enable CRS and start it


[root@rac02 install]# crsctl enable crs

CRS-4622: Oracle High Availability Services autostart is enabled.
[root@rac02 install]# crsctl start crs

CRS-4123: Oracle High Availability Services has been started.


[root@rac02 install]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
ora.MGMT.dg
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
ora.OCR.dg
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
ora.chad
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
ora.net1.network
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
ora.ons
               ONLINE  ONLINE       rac01                    STABLE
               ONLINE  ONLINE       rac02                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac02                    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac01                    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac01                    STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac01                    169.254.27.231 192.1
                                                             68.200.121,STABLE
ora.asm
      1        ONLINE  ONLINE       rac01                    Started,STABLE
      2        ONLINE  ONLINE       rac02                    Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       rac01                    STABLE
ora.dbprod.db
      1        ONLINE  ONLINE       rac01                    Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             .2/db_1,STABLE
      2        ONLINE  ONLINE       rac02                    Open,HOME=/u01/app/o
                                                             racle/product/12.1.0
                                                             .2/db_1,STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac01                    Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       rac01                    STABLE
ora.rac01.vip
      1        ONLINE  ONLINE       rac01                    STABLE
ora.rac02.vip
      1        ONLINE  ONLINE       rac02                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac02                    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac01                    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac01                    STABLE
--------------------------------------------------------------------------------




oracle restart

[oracle@orapdr01 ~]$ cat /etc/oracle-release 
Oracle Linux Server release 7.6
[oracle@orapdr01 ~]$ uname -r
4.14.35-1902.3.1.el7uek.x86_64


[oracle@orapdr01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       orapdr01                 STABLE
ora.ons
               OFFLINE OFFLINE      orapdr01                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                               STABLE
ora.dbproddr.db
      1        ONLINE  INTERMEDIATE orapdr01                 Mounted (Closed),HOM
                                                             E=/u01/app/oracle/pr
                                                             oduct/12.1.0.2/db_1,
                                                             STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orapdr01                 STABLE
--------------------------------------------------------------------------------
[oracle@orapdr01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       orapdr01                 STABLE
ora.ons
               OFFLINE OFFLINE      orapdr01                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                               STABLE
ora.dbproddr.db
      1        ONLINE  INTERMEDIATE orapdr01                 Mounted (Closed),HOM
                                                             E=/u01/app/oracle/pr
                                                             oduct/12.1.0.2/db_1,
                                                             STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orapdr01                 STABLE
--------------------------------------------------------------------------------
[oracle@orapdr01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       orapdr01                 STABLE
ora.ons
               OFFLINE OFFLINE      orapdr01                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                               STABLE
ora.dbproddr.db
      1        ONLINE  INTERMEDIATE orapdr01                 Mounted (Closed),HOM
                                                             E=/u01/app/oracle/pr
                                                             oduct/12.1.0.2/db_1,
                                                             STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orapdr01                 STABLE
--------------------------------------------------------------------------------
[oracle@orapdr01 ~]$ crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'orapdr01'
CRS-2673: Attempting to stop 'ora.dbproddr.db' on 'orapdr01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orapdr01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orapdr01' succeeded
CRS-2677: Stop of 'ora.dbproddr.db' on 'orapdr01' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'orapdr01'
CRS-2677: Stop of 'ora.evmd' on 'orapdr01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'orapdr01' has completed
CRS-4133: Oracle High Availability Services has been stopped.



reboot


[oracle@orapdr01 ~]$ . oraenv
ORACLE_SID = [cdbrac1] ? RESTART
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@orapdr01 ~]$ ls -l $ORACLE_HOME/rdbms/lib/config.o
-rw-r--r--. 1 oracle oinstall 1352 Aug  6 13:45 /u01/app/19/grid/rdbms/lib/config.o
[oracle@orapdr01 ~]$ mv $ORACLE_HOME/rdbms/lib/config.o $ORACLE_HOME/rdbms/lib/config.o_BAK
[oracle@orapdr01 ~]$ ls -l $ORACLE_HOME/rdbms/lib/config.o*
-rw-r--r--. 1 oracle oinstall 1352 Aug  6 13:45 /u01/app/19/grid/rdbms/lib/config.o_BAK
[oracle@orapdr01 ~]$ mv \$ORACLE_HOME/rdbms/lib/config.o_BAK $ORACLE_HOME/rdbms/lib/config.o
mv: cannot stat ‘$ORACLE_HOME/rdbms/lib/config.o_BAK’: No such file or directory
[oracle@orapdr01 ~]$ mv $ORACLE_HOME/rdbms/lib/config.o_BAK $ORACLE_HOME/rdbms/lib/config.o
[oracle@orapdr01 ~]$ su -
Password: 
Last login: Tue Sep 17 12:22:38 CEST 2019 on pts/0
[root@orapdr01 ~]# . oraenv
ORACLE_SID = [root] ? RESTART
The Oracle base has been set to /u01/app/oracle
[root@orapdr01 ~]# $ORACLE_HOME/crs/install/roothas.sh -unlock
Using configuration parameter file: /u01/app/19/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/orapdr01/crsconfig/haunlock__2019-09-17_12-27-54AM.log
2019/09/17 12:28:07 CLSRSC-347: Successfully unlock /u01/app/19/grid
[root@orapdr01 ~]# exit
logout
[oracle@orapdr01 ~]$ ls -l $ORACLE_HOME/rdbms/lib/config.o
-rw-r--r--. 1 oracle oinstall 1352 Aug  6 13:45 /u01/app/19/grid/rdbms/lib/config.o
[oracle@orapdr01 ~]$ mv $ORACLE_HOME/rdbms/lib/config.o $ORACLE_HOME/rdbms/lib/config.o_BAK
[oracle@orapdr01 ~]$ ls -l $ORACLE_HOME/rdbms/lib/config.o*
-rw-r--r--. 1 oracle oinstall 1352 Aug  6 13:45 /u01/app/19/grid/rdbms/lib/config.o_BAK
[oracle@orapdr01 ~]$ relink all
writing relink log to: /u01/app/19/grid/install/relinkActions2019-09-17_12-28-47PM.log
[oracle@orapdr01 ~]$ su -
Password: 
Last login: Tue Sep 17 12:27:23 CEST 2019 on pts/0
[root@orapdr01 ~]# . oraenv
ORACLE_SID = [root] ? RESTART
The Oracle base has been set to /u01/app/oracle
[root@orapdr01 ~]# $ORACLE_HOME/rdbms/install/rootadd_rdbms.sh
[root@orapdr01 ~]# $ORACLE_HOME/crs/install/roothas.sh -postpatch
Using configuration parameter file: /u01/app/19/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/orapdr01/crsconfig/hapatch_2019-09-17_12-30-51AM.log
2019/09/17 12:30:54 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2019/09/17 12:31:18 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
[root@orapdr01 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       orapdr01                 STABLE
ora.ons
               OFFLINE OFFLINE      orapdr01                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                               STABLE
ora.dbproddr.db
      1        ONLINE  INTERMEDIATE orapdr01                 Mounted (Closed),HOM
                                                             E=/u01/app/oracle/pr
                                                             oduct/12.1.0.2/db_1,
                                                             STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orapdr01                 STABLE
--------------------------------------------------------------------------------
[root@orapdr01 ~]# crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.

[root@orapdr01 ~]# cat /etc/oracle-release 
Oracle Linux Server release 7.7
[root@orapdr01 ~]# uname -r
4.14.35-1902.5.2.el7uek.x86_64