Cloning using the oracle provided perl utility is seemingly one of the most useful features that oracle provide (and seemingly the most underused - why does no-one use this?).
I've blogged about it before here for the normal database home http://dbaharrison.blogspot.com/2011/02/cloning-oracle-software-why-does-nobody.html
We're currently beginning an ASM rollout across the estate (all single instance - no RAC here) and I don't want to do loads of GUI installs - it's a pain and it involves co-ordination with our service provider to run various commands as root during that gui install - which is easier said than done. I'd rather just get the software on there, get the root commands run as and wen by the supplier and then come back and finish the asm/listener setup later.
So is this possible to do with cloning - thats what i wanted to find out, the docs imply it will work but i couldn;t find a specific example for s standalone instance, and there was no exact post anywhere i found on the net.
So i decided to just try it out and see if it would work, in short it does work and it is pretty simple but it took me a fair while to figure out all the steps and get the syntax right - the following is the abridged version of that journey (when you get to the end you'll be asking was that abridged? - well yes it was is the answer to that)
So here goes:
First up we tar up an existing GI install that we know works ok (it doesn't need to be stopped)
tar -cvf /tmp/grid.tar $GI_HOME
We then copy that tar file to the destination server using whatever tool you see fit scp/sftp etc. We then untar in my case (you could have used zip earlier) that file into the destination directory - i used the same path as the original server but as with the database software this shouldn't matter
tar -xvf /oracle/product/12.1.0/grid.tar
So now we have a home that appears at first glance to be 'valid', however it's not it needs to be registered in the inventory and recompiled - this is where clone.pl comes in - lets now run that - we need to specify 3 parameters when we call it, ORACLE_HOME (self explanatory), ORACLE_BASE (ditto) and ORACLE_HOME_NAME (this is just the 'label' name in the inventory
oracle@server-name:/oracle/product/12.1.0/grid/clone/bin> ./clone.pl ORACLE_HOME=/oracle/product/12.1.0/grid ORACLE_HOME_NAME=GI12c ORACLE_BASE=/oracle/product/12.1.0
oracle@server-name:/oracle/product/12.1.0/grid/clone/bin> ./clone.pl ORACLE_HOME=/oracle/product/12.1.0/grid ORACLE_HOME_NAME=GI12c ORACLE_BASE=/oracle/product/12.1.0
./runInstaller -clone -waitForCompletion "ORACLE_HOME=/oracle/product/12.1.0/grid" "ORACLE_HOME_NAME=GI12c" "ORACLE_BASE=/oracle/product/12.1.0" -silent -paramFile
/oracle/product/12.1.0/grid/clone/clone_oraparam.ini
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 500 MB. Actual 1161 MB Passed
Checking swap space: must be greater than 500 MB. Actual 100295 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-05-13_03-52-49PM. Please wait ...You can find the log of this install session at:
/oracle/oraInventory/logs/cloneActions2014-05-13_03-52-49PM.log
.................................................. 5% Done.
.................................................. 10% Done.
.................................................. 15% Done.
.................................................. 20% Done.
.................................................. 25% Done.
.................................................. 30% Done.
.................................................. 35% Done.
.................................................. 40% Done.
.................................................. 45% Done.
.................................................. 50% Done.
.................................................. 55% Done.
.................................................. 60% Done.
.................................................. 65% Done.
.................................................. 70% Done.
.................................................. 75% Done.
.................................................. 80% Done.
.................................................. 85% Done.
.................................................. 90% Done.
.................................................. 95% Done.
Copy files in progress.
Copy files successful.
Link binaries in progress.
Link binaries successful.
Setup files in progress.
Setup files successful.
Setup Inventory in progress.
Setup Inventory successful.
Finish Setup successful.
The cloning of GI12c was successful.
Please check '/oracle/oraInventory/logs/cloneActions2014-05-13_03-52-49PM.log' for more details.
As a root user, execute the following script(s):
1. /oracle/product/12.1.0/grid/root.sh
.................................................. 100% Done.
oracle@server-name:/oracle/product/12.1.0/grid/clone/bin>
And bang clone done (in less than a minute for me), now we just need to get our friendly unix team to run root.sh shown above along with the standard script to setup GI for the first time - shown below here
/oracle/product/12.1.0/grid/perl/bin/perl -I /oracle/product/12.1.0/grid/perl/lib -I /oracle/product/12.1.0/grid/crs/install /oracle/product/12.1.0/grid/crs/install/roothas.pl
they ran them for me (eventually) and sent me the output
server-name:~ # id
uid=0(root) gid=0(root) groups=0(root)
server-name:~ # /oracle/product/12.1.0/grid/root.sh
Check
/oracle/product/12.1.0/grid/install/root_server-name_2014-05-14_01-23-47.log for
the output of root script
server-name:~ # /oracle/product/12.1.0/grid/perl/bin/perl
-I /oracle/product/12.1.0/grid/perl/lib -I
/oracle/product/12.1.0/grid/crs/install /oracle/product/12.1.0/grid/crs/install/roothas.pl
Using configuration parameter file:
/oracle/product/12.1.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp
'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node server-name successfully pinned.
2014/05/14 01:24:08 CLSRSC-330: Adding Clusterware
entries to file '/etc/inittab'
server 2014/05/14
01:24:35 /oracle/product/12.1.0/grid/cdata/server-name/backup_20140514_012435.olr
2014/05/14 01:24:41 CLSRSC-327: Successfully
configured Oracle Grid Infrastructure for a Standalone Server
/oracle/product/12.1.0/grid/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/12.1.0/grid -defaultHomeName CLUSTER_NODES= CRS=TRUE
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 100287 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
So thats it the software at least is cloned and should be working, easy huh? The next bit took me a little more time to sort out.......
Lets try and run the asmca (again using command line version not GUI here)
I specify everything that should be needed to do this in one pass
asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx -diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL
And we get this output
An ASM Instance with SID +ASM is already detected. ASM failed to start. Check /oracle/product/12.1.0/cfgtoollogs/asmca/asmca-140514AM073949.log for details.
After a quick investigation this was down to the entry i added in the oratab to make setting the environment easy for me - lets comment out +ASM from oratab file as it doesn't exist yet and it's confusing the asmca tool.
Lets try it again
Configuring ASM failed with the following message:
Configuring HA resource failed. The following error occurred:
PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER_+ASM'
. For details refer to "(:CLSN00107:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.asm' on 'server-name' failed
OK - still an obvious message, a listener needs to be there for it to register against and there isn't one (at least not the one in wants anyway)
Lets add a default one on 1521
[oracle@server-name]:GI:[~]# srvctl add listener
[oracle@server-name]:GI:[~]# srvctl start listener
PRCR-1079 : Failed to start resource ora.LISTENER.lsnr
CRS-5016: Process "/oracle/product/12.1.0/grid/bin/lsnrctl" spawned by agent "/oracle/product/12.1.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in
"/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-5016: Process "/oracle/product/12.1.0/grid/bin/lsnrctl" spawned by agent "/oracle/product/12.1.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in
"/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'server-name' failed
Again after investigating the problem was the listener.ora from the cloned home - it contained the old server name in it - causing the issue, lets remove listener.ora from GI home
srvctl start listener
now starts OK (well no errors anyway). Lets check the status of it.
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): server-name
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle 5591 1 0 08:21 ? 00:00:02 /oracle/product/11.2.0.2.0/bin/tnslsnr LISTENER -inherit
oracle 28305 1 0 14:44 ? 00:00:00 /oracle/product/12.1.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle 32420 24977 0 14:45 pts/0 00:00:00 grep lsnr
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
Hmmm, 2 listeners with the same name that shouldn;t be possible should it..... The lsnrctl status command shows the status of the wrong listener. This isn;t going to work out lets get rid of the listener i tried to add and instead just have one listener for everything on the box
Lets stop the GI one and hope it does just stop the GI one.....
srvctl stop listener
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle 5591 1 0 08:21 ? 00:00:02 /oracle/product/11.2.0.2.0/bin/tnslsnr LISTENER -inherit
oracle 15980 24977 0 14:49 pts/0 00:00:00 grep lsnr
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
So it has stopped the correct one ......
Lets point GI at the already existing listener.ora in /etc and use that as the single listener
so we need to add it again and then modify the config. So first up i stop the 11.2.0.2 listener and then start it manually using lsnrctl from the GI home. I then add the listener to restart as previously with srvctl add listener
I then need to amend the environment to tell it to use the listener.ora from/etc
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl getenv listener
LISTENER:
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl setenv listener -t TNS_ADMIN=/etc
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl getenv listener
LISTENER:
TNS_ADMIN=/etc
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
So now that's done, i should be able to 'start' it (though it's already running so it's just a check essentially)
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl start listener
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): server-name
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle 5591 1 0 08:21 ? 00:00:02 /oracle/product/12.1.0/grid/bin/tnslsnr LISTENER -inherit
oracle 15293 24977 0 14:57 pts/0 00:00:00 grep lsnr
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
So that all seems ok, lets do a clean stop and start to confirm everything is OK, the stop works fine but when i start it again this happens:
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl start listener
PRCR-1079 : Failed to start resource ora.LISTENER.lsnr
CRS-5010: Update of configuration file "/etc/listener.ora" failed: details at "(:CLSN00014:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.LISTENER.lsnr' on 'server-name' failed
At first i tried messing around with permissions on the file in /etc but it just wasn't fixing it. So i decided to switch off from using /etc and just use the file from the 'normal' $ORACLE_HOME location - so first we switch off TNS_ADMIN then try again
srvctl setenv listener -t TNS_ADMIN=
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl getenv listener
LISTENER:
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl start listener
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): server-name
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ps -ef |grep lsnr
oracle 24526 1 0 15:07 ? 00:00:00 /oracle/product/12.1.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
oracle 26436 24977 0 15:08 pts/0 00:00:00 grep lsnr
this has now worked ok, switching the location reveals the problem, when the process starts it wants to create a backup file - as /etc is owned by root it can't do it there...... (see example backup file below)
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ls -lrt
total 24
-rw-r--r-- 1 oracle oinstall 3083 May 14 15:06 listener.ora.bak.server-name
-rw-r--r-- 1 oracle oinstall 3221 May 14 15:07 listener.ora
Interestingly though the backup is taken in $ORACLE_HOME but /etc is actually overriding the $ORACLE_HOME anyway - see here.......look at the listener parameter file location....... so at some point we need to get the /etc file removed it will confuse things at some point in the future.
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# lsnrctl stat
LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 14-MAY-2014 15:13:55
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server-name.e-ssi.net)(PORT=1523)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.1.0.1.0 - Production
Start Date 14-MAY-2014 15:07:41
Uptime 0 days 0 hr. 6 min. 13 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /etc/listener.ora
Listener Log File /oracle/product/12.1.0/diag/tnslsnr/server-name/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1523)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1531)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1527)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1524)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1525)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name.e-ssi.net)(PORT=1526)))
Now the listener on this server was actually listening on loads of ports - but none of them was 1521
however the output of this
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl config listener
Name: LISTENER
Home: /oracle/product/12.1.0/grid
End points: TCP:1521
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
shows 1521 - so what is the point of this config?
lsof confirms 1521 not in use
Now at this point i got very confused about this whole listener endpoints thing and the mismatch between the config above and what is actually in the running listener. I ignored that for the moment and blundered on regardless
So now we have a listener running in oracle restart with the correct config to service all of my running databases - now we can use that for ASM. I need to add listener to tnsnames.ora so asmca can find it and i need to tell asmca what its called ( i did at one point try to name it with the default name it seems to want but that caused more pain so i just stuck with listener)
Here we go again then - note the extra parameter at the end
asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx-diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL -param LOCAL_LISTENER=LISTENER
and......
Configuring ASM failed with the following message:
Configuring HA resource failed. The following error occurred:
PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER'
. For details refer to "(:CLSN00107:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.asm' on 'server-name' failed
So it can't fine the name listener in the tnsnames file - the one i put there 2 mins ago...
At this point i realised /etc was likely involved again.....
Sure enough the file exists but is empty
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# cat /etc/tnsnames.ora
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# ls -l /etc/tnsnames.ora
-rw-r--r-- 1 oracle oinstall 0 Dec 19 08:16 /etc/tnsnames.ora
Setting TNS_ADMIN and then i can tnsping it ok
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# export TNS_ADMIN=/oracle/product/12.1.0/grid/network/admin
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# tnsping listener
TNS Ping Utility for Linux: Version 12.1.0.1.0 - Production on 14-MAY-2014 20:45:55
Copyright (c) 1997, 2013, Oracle. All rights reserved.
Used parameter files:
/oracle/product/12.1.0/grid/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1523))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)
(PORT = 1531))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1527))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1524))) (DESCRIPTION
= (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1525))) (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = server-name.e-ssi.net)(PORT = 1526))))
OK (0 msec)
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
So lets set TNS_ADMIN in the environment and try again
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# export TNS_ADMIN=/oracle/product/12.1.0/grid/network/admin
asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx-diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL -param LOCAL_LISTENER=LISTENER
Configuring ASM failed with the following message:
Configuring HA resource failed. The following error occurred:
PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER'
. For details refer to "(:CLSN00107:)" in "/oracle/product/12.1.0/grid/log/server-name/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.asm' on 'server-name' failed
hmm - that didn;t work - lets just put the tnsnames entry in /etc where it wants to find it
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# unset TNS_ADMIN
asmca -silent -configureASM -sysAsmPassword xxxxx -asmsnmpPassword xxxxxx-diskString '/dev/mapper/oracleasm*' -diskGroupName DATA -disk '/dev/mapper/oracleasm-disk1' -redundancy EXTERNAL -diskGroupName FRA -disk '/dev/mapper/oracleasm-disk2' -redundancy EXTERNAL -param LOCAL_LISTENER=LISTENER
ASM created and started successfully.
Disk Group DATA created successfully.
Disk Group FRA created successfully.
Wow - it only worked!
lets see what state everything is in
crsctl stat res -t
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE server-name STABLE
ora.FRA.dg
ONLINE ONLINE server-name STABLE
ora.LISTENER.lsnr
ONLINE INTERMEDIATE server-name Not All Endpoints Re
gistered,STABLE
ora.asm
ONLINE ONLINE server-name Started,STABLE
ora.ons
OFFLINE OFFLINE server-name STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE server-name STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE server-name STABLE
--------------------------------------------------------------------------------
So ASM looks good, but whats going on with the listener (should have checked this earlier) - this must be due to the mismatch between the restart config and the running listener
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl config listener
Name: LISTENER
Home: /oracle/product/12.1.0/grid
End points: TCP:1521
lsnrctl stat
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1523)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1531)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1527)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1524)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1525)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server-name)(PORT=1526)))
So how do we fix - we can use srvctl to update the ports it should be checking the listener is actually on
srvctl modify listener -p "TCP:1523,1531,1527,1524,1525,1526"
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]# srvctl config listener
Name: LISTENER
Home: /oracle/product/12.1.0/grid
End points: TCP:1523,1531,1527,1524,1525,1526
[oracle@server-name]:GI:[/oracle/product/12.1.0/grid/network/admin]#
So thats now updated OK - lets check the status
[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE server-name STABLE
ora.FRA.dg
ONLINE ONLINE server-name STABLE
ora.LISTENER.lsnr
ONLINE INTERMEDIATE server-name Not All Endpoints R e
gistered,STABLE
ora.asm
ONLINE ONLINE server-name Started,STABLE
ora.ons
OFFLINE OFFLINE server-name STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE server-name STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE server-name STABLE
--------------------------------------------------------------------------------
Still not right - lets restart the listener and see what happens
[oracle@server-name]:GI:[~]# srvctl stop listener
[oracle@server-name]:GI:[~]# srvctl start listener
[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE server-name STABLE
ora.FRA.dg
ONLINE ONLINE server-name STABLE
ora.LISTENER.lsnr
ONLINE ONLINE server-name STABLE
ora.asm
ONLINE ONLINE server-name Started,STABLE
ora.ons
OFFLINE OFFLINE server-name STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE server-name STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE server-name STABLE
--------------------------------------------------------------------------------
[oracle@server-name]:GI:[~]#
Brilliant - all now looks ok - lets restart the whole stack and check it comes back ok
[oracle@server-name]:GI:[~]# srvctl stop asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.DATA.dg', but the force option was not specified
[oracle@server-name]:GI:[~]# srvctl stop asm -f
[oracle@server-name]:GI:[~]#
[oracle@server-name]:GI:[~]#
[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE OFFLINE server-name STABLE
ora.FRA.dg
ONLINE OFFLINE server-name STABLE
ora.LISTENER.lsnr
ONLINE ONLINE server-name STABLE
ora.asm
OFFLINE OFFLINE server-name Instance Shutdown,ST
ABLE
ora.ons
OFFLINE OFFLINE server-name STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE server-name STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE server-name STABLE
--------------------------------------------------------------------------------
[oracle@server-name]:GI:[~]# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'server-name'
CRS-2673: Attempting to stop 'ora.evmd' on 'server-name'
CRS-2677: Stop of 'ora.evmd' on 'server-name' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'server-name'
CRS-2677: Stop of 'ora.cssd' on 'server-name' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'server-name' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[oracle@server-name]:GI:[~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[oracle@server-name]:GI:[~]#
[oracle@server-name]:GI:[~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE server-name STABLE
ora.FRA.dg
ONLINE ONLINE server-name STABLE
ora.LISTENER.lsnr
ONLINE ONLINE server-name STABLE
ora.asm
ONLINE ONLINE server-name Started,STABLE
ora.ons
OFFLINE OFFLINE server-name STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE server-name STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE INTERMEDIATE server-name STABLE
--------------------------------------------------------------------------------
So everything but evmd looks ok - lets wait a little longer and afew secs later we get this
ora.evmd
1 ONLINE ONLINE server-name STABLE
--------------------------------------------------------------------------------
so all seems well - i successfully create files in ASM from a database instance on the server
We cloned oracle restart! back slapping all round - job well done and all that.......
And i even sorted out my earlier confusion about the whole endpoints config.
Note to self - sort out /etc......
very clear and detailed.
ReplyDeleteThanks for post , did You copied GI binaries when clusterware stack was up ? AFAIK it is not supported and You have to unlock GI and stop clusterware before .
ReplyDeleteRegards
GG
Hi GG,
ReplyDeleteEverything was done online....
And everything seems fine.....
I guess problems could possible be revealed when we try to patch this...
I'll post again if this all goes horribly wrong :-)
Cheers,
Rich
Thank you for posting CRS-5010 resolution. Helped me over last hurdle with configuring listener on 2nd network (12c feature). I now have clusterware on 10.x.x.x and 2nd listener on 192.x.x.x for testing.
ReplyDeleteNice post thanks
ReplyDeleteNice post thanks
ReplyDeleteNice post very useful! thank you
ReplyDelete