Home » Server Options » RAC & Failsafe » Unable to start crs in a 10g RAC (Oracle 10.2.0.3.0 on AIX 5.3)
Unable to start crs in a 10g RAC [message #329757] Thu, 26 June 2008 07:38 Go to next message
divroro12
Messages: 105
Registered: March 2008
Location: Laurel, MD USA
Senior Member
Hi all,

I have a 2 node 10g RAC running on AIX 5.3 with ASM. Recently i noticed the size (originally 33GB) of my /u01 directory growing very fast & reaching 100%. My backups started failing & after some researching found a trace file over 20G in size. I deleted this file & bounced the database. However, when i did a quick check of my database, i found out that the crs was down. For the past two days i've been unable to start the crs. A query of all my RAC resources using both srvctl & crs_stat (shown below) shows that both listener, ASM & database are down on this server. However sqlplus tells me the contrary (shown below).
(oracle@pgoratt1) /home/oracle > srvctl status db -d pegatt 
Instance pegatt1 is not running on node pgoratt1 
Instance pegatt2 is running on node pgoratt2 
(oracle@pgoratt1) /home/oracle > 

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsstat 
HA Resource Target State 
----------- ------ ----- 
ora.pegatt.db ONLINE ONLINE on pgoratt2 
ora.pegatt.pegatt1.inst ONLINE OFFLINE 
ora.pegatt.pegatt2.inst ONLINE ONLINE on pgoratt2 
ora.pgoratt1.ASM1.asm ONLINE OFFLINE 
ora.pgoratt1.LISTENER_PGORATT1.lsnr ONLINE OFFLINE 
ora.pgoratt1.gsd ONLINE OFFLINE 
ora.pgoratt1.ons ONLINE OFFLINE 
ora.pgoratt1.vip ONLINE ONLINE on pgoratt1 
ora.pgoratt2.ASM2.asm ONLINE ONLINE on pgoratt2 
ora.pgoratt2.LISTENER_PGORATT2.lsnr ONLINE ONLINE on pgoratt2 
ora.pgoratt2.gsd ONLINE ONLINE on pgoratt2 
ora.pgoratt2.ons ONLINE ONLINE on pgoratt2 
ora.pgoratt2.vip ONLINE ONLINE on pgoratt2 
(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > 

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsctl start resources 
Starting resources. 
Error while starting resources. Possible cause: CRSD is down. 

SQL> select instance_name,status from gv$instance; 

INSTANCE_NAME STATUS 
---------------- ------------ 
pegatt1 OPEN 
pegatt2 OPEN 


SQL> select instance_name,status from gv$instance; 

INSTANCE_NAME STATUS 
---------------- ------------ 
+ASM1 STARTED 
+ASM2 STARTED 



I'm really confused here because the /u01 directory started increasing in size again, for a reason i can't seem to find since there are no large trace files this time around (increased from 47% to 74% in four days)

I'll be very grateful for any positive input to help me resolve this issue.

Thanks,

[Updated on: Thu, 26 June 2008 09:57] by Moderator

Report message to a moderator

Re: Unable to start crs in a 10g RAC [message #329873 is a reply to message #329757] Thu, 26 June 2008 20:20 Go to previous messageGo to next message
ebrian
Messages: 2794
Registered: April 2006
Senior Member
What was the trace file from?

What does crsctl check crs show?

Have you tried just rebooting the pgoratt1 node?
Re: Unable to start crs in a 10g RAC [message #330177 is a reply to message #329757] Fri, 27 June 2008 15:43 Go to previous messageGo to next message
divroro12
Messages: 105
Registered: March 2008
Location: Laurel, MD USA
Senior Member
The trace file was from /u01/crs/oracle/product/10.2.0/crs/trace.

(oracle@pgoratt1) /home/oracle > crsctl check crs
CSS appears healthy
Cannot communicate with CRS
EVM appears healthy

Thanks,

divroro12
Re: Unable to start crs in a 10g RAC [message #330180 is a reply to message #329757] Fri, 27 June 2008 15:53 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
$CRS_HOME/bin/olsnodes -n
produces what results?
Re: Unable to start crs in a 10g RAC [message #330188 is a reply to message #330177] Fri, 27 June 2008 16:38 Go to previous messageGo to next message
ebrian
Messages: 2794
Registered: April 2006
Senior Member
Well apparently CRS is having some problems if it is generating that type of trace file. I'd suggesting checking the details of that trace file and try to determine what problem CRS is having.
Re: Unable to start crs in a 10g RAC [message #330189 is a reply to message #329757] Fri, 27 June 2008 16:44 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
$CRS_HOME/bin/crs_stat
produces what results?


$CRS_HOME/bin/crsctl check install -wait 600
produces what results?

[Updated on: Fri, 27 June 2008 16:45] by Moderator

Report message to a moderator

Re: Unable to start crs in a 10g RAC [message #330563 is a reply to message #329757] Mon, 30 June 2008 08:08 Go to previous messageGo to next message
divroro12
Messages: 105
Registered: March 2008
Location: Laurel, MD USA
Senior Member
The result of $CRS_HOME/bin/crs_stat is shown above with a formatted output ($CRS_HOME/bin/crsstat).

(oracle@pgoratt1) /home/oracle > crsctl check install -wait 600
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
pgoratt1
pgoratt2
CSS is active on all nodes.


Thanks,

divroro12
Re: Unable to start crs in a 10g RAC [message #331230 is a reply to message #329757] Wed, 02 July 2008 12:49 Go to previous message
divroro12
Messages: 105
Registered: March 2008
Location: Laurel, MD USA
Senior Member
Sorry i forgot to provide this earlier request by anacedent.

(oracle@pgoratt1)/u01/crs/oracle/product/10.2.0/bin > olsnodes -n
pgoratt1 1
pgoratt2 2
Previous Topic: Real Application Clusters (merged)
Next Topic: Driveletters assigned after install?
Goto Forum:
  


Current Time: Thu Mar 28 20:26:41 CDT 2024