cloud control won't start!



Hot on the heels of the db block corruption i then get this (same server) - not sure it's directly related but could well be...

[oracle@ccserver ~]$ emctl start oms 
Oracle Enterprise Manager Cloud Control 13c Release 1 
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. 
Starting Oracle Management Server... 
WebTier Successfully Started 
Oracle Management Server Could Not Be Started 
Check EM Server log file for details: /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out 
Oracle Management Server is Down 
JVMD Engine is Down 
Starting BI Publisher Server ... 
BI Publisher Server Successfully Started 
BI Publisher Server is Up 

Trawling through the log i find this hidden amongst all the stack trace...

weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreFatalException: [Store:280040]An invalid record was found in the store: Code=30. 

Great a weblogic error (something i know almost nothing about).

What now? Googling didn't really find anything nor did a direct search on MOS - so i raised an SR. They of course wanted to see every logfile i'd ever looked at in the last 20 years to help them investigate - complete ignoring the fact i had shown them the clear error message relating to the problem! aaargh!

Rather than spend the next hour collating all this i did some more digging looking on various related search terms. I cam across another MOS note that had similar behaviour but not exactly the same - the fix here was to make sure everything was down then remove some .DAT files which seemed to relate to weblogic state. I thought it was worth a try so i did the following (the first part of your path would likely be different)

rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/default/_WLS_EMGC_OMS1000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/default/_WLS_EMGC_ADMINSERVER000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/config/config.lok

After doing that i tried starting again and voila it all worked(though startup did take an absolute age).

I do find it incredibly annoying that support insist on asking for tons of logfiles when almost all of them will be irrelevant - it's like they don't even read when you put until they have every conceivable logfile - then they suddenly say it seems to be 'x' - the same 'x' i told them about at the start of the SR.


2 comments:

  1. Had the exact same problem this morning, this post helped me out, thanks :)

    -Adam vonNieda, Kansas City, USA

    ReplyDelete
  2. Thank you so much for sharing this. Helped.

    ReplyDelete