Hot on the heels of the db block corruption i then get this (same server) - not sure it's directly related but could well be...
[oracle@ccserver ~]$ emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Could Not Be Started
Check EM Server log file for details: /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
Oracle Management Server is Down
JVMD Engine is Down
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up
Trawling through the log i find this hidden amongst all the stack trace...
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreFatalException: [Store:280040]An invalid record was found in the store: Code=30.
Great a weblogic error (something i know almost nothing about).
What now? Googling didn't really find anything nor did a direct search on MOS - so i raised an SR. They of course wanted to see every logfile i'd ever looked at in the last 20 years to help them investigate - complete ignoring the fact i had shown them the clear error message relating to the problem! aaargh!
Rather than spend the next hour collating all this i did some more digging looking on various related search terms. I cam across another MOS note that had similar behaviour but not exactly the same - the fix here was to make sure everything was down then remove some .DAT files which seemed to relate to weblogic state. I thought it was worth a try so i did the following (the first part of your path would likely be different)
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/default/_WLS_EMGC_OMS1000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/default/_WLS_EMGC_ADMINSERVER000000.DAT
rm /oracle/gc_inst/user_projects/domains/GCDomain/config/config.lok
After doing that i tried starting again and voila it all worked(though startup did take an absolute age).
I do find it incredibly annoying that support insist on asking for tons of logfiles when almost all of them will be irrelevant - it's like they don't even read when you put until they have every conceivable logfile - then they suddenly say it seems to be 'x' - the same 'x' i told them about at the start of the SR.
Had the exact same problem this morning, this post helped me out, thanks :)
ReplyDelete-Adam vonNieda, Kansas City, USA
Thank you so much for sharing this. Helped.
ReplyDeletethanks, it helped me out..
ReplyDeleteSame problem
ReplyDeleteThanks, it helped me too
Same problem, many thanks!
ReplyDelete