12c and the curious case of the database that wouldn't start

Our 12c rollout continues apace at the moment yesterday one of my colleagues came across this very odd issue

When creating a new database with a very basic parameter set

*.backup_tape_io_slaves=TRUE
*.compatible='12.1.0.1'
*.control_file_record_keep_time=14
*.db_create_file_dest='/oracle/DBNAME/oradata'
*.db_name='DBNAME'
*.db_recovery_file_dest='/oracle/DBNAME/recovery_area'
*.db_recovery_file_dest_size=107374182400
*.db_unique_name='DBNAME'
*.diagnostic_dest='/oracle/admin/DBNAME'
*.filesystemio_options='setall'
*.local_listener='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server.domain)(PORT=1521)))'
*.LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST'
*.nls_date_format='DD-MON-RRRR HH24:MI:SS'
*.nls_language='ENGLISH'
*.nls_territory='UNITED KINGDOM'
*.open_cursors=3000
*.pga_aggregate_target=1G
*.processes=100
*.sec_case_sensitive_logon=FALSE
*.sga_max_size=1G
*.sga_target=1G

A startup resulted in this error

SQL> startup nomount
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORA-03113: end-of-file on communication channel

A look at the alert log showed this

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x80] [PC:0x4D581CF, nztGetCertChain()+47] [flags: 0x0, count: 1]
Errors in file /oracle/admin/DBNAME/diag/rdbms/DBNAME/DBNAME/trace/DBNAME_ora_12401.trc  (incident=60036):
ORA-07445: exception encountered: core dump [nztGetCertChain()+47] [SIGSEGV] [ADDR:0x80] [PC:0x4D581CF] [Address not mapped to object] []
Incident details in: /oracle/admin/DBNAME/diag/rdbms/DBNAME/DBNAME/incident/incdir_60036/DBNAME_ora_12401_i60036.trc

Very odd......



We have loads of other databases on this server and we initially thought that maybe it's some os issue, so we shutdown some databases to free up kernel resources and tried again - but to no avail.

So what's going on?

A look at the error message again and we see the function it's complaining about is this

nztGetCertChain()+47

Now that sounds like certificate related stuff to me - but this database doesn't even exist yet and what part could certificates even play in that?

A random look around the server and I found this directory

/oracle/admin/DBNAME/xdb_wallet

which contained some wallet files - this must surely be the cause?

so i

mv /oracle/admin/DBNAME/xdb_wallet /tmp

and try again


SQL> startup nomount
ORACLE instance started.

Total System Global Area 1068937216 bytes
Fixed Size                  2858752 bytes
Variable Size             293601536 bytes
Database Buffers          767557632 bytes
Redo Buffers                4919296 bytes
SQL>

And now it's happy.....

Somehow the wallets needed for xdb got created here and were messing everything up - we had been doing some stuff with 12.1.0.2 and 12.1.0.1 using this db name and i think somehow this got created/corruped/left behind and it's causing this issue.

Anyway - something to be aware of i guess - bit of a strange one.....