What on earth is SCHEMA_EXPORT/STATISTICS/MARKER in datapump?



Recently when doing some datapump work i noticed an odd extra line appearing in 12c when doing schema exports - but only when exporting from a PDB - let me show you what i mean.

So this is the output from a schema export in a plain old (pre 12c style database)

expdp marker/marker reuse_dumpfiles=y directory=tmp

Export: Release 12.1.0.1.0 - Production on Fri Sep 5 21:20:11 2014

Copyright (c) 1982, 2013, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
Starting "MARKER"."SYS_EXPORT_SCHEMA_01":  marker/******** reuse_dumpfiles=y directory=tmp
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 0 KB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/COMMENT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Master table "MARKER"."SYS_EXPORT_SCHEMA_01" successfully loaded/unloaded
******************************************************************************
Dump file set for MARKER.SYS_EXPORT_SCHEMA_01 is:
  /tmp/expdat.dmp
Job "MARKER"."SYS_EXPORT_SCHEMA_01" successfully completed at Fri Sep 5 21:22:43 2014 elapsed 0 00:02:29



When we do the same thing against a PDB we see this (note you have to do an @ connection to datapump from a PDB)

expdp test/test@//localhost:1521/MARKER schemas=test directory=tmp

Export: Release 12.1.0.1.0 - Production on Fri Sep 5 09:15:14 2014

Copyright (c) 1982, 2013, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
Starting "TEST"."SYS_EXPORT_SCHEMA_01":  test/********@//localhost:1521/MARKER schemas=test directory=tmp
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 0 KB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/COMMENT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type SCHEMA_EXPORT/STATISTICS/MARKER
Master table "TEST"."SYS_EXPORT_SCHEMA_01" successfully loaded/unloaded
******************************************************************************
Dump file set for TEST.SYS_EXPORT_SCHEMA_01 is:
  /tmp/expdat.dmp
Job "TEST"."SYS_EXPORT_SCHEMA_01" successfully completed at Fri Sep 5 09:15:45 2014 elapsed 0 00:00:29


So pretty much exactly the same - apart from 1 line

Processing object type SCHEMA_EXPORT/STATISTICS/MARKER

 Sowhat is this mysterious 'marker' - the fact that it is mentioned at all implies it thinks it needs to extract something - but what is that something?

Trying an import to a sqlfile to see what metadata it contained revealed nothing - no mention of anything to do with it.

Hmmmm

Lets try and do a bit of detective work

A look in DBA_EXPORT_OBJECTS makes no reference to this - so there is no comment to help us. So a bit of a dead end there.

Lets trace datapump and se if that tells us anything - activating full tracing and we see this in the trace file

KUPW:10:21:47.728: 1: DBMS_LOB.TRIM returned.
KUPW:10:21:47.728: 1: DBMS_METADATA.FETCH_XML_CLOB called. Handle: 200001
META:10:21:47.728: OBJECTS_FETCHED for TABLE_STATISTICS = 1
META:10:21:47.729:  get_xml_inputs SCHEMA_EXPORT/STATISTICS/MARKER:
SELECT /*+rule*/ SYS_XMLGEN(VALUE(KU$), XMLFORMAT.createFormat2('MARKER_T', '7')), 0 ,KU$.MARKER ,'MARKER' FROM SYS.KU$_MARKER_VIEW KU$
rowtag: ROW objnum_count: 0 callout: 0 Bind count: 0 Bind values:


Aha - some kind of progress - so there is a view KU$_MARKER_VIEW which datapump is using - maybe this contains some useful comments?

A quick look at the source code - and it's very basic......

select '1', '0', dbms_metadata_util.get_marker
  from dual


which when run shows

select * from SYS.KU$_MARKER_VIEW;

V V     MARKER
- - ----------
1 0         42


So lets have a look at the source of this.... (well as much as we can do as the package body is wrapped).

The only bit of any note is this

-- PACKAGE VARIABLES

  marker NUMBER := 42;  -- marker number: used in worker/mcp to take actions
                        -- at appropriate times, without depending on
                        -- pathnames


Which doesn't really tell us anything.

And that's pretty much the end of the road, we all know about 42 of course....... http://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy

So it's either a coincidence that 42 exists or there is some deeper meaning to any of this existing (i suspect the former in this case :-)).

Anyone know what this is actually doing? It's off that it seems to be a 'schema level' thing to export not something associated with segments directly





Comments

  1. I was wondering the same because my exports goes really well then I hit the marker:

    Worker 1 Status:
    Instance ID: 2
    Instance name: proda2
    Host name: pdbhost02
    Process Name: DW00
    State: EXECUTING
    Object Schema: SYS
    Object Type: SCHEMA_EXPORT/STATISTICS/MARKER
    Completed Rows: 59,655
    Completed Bytes: 7,182,304
    Percent Done: 100
    Worker Parallelism: 1

    It just sits there forever. A bit frustrating when it's notifying: 100% completed.

    ReplyDelete
    Replies
    1. Hello Nicolas,

      Did the process ever end? Coz I have mine running for 18hrs now. It's rather frustrating this thing.

      Regards,
      Kenince

      Delete
  2. Oh man, nothing to add but that comment cracked me up. Get yourself on a Burleson cruise!

    ReplyDelete
  3. Hi Guys, I am stuck at the same point. Exported from my 12.2.0.1 DB (non CBD) and importing in a non production DB (non CDB). My impdp hung at this MARKER stage only :(
    Oracle says to Re Run the DataPump job with exclude=statistics.

    Please let me know if any better suggestion cause rerunning can take hell lot of time.

    ReplyDelete
  4. I am in a 12.1.0.2 CDB and ran into this. Will try exclude=statistics.
    When is the next DBA cruise, need a vacay!

    ReplyDelete
  5. 1. Upgrade to 12.2.0.1 or higher where this is fixed.

    OR

    2. Download and apply the Generic Platform Patch:20345554.

    OR

    3. A potential workaround will be to exclude statistics during the import using EXCLUDE=STATISTICS in the import command.

    ReplyDelete
  6. hi~~

    (Doc ID 2135702.1) Data pump Import Job Is Hanging On STATISTICS/MARKER

    SYMPTOMS
    =====================
    While performing an import, the job is hanging during the "DATABASE_EXPORT/STATISTICS/MARKER" or "SCHEMA_EXPORT/STATISTICS/MARKER" section

    Review of the SQL that is being executed during this period shows that we are very slow on the following syntax:
    -------------------
    merge into optstat_user_prefs$ d using (select o.obj#, s.c2 pname, s.n1 valnum, s.cl1 valchar, s.d1 chgtime, s.n2 spare1
    from "SYS"."IMPDP_STATS" s, obj$ o, user$ u
    where s.c5= :ownname and s.c1= :tabname
    and s.type = 'P' and SYS_OP_MAP_NONNULL(s.statid) = SYS_OP_MAP_NONNULL(:stid)
    and o.owner#=u.user# and u.name= :ownname and o.type# = 2
    and o.name= :tabname ) s on (d.obj#=s.obj# and d.pname=s.pname)
    when matched then update set valchar=s.valchar when
    NOT matched then insert (obj#, pname, valnum, valchar, chgtime, spare1) values(s.obj#, s.pname, s.valnum, s.valchar, s.chgtime, s.spare1)
    -------------------
    <===== bad recursive sql (= bad internal sql)

    SOLUTION
    =====================
    1. Upgrade to 12.2.0.1 or higher where this is fixed.
    OR
    2. Download and apply the Generic Platform Patch:20345554.
    OR
    3. A potential workaround will be to exclude statistics during the import using EXCLUDE=STATISTICS in the import command.

    ReplyDelete
  7. Lovely blog thanks for sharing

    ReplyDelete

Post a Comment