fs_clone failed :[UNEXPECTED]Error occurred running "perl /ora-data/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl

Dear DBA-Mates,

Hope you all are doing well!!!

Little busy with work, so can't get time to meet you guys here...

We have many things to update you guys which we will do it one by one and hope will be useful to you all.

So, let's move to topic...

Recently, we were applying patch and faced the below issue as fs_clone phase failed with error:

ERROR:

[EVENT] Performing CLONE steps.

[PROCEDURE] CLONE Patch File System from Run File System

[PROCEDURE] Calling: /ora-data/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl

[EVENT] Log: /ora-data/applmgr/fs_ne/EBSapps/log/adop/209/date_time/fs_clone/SKU

[UNEXPECTED]Error occurred running "perl /ora-data/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl

-contextfile=/ora-data/inst/fs1/inst/apps/ora-data_SKU/appl/admin/ora-data_SKU.xml

-patchcontextfile=/ora-data/inst/fs2/inst/apps/ora-data_SKU/appl/admin/ora-data_SKU.xml

-promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=209 -timestamp=date_time

-outdir=/ora-data/applmgr/fs_ne/EBSapps/log/adop/209/date_time/fs_clone/SKU"

[UNEXPECTED]occurred during CLONE Patch File System from Run File System,

running command: "perl /ora-data/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl

-contextfile=/ora-data/inst/fs1/inst/apps/ora-data_SKU/appl/admin/ora-data_SKU.xml

-patchcontextfile=/ora-data/inst/fs2/inst/apps/ora-data_SKU/appl/admin/ora-data_SKU.xml -promptmsg=hide -console=off

-mode=create -sync_mode=copy -sessionid=209 -timestamp=date_time -outdir=/ora-data/applmgr/fs_ne/EBSapps/log/adop/209/date_time/fs_clone/SKU".

[EVENT] Releasing managed server ports.

[PROCEDURE] Releasing ports

[STATEMENT] Calling task: "Releasing ports on SKU"; script: "/ora-data/inst/fs1/inst/apps/ora-data_SKU/admin/install/txkCloneAcquirePort.pl"; args: " -option=close -server=SKU.oracleoutsourcing.com -ports=5557,7002,17002,7602,7402,7202,7204,7802,6101,6201,10000 -log=/ora-data/applmgr/fs_ne/EBSapps/log/adop/209/date_time/fs_clone/SKU/txkCloneAcquirePort.log"

[PROCEDURE] Releasing ports on SKU

[PROCEDURE] Calling: /ora-data/inst/fs1/inst/apps/ora-data_SKU/admin/install/txkCloneAcquirePort.pl

[EVENT] Log: /ora-data/applmgr/fs_ne/EBSapps/log/adop/209/date_time/fs_clone/SKU/txkCloneAcquirePort.log

[PROCEDURE] Releasing ports on SKU

[PROCEDURE] Releasing ports

[PROCEDURE] Updating status of patches

[STATEMENT] SQL statement : " update ad_adop_session_patches

set status='F' where applied_file_system_base = '/ora-data/applmgr/fs1' and patch_file_system_base = '/ora-data/applmgr/fs2' and bug_number='CLONE' and status <> 'Y' and appltop_id=197 and node_name='SKU'"

[PROCEDURE] Updating status of patches

[EVENT] Stopping services on patch file system.

[EVENT] Stopping admin server.

[PROCEDURE] Check and stop patch Admin Server

[PROCEDURE] Check and Stop patch FS Admin Server

[UNEXPECTED]Error occurred running "sh /ora-data/inst/fs2/inst/apps/ora-data_SKU/admin/scripts/adadminsrvctl.sh stop -nopromptmsg"

[UNEXPECTED]Error running adadminsrvctl command

[PROCEDURE] Unlocking sessions table

[STATEMENT] Unlocking ad_adop_sessions table for SKU with wait interval of 60 seconds and number of tries 2

[PROCEDURE] Unlocking sessions table

Then as usual like every DBA, we searched and found that adxmlctx temp file in custom folder which we need to remove it and run fs_clone phase again as option force=yes as shown below:

Action:

As per Doc id for ref.: FSClone fails while running txkADOPPreparePhaseSynchronize.pl (Doc ID 2094689.1)

[ora-data@SKU <RUN> ]$ ls -ltr $AD_TOP/admin/template/custom/adxmlctx.tmp

-rw-r--r-- 1 ora-data ora-data 174898 Jan 26 12:16 /dpea9i/applmgr/fs2/EBSapps/appl/ad/12.0.0/admin/template/custom/adxmlctx.tmp

[ora-data@SKU <RUN> ]$ mv $AD_TOP/admin/template/custom/adxmlctx.tmp $AD_TOP/admin/template/custom/adxmlctx.tmp_bkp10feb2019

[ora-data@SKU <RUN> ]$

Then we need to run again fs_clone phase as:

$adop phase=fs_clone force=yes

Some important useful links:

How to take backup Apps Tier using gtar command in R12

How to Restore DR database from RMAN

Performance Issue and Solution