Wednesday 28 November 2018

How to take Apps Tier backup using gtar command in R12

Hi DBA-Mates, Topic: Backup Apps Tier using gtar command in R12
We would like to share this command because sometimes we got confused while taking backup for Apps Tier in real time scenario.
The confusion is that current APPS directory should not get zipped, really many times such situation came with our junior.

So, I thought lets prepare a document and share with them as well as you all here.
So, hope this may useful and helpful for future references.

One more thing should be noted that, we can take backups for apps tier when the services are up & running.
So, please don't get confused with it, especially for juniors...

Backup can taken in two ways either screen or nohup command. Both are used for backend running process.

How to use screen command:
1. screen -ls  ## it will give output for list of screen which earlier people has used.

2. screen -l   ## this command is used to start new screen, once you type the this command it means you are in New screen where you can perform your activities.

3. Ctrl+a+d    ##  Always use this command to exit from current screen i.e. Press together Ctrl,a,d otherwise your screen will terminated/closed.

4. screen -r screen_name ## this command is used for re-attached old screen, screen_name you can get from screen -ls command.

Note: Before using old screen_name , make sure there should not be running old command or scripts.

If you want to use nohup command then:
nohup < your command for backup > &

Now, let's see the command:
Syntax:  $gtar -czvf </path/backup_filename_date.tar.gz> <directory which you want to be zipped>

Example: $gtar -czvf /u01/backup/apps_12June2018.tar.gz apps

Explanation:
/u01/backup/apps_12June2018.tar.gz -- file backup for cloning.
apps -- we want backup for this directory.

applmgr@ora-data:/u01/ebs->ps -ef|grep gtar
 applmgr  7366  6943   1 23:56:42 pts/15      0:37 gtar -czvf /u01/backup/apps_12June2018.tar.gz apps
 applmgr  7996  9516   1 00:00:43 pts/6       0:03 gtar -czvf /u01/backup/inst_12June2018.tar.gz inst
 applmgr  8007 21175   0 00:00:59 pts/1       0:00 grep gtar
applmgr@ora-data:/u01/ebs->

SCP command syntax with example:
Syntax:
scp user@host:/<source file path> <destination path>  ## running on destination host

Example:
scp oracle@host:/u01/PROD/backup/PROD_20170719_1105702_1_FULL ./

Note: It will ask for password, please provide username's password like oracle user.

Some more important Points:
Oracle Deadlocks
Regards,

Sunday 18 November 2018

Performance Issue and Solution

Dear DBA-Mates,
Recently, we received an email from users that they are facing performance issues.
They are not able to run SELECT query on a particulat schema's table.
When they run the Query as shown below:
select * from abc where rownum=5;

This query got hanged, neither error nor any output. Users waited for long time but neither output nor error. Even we also tried but same issue.

Note: Actually, I suggested them for GSS in starting only but they want some investigation before GSS that. How to run GSS for table in Oracle we can see below.
Then we Checked the below General Points and thought may helpful but didn’t work…

1. Is there any high CPU or Memory usage by using "top" command?
2. Is there any Blocking session?
3. Is there any Lock session?
4. Checked alert log

After checking all above steps we didn't get any things, everything was fine. There was no High CPU or memory usage, no lock and no blocking sessions.
Then we decided to investigate more... We checked alert logs we found some archive was failed but that was not related to it.

Then we checked the Tablespace and mount point file system spaces. It was also fine.
Now we checked the last analyzed date/time for that schema's table. Then we were surprised.
Then we suggested them that we need to run GSS for that particular Schema's table.
We ran the GSS and issues got resolved.

Before:
SQL>select substr(OWNER,1,8) OWNER
, TABLE_NAME
, to_char(LAST_ANALYZED,'DD-MON-RR HH24:MI:SS') LAST_ANALYZED
from dba_tables
where OWNER = 'ABC' and table_name='XYZ';  2    3    4    5

OWNER                            TABLE_NAME
-------------------------------- ------------------------------
LAST_ANALYZED
---------------------------
ABC                           XYZ

SQL> exec dbms_stats.gather_table_stats('ABC','XYZ',estimate_percent => dbms_stats.auto_sample_size);
PL/SQL procedure successfully completed.
SQL>

After:
SQL> set pages 200
SQL> set lines 200
SQL> /

OWNER                            TABLE_NAME                     LAST_ANALYZED
-------------------------------- ------------------------------ ---------------------------
ABC                                            XYZ               19-NOV-17 18:20:58

Some more useful links:
Regards,

Sunday 11 November 2018

How to increase Java heap size in Hyperion

Hi DBA-Mates, Good news that we got one more new error, hurrayyyy...
Recently, we got an error Exception in thread "main" java.lang.OutOfMemoryError: Java heap space … in Hyperion which we would like share with you all DBA’s…

Actually, one of our colleagues was thinking as increasing Java Heap Size is same in Hyperion and in Oracle EBS, but it completely different which we have shown below.
Login to correct Hyperion server where the Node manager, WebLogic and java are running as per our environment. But you can look accordingly as per your environment.

Here path may vary but the main file name is setHPenv.sh where we need to make changes for java heap size.
[oracle@host ~]$ hostname; id
ora-data.blogspot.com
uid=225(oracle) gid=2259(oinstall) groups=209(dba),2259(oinstall)

[oracle@host ~]$ cd /p02/apps/oracle/product/fin/ora-data/fmw/user_projects/orasystem1/planning/planning1
[oracle@host ~]$ pwd
/p02/apps/oracle/product/fin/ora-data/fmw/user_projects/orasystem1/planning/planning1
[oracle@host ~]$ ls -ltr setHPenv.sh*
-rwxr-x---+ 1 oracle oinstall 3776 Jun 23  2017 setHPenv.sh.512
-rwxr-x---+ 1 oracle oinstall 3777 Jun 23  2017 setHPenv.sh.10192015
-rwxr-x---+ 1 oracle oinstall 3983 Nov  8 02:45 setHPenv.sh.210818_bkp
-rwxr-x---+ 1 oracle oinstall 3984 Nov  8 02:59 setHPenv.sh
[oracle@host ~]$
As below we can see that Java heap size value is 512 MB in backup file and now we can increased to 1024 MB (1GB).

Note: Always take backup of file before editing or making any changes in file either on PROD or Test, otherwise you are good to know what can be happen hhhhhh...
Also, always check for "diff" value after made any changes in file and backup file as shown below

[oracle@esu2e005 planning1]$ diff setHPenv.sh.210818_bkp setHPenv.sh
< HP_JAVA_OPTIONS="${HP_JAVA_OPTIONS} -Xmx512M"
---
> HP_JAVA_OPTIONS="${HP_JAVA_OPTIONS} -Xmx1024M"
[oracle@esu2e005 planning1]$
Error Log details:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
                at java.lang.reflect.Array.newArray(Native Method)
                at java.lang.reflect.Array.newInstance(Array.java:52)
                at oracle.jdbc.driver.BufferCache.get(BufferCache.java:229)
                at oracle.jdbc.driver.PhysicalConnection.getByteBuffer(PhysicalConnection.java:12319)
                at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:1099)
                at oracle.jdbc.driver.OracleStatement.check_row_prefetch_changed(OracleStatement.java:3989)
                at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:599)
                at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:520)
                at com.hyperion.planning.sql.HspSQLImpl.executeQuery(HspSQLImpl.java:228)
                at com.hyperion.planning.sql.HspSQLImpl.executeQuery(HspSQLImpl.java:108)
                at com.hyperion.planning.sql.JDBCCacheLoader.loadObjects(JDBCCacheLoader.java:62)
                at com.hyperion.planning.sql.JDBCCacheLoader.loadObjects(JDBCCacheLoader.java:44)
                at com.hyperion.planning.sql.GenericCache.loadCache(GenericCache.java:383)
                at com.hyperion.planning.sql.GenericCache.getCache(GenericCache.java:150)
                at com.hyperion.planning.sql.GenericCache.getObject(GenericCache.java:294)
Some more useful link:
Regards,

Friday 9 November 2018

How to start MRP process on Dataguard -- standby database

Hi DBA-Mates, Hope you all are doing Great!!!
Usually, we have faced this issue Archive logs not applied on physical standby and this is most important error/issue in our DBA task.

Also, 99% people ask this question in the interviews.
We would like to share our real time scenario here which may useful and helpful to everyone.
Whenever we faced this type of issue, we need to check some important below points:

1. We should check the status of database, which should be mounted status.
SQL> select name, open_mode from v$database;

It may also show in Open (Read-only) mode because it may open for any backup or reporting purpose.

2. Then we need to check the MRP process is running or not as shown below command:
1. Check the MRP process is running or not?
$ ps -ef|grep mrp
$
2. In our case MRP is down. If the MRP process is not running then we need to START the MRP to sync from below command.

As per DBA guidance, we should always cross verify the correct instance and database.

$hostname; id
$echo $ORACLE_SID
PRD02
$sqlplus '/as sysdba'
SQL> alter database recover managed standby database disconnect;
Database altered.
SQL>

Once you start the MRP then archive log will start to sync.
Alert log error details: Providing below alert details , so we can see how the above command is working...

$ cd /p01/app/oracle/diag/rdbms/ora-data/ORA-DATA/trace
$ view alert_ORA-DATA.log
Fri Nov 02 08:41:14 2018
Archived Log entry 1014432 added for thread 1 sequence 608497 rlc 810827692 ID 0x8c1e6f64 dest 2:
RFS[2]: No standby redo logfiles available for thread 1
RFS[2]: Opened log for thread 1 sequence 608499 dbid -1944187801 branch 810827692
Fri Nov 02 12:42:47 2018
alter database recover managed standby database disconnect
Attempt to start background Managed Standby Recovery process (ORA-DATA)
Fri Nov 02 08:42:47 2018
MRP0 started with pid=54, OS id=7948
MRP0: Background Managed Standby Recovery process started (ORA-DATA)
 started logmerger process
Fri Nov 02 08:42:53 2018
Managed Standby Recovery not using Real Time Apply
Fri Nov 02 08:42:53 2018
Reconfiguration started (old inc 4, new inc 6)
List of instances:
 1 2 (myinst: 1)
 Global Resource Directory frozen
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Fri Nov 02 08:42:53 2018
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Fri Nov 02 08:42:53 2018
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Fri Nov 02 08:42:53 2018
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
Parallel Media Recovery started with 32 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Completed: alter database recover managed standby database disconnect
Fri Nov 02 08:42:57 2018
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_1_seq_608453.131634.991106577
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_2_seq_423209.63201.991105597
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_2_seq_423210.119301.991106405
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_1_seq_608454.137049.991106667
Fri Nov 02 08:43:12 2018
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_1_seq_608455.141399.991107219
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_1_seq_608456.122183.991107343
Media Recovery Log +RECO_SCL1/ebsprd02/archivelog/2018_11_02/thread_2_seq_423211.111046.991107213
Fri Nov 02 08:43:24 2018

To Stop the MRP process:
$sqlplus '/as sysdba'
SQL> alter database recover managed standby database cancel;
Database altered.
SQL>

Next – How to check Archive Gap and details…(soon)
Some more useful links:
Regards,