Stuck Thread in WebLogic Error and solution

Hi DBA-Mates, Stuck Thread in WebLogic Error and solution

Hope you all are doing your best!!!

We would like to share Stuck Thread in WebLogic Error and work-around one of the important task and frequently getting such below error in WebLogic.

It is very danger also because if it occurs then it will slow down the performance also.

And also we can't do anything that time if it occurs in PROD in middle of Business hours except to wait it to clear.

Another option is to bounce the clear the Thread. So, it is very critical as well as very difficult situation also.

Error details from log:

####<Dec 6, 2019 1:29:51 PM IST> <Info> <WebLogicServer> <ora-data> <SKU_Managed_Server> <[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <a7480fd28a732992:-912e7549:f33b68edc18:-7fec-0000000000000530> <1575656991618> <BEA-000339> <[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>

####<Dec 6, 2019 1:29:51 PM IST> <Info> <WebLogicServer> <ora-data> <SKU_Managed_Server> <[STUCK] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <a7480fd28a732992:-912e7549:f33b68edc18:-7fec-0000000000000531> <1575656991619> <BEA-000339> <[STUCK] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>

####<Dec 6, 2019 1:30:11 PM IST> <Info> <WebLogicServer> <ora-data> <SKU_Managed_Server> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <a7480fd28a732992:-912e7549:f33b68edc18:-7fec-0000000000000532> <1575657011586> <BEA-000339> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>

####<Dec 6, 2019 1:30:15 PM IST> <Info> <WebLogicServer> <ora-data> <SKU_Managed_Server> <[STUCK] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <a7480fd28a732992:-912e7549:f33b68edc18:-7fec-0000000000000533> <1575657015170> <BEA-000339> <[STUCK] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>

Work-around:

If any stuck thread comes then we can check the below things:

1. First check the Managed Server log file for which got alert.

ex: /ora-data/fmw/spl/SID/splapp/servers/Managed_Server/logs/Managed_Server.log

Here, sometimes stuck thread automatically will get clear.

2. Second, check for the status of the Managed Server.

3. Third, Check the Performance and Thread of the Managed Server.

i.e. Click on Environment --> Servers --> Managed Server --> Monitoring --> Performance.

Here check for “Heap Free Percent” (It should be more percentage free)

Also, check for count of the “Hogging Thread Count” which should be either 0 or very less.

i.e. Click on Environment --> Servers --> Managed Server --> Monitoring --> Threads.

If all above looks all fine and still Stuck Thread is not clearing then last option is to bounce the Managed Servers.

 

Some more useful links:

Why we enable Maintenance Mode in R12.1.3

R12.2 Cloning Steps

How to Duplicate a database using RMAN backup

Oracle RMAN Interview Questions and Answers

How to create weblogic user

Regards,

How to create weblogic user

Dear DBA-mates, Hope you all are doing well!!!

How to create Weblogic user is a Weblogic administrative task. 

We are not sure if an Oracle Apps DBA also do it because we have never been done it as Oracle Apps DBA.

But it is important and easy task which we should know it. It may help in either in interview or work environment.

So, let’s start the steps for creating user in Weblogic:

1. Login to weblogic console url with Weblogic user and password.

2. Then click on "Security Realms" under Domain Structure, just below Services.

3. Then click on "myrealm".

4. After that click on "Users and Groups" tab (i.e 2^nd tab). Then select "User" tab (just below 1^st tab).

5. Then click on "New".

6. Then we will see "Create New user" form where you have to fill-up below details:

1. Name: Required Field

2. Description:

3. Provider:

4. Password: Required Field

5. Confirm Password: Required Field

Note: Name, password and Confirm password are Mandatory/Required fields which we need to provide value.

7. Now click on "Ok".

Please find the below path in shot-cut which may more helpful:

Home > Summary of Security Realms > myrealm > Users and Groups >New > Create New User

Also you can create here user, groups, members etc…

Note:

We have a complain about you guys that you people don’t answer the question’s here… hahahah, hmmm everyone wants ready meals: P

Hope these steps are clear and helpful. Please let us know for any concerns either to send email or in our comment box.

Some more useful links: 

What is PDB and CDB in oracle 12c

Why We are not Rich

R12.2 Adop Phases in Oracle EBS

Real Time Oracle Apps DBA Interview Questions

How to Restore DR database from RMAN

Regards,

Realtime Oracle Apps DBA Interview Questions and Answers

Dear DBA-Mates, Latest Apps DBA Interview Questions and Answers.

Hope you all are doing Good!!! We have some topics but due to busy scheduled not able to present here.

But back with some New Questions and Answers which may useful here, also I found as these are simple and good questions.

Questions:

Q 1. Please describe about your job and task activity?

Note: As we all know this is very common question and answer… It depends on person to person…

Q 2. How to do troubleshoot oacore?

Q 3. Where is location path of oacore log file?

Q 4. How to find out which user is pointing to which oacore in case we have 5 oacore?

Note: Please answer this questions if anyone aware about it.

Q 5. What is http server and how it works?

Q 6. What is http.conf file and ohs.conf file and where is the location?

Q 7. What is EBS Login flow?

Q 8. How to troubleshoot Long running program if there is no lock, no server load and everything is normal?

Q 9. What is EBS version?

Q 10. What will happened if you run Prepare phase?

Q 11. What is location of Patch log?

Q 12. If we have created custom top, how adop will sync to patch file system?

Q 13. How the prepare phase knows to sync Custom_top to patch file system?

Q 14. If we have applied small patches then do we need to run fs_clone phase?

Q 15. If cutover failed due to some reason, how to troubleshooting?

Q 16. If u ran prepare phase, 20 programs are running, and during cutover phase once completed what will be the status of those programs?

Q 17. How to create Weblogic user?

Q 18. Have you worked on any other technique other than EBS/DB?

Q 19. Have you done patching in Endeca?

Q 20. How ebs patch sync with Endeca?

Q 21. What is the version of RAC?

Q 22. What is flaks in 12c RAC , new feature in RAC?

Q 23. How you worked on RMAN?

Q 24. What is backup strategy?

Q 25. How to find out database wait events?

Note: Answer will be publish very soon.

Some more useful and helpful links:

How to change sysadmin password FNDCPASS in R12.1.3

What is PDB and CDB in oracle 12c

ORA-01110 data file 1: system01.dbf recovery

R12.2 ADOP Patching Phases Finalize and Cutover

Concurrent Manager Status System Hold, Fix Manager Solved...

Regards,

How to change sysadmin password FNDCPASS in R12.1.3

Dear DBA-Mates, Hope you all are doing Great Job!!!

We know this is very simple and easy to change the sysadmin password in R12.1.3 but some people will may have confusion in it.

Because whenever I asked to change sysadmin password for my junior team they will hesitate and afraid. So thought to lets share which may useful here all.

Confusion:

 

Question: Do we need to run Autoconfig?

Answer: No.

Question: Do we need to bounce the Application Services?

Answer: No.

Note:

 

If found any issue then only need to restart the Apache, Forms, Reports, 8.0.6 listener and concurrent manager services.

Question: Do we need to change any others user’s password for this?

Answer: No.

Simply, we can change the password from FNDCPASS utility as shown below:

1. Run the env file.

$ . ./ora-data.blogspot.env

2. Cross check for correct server/host.

$ echo $TWO_TASK

ORA-DATA

$ FNDCPASS apps/<apps_password> 0 Y system/<system_password>  USER SYSADMIN <new_sysadmin_password>

Log filename : L1234567.log

Report filename : O1234567.out

bash-3.2$ cat L1234567.log

+---------------------------------------------------------------------------+

Application Object Library: Version : 12.1

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDCPASS:

+---------------------------------------------------------------------------+

Current system time is 19-OCT-2019 06:52:00

+---------------------------------------------------------------------------+

Arguments

   FNDCPASS system/***** USER SYSADMIN *****

+----------------------------------------------------------------------------+

Working...

Password is changed successfully for user SYSADMIN.

FNDCPASS completed successfully.

+---------------------------------------------------------------------------+

Concurrent request completed successfully

Current system time is 19-OCT-2019 06:52:01

+---------------------------------------------------------------------------+

$ exit

exit

You have new mail in /var/spool/mail/ora-data

$ exit

logout

How to set password in pbrun command in linux:

$pbrun ohsdba password-manager <Database_name> -set sysadmin

What is the current 'sysadmin' password?:<current_sysadmin_password>

Once more:<current_sysadmin_password>

$

Some more useful links:

FRM-92060: Failed to connect to the Server. Bad machine specification: hostname.domain.com:port

What is PDB and CDB in oracle 12c

[Fixed] Weblogic Error: BEA-141281 unable to get file lock, will retry ...

ORA-01110 datafile 1: system01.dbf recovery

Concurrent Manager Performance Tuning Tips in Oracle EBS R12

Regards,

FRM-92060: Failed to connect to the Server. Bad machine specification:

Hi DBA-mates,Hope you all are doing well!!!

FRM-92060: Failed to connect to the Server. Bad machine specification: hostname.domain.com:port

We are back again with one of very important forms error FRM-92060: Failed to connect to the Server which may help you guys here...

We used to face this issue whenever we run the Autoconfig on applications node.

The issue is when we run Autoconfig on Application node and start the services the forms will open with error as "FRM-92060: Failed to connect to the Server. Bad machine specification: hostname.domain.com:port..."

But later on we found that because of running autoconfig on Apps nodes the previous/custom configurations files got updated and made some changes, so the forms won't open.

So, below is the one of the work around or solution which may help us here in future for quick references. 

Error:
FRM-92060: Failed to connect to the Server. 
Bad machine specification: hostname.domain.com:port

Details...
Java Exception:
java.net.UnknownHostException: hostname.domain.com
at java.net.AbstractPlainSocketlmpl.connect(Unknown Source)
at oracle.forms.net.SocketConnection.connect(Unknown Source)
at oracle.forms.engine.Runform.initConnection(Unknown Source)
at oracle.forms.engine.Runform.startRunform(Unknown Source)
at oracle.forms.engine.Main.createRunform(Unknown Source)
at oracle.forms.engine.Main.start(Unknown Source)
at sun.applet.AppletPanel.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Form solution:

=========

$su – applmgr 

Password:****

$cd $ORA_CONFIG_HOME/10.1.2/forms/server

$ls –ltr appsweb.cfg

Changed from

=========

serverPort=1234

serverName=<hostname as per unix>

domainName=<domain name>

To

====

serverPort=1234

serverName=<hostname as per ebs url>

domainName=<domain name>

Friends, this is one of the work around which worked for us.

In above example, almost time we saw that the “serverName” got changed to hostname which shows in UNIX but if we change it as per our ebs url, the forms open fine and worked correctly.

There is one more option if still not working above work-around, you can bounce the apache and oacore which may help in it.

Also, we have shared other issue and solutions related to forms which may help you.

Please go through that like:

FRM-92101: A failure has occurred on the Server. You will need to reestablish your session

Oracle forms opening slowly troubleshooting

oacore not starting up troubleshooting tips

Some more useful and helpful links:

Real Time Oracle Apps DBA Interview Questions

ORA-01110 data file 1: system01.dbf recovery

R12.2 ADOP Patching Phases Finalize and Cutover

Why We are not Rich

Regards,

Ora-data Team