In this article I am going to describe how we can analyse the OIM performance using java thread dump. In many situation we have observed that OIM performance is very poor and response time is quite long. There are many reasons behind this killing slowness and poor server response. Some observed reasons are highlighted below:
1. Re-Try task schedule job is running and queuing huge number of failed tasks.
2. Re-Try failed reconciliation events job running with huge number of failed event.
3. Some event handler is stuck and trying to connect external web service.
4. Some admin role (with huge number of members) taking unexpected time to open and hanged the system etc.
There can many other reason for the poor performance.
In Unix machine, when we run any service (like OIM, SOA etc), a process gets generated. That process ID contains lot of information that we can use to identify reason for poor performance. Me and my colleague Nicola (https://www.linkedin.com/in/nicola-scendoni) together we did some experiment and found a way to detect which processes are causing the issue.
Here I am explaining the steps we performed:
1. first we run ps -ef|grep wls to get the PID of OIM
2. Once we get the PID, we ran below command to get all the threads of OIM.
top -H -p <PID>
as you can see, the command top displaying all the threads of OIM PID. From these individual threads, identify those which are taking more CPU and Memory. Copy the thread ids into a notepad.
3. Now we need to convert these thread IDs to its corresponding HEX value. In order to convert the decimal PID values to HEX run the below command
printf "%x\n" <THREAD ID>
4. Now we will get the thread dump of entire OIM process using JStack
[JAVA_HOME]/bin/jstack [OIM PID] > /tmp/jstack_dump.txt
5. Open the thread dump /tmp/jstack_dump.txt and search the HEX value of the thread. This will give you the thread which is consuming lots of CPUs and MEMs.
6. Analyse the thread and its classes, and identify the process of OIM which is causing issue. Take the necessary action based on the identified process.
Hope, this blog will help you to identify causes of poor response time of OIM.
Cheers !!!!!!!!!!!!!!!
1. Re-Try task schedule job is running and queuing huge number of failed tasks.
2. Re-Try failed reconciliation events job running with huge number of failed event.
3. Some event handler is stuck and trying to connect external web service.
4. Some admin role (with huge number of members) taking unexpected time to open and hanged the system etc.
There can many other reason for the poor performance.
In Unix machine, when we run any service (like OIM, SOA etc), a process gets generated. That process ID contains lot of information that we can use to identify reason for poor performance. Me and my colleague Nicola (https://www.linkedin.com/in/nicola-scendoni) together we did some experiment and found a way to detect which processes are causing the issue.
Here I am explaining the steps we performed:
1. first we run ps -ef|grep wls to get the PID of OIM
2. Once we get the PID, we ran below command to get all the threads of OIM.
top -H -p <PID>
3. Now we need to convert these thread IDs to its corresponding HEX value. In order to convert the decimal PID values to HEX run the below command
printf "%x\n" <THREAD ID>
4. Now we will get the thread dump of entire OIM process using JStack
[JAVA_HOME]/bin/jstack [OIM PID] > /tmp/jstack_dump.txt
5. Open the thread dump /tmp/jstack_dump.txt and search the HEX value of the thread. This will give you the thread which is consuming lots of CPUs and MEMs.
6. Analyse the thread and its classes, and identify the process of OIM which is causing issue. Take the necessary action based on the identified process.
Hope, this blog will help you to identify causes of poor response time of OIM.
Cheers !!!!!!!!!!!!!!!
Learned a lot of new things from your post ,It's amazing blog
ReplyDeleteOracle SOA Online Training Bangalore