OIM performance analysis using thread dump

In this article I am going to describe how we can analyse the OIM performance using java thread dump. In many situation we have observed that OIM performance is very poor and response time is quite long. There are many reasons behind this killing slowness and poor server response. Some observed reasons are highlighted below:

1. Re-Try task schedule job is running and queuing huge number of failed tasks.
2. Re-Try failed reconciliation events job running with huge number of failed event.
3. Some event handler is stuck and trying to connect external web service.
4. Some admin role (with huge number of members) taking unexpected time to open and hanged the system etc.

There can many other reason for the poor performance.

In Unix machine, when we run any service (like OIM, SOA etc), a process gets generated. That process ID contains lot of information that we can use to identify reason for poor performance. Me and my colleague Nicola (https://www.linkedin.com/in/nicola-scendoni) together we did some experiment and found a way to detect which processes are causing the issue.

Here I am explaining the steps we performed:

1. first we run ps -ef|grep wls to get the PID of OIM
2. Once we get the PID, we ran below command to get all the threads of OIM.

          top -H -p <PID>

as you can see, the command top displaying all the threads of OIM PID. From these individual threads, identify those which are taking more CPU and Memory. Copy the thread ids into a notepad.

3. Now we need to convert these thread IDs to its corresponding HEX value. In order to convert the decimal PID values to HEX run the below command

                      printf "%x\n" <THREAD ID>

4. Now we will get the thread dump of entire OIM process using JStack

                [JAVA_HOME]/bin/jstack [OIM PID] > /tmp/jstack_dump.txt

5. Open the thread dump /tmp/jstack_dump.txt and search the HEX value of the thread. This will give you the thread which is consuming lots of CPUs and MEMs.

6. Analyse the thread and its classes, and identify the process of OIM which is causing issue. Take the necessary action based on the identified process.


Hope, this blog will help you to identify causes of poor response time of OIM.

Cheers !!!!!!!!!!!!!!!

Workflow Policy Showing NullPointerException - OIM 11gR2 PS3

Some time we have seen in OIM 11gR2 PS3 sysadmin console, that Workflow policy showing NullPointerException. You can see all the workflow policies, but if you click on any of them it will show NullPointerException like below.

The server log will display errors like below


oracle.iam.exception.OIMServiceException: Error occurred while evaluating workflow policies. Could not determine if Modify Role operation requires approvals. Corresponding error message is: null.
        at oracle.iam.request.impl.RequestEngine.evaluateWorkflowPolicies(RequestEngine.java:4595)
        at oracle.iam.request.impl.RequestEngine.performAuthzAndRequestOverrideChecks(RequestEngine.java:4566)
        at oracle.iam.request.impl.RequestEngine.doOperation(RequestEngine.java:4523)
        at oracle.iam.impl.OIMServiceImpl.doOperation(OIMServiceImpl.java:43)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at oracle.iam.platform.utils.DMSMethodInterceptor.invoke(DMSMethodInterceptor.java:35)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy366.doOperation(Unknown Source)
        at oracle.iam.identity.utils.Utils.invokeUnifiedService(Utils.java:3849)


Solution

1. Export the MDS of OIM
[OIM_HOME]/common/bin/wlst.sh
wls:/offline> connect('weblogic','password','t3://weblogic.host:7001')
wls:/OIMDomain/serverConfig>exportMetadata(application='OIMMetadata', server='[OIM_MANAGED_SERVER_NAME]', toLocation=/tmp/MDS');

2. Go to /tmp/MDS

3. Delete metadata/iam-features-reconciliation/entity-definition/RDBMSChildDataProvider.xml file

4. ZIP the MDS zip -r ../mds.zip . >/dev/null

5. Import the MDS again

wls:/OIMDomain/serverConfig>importMetadata(application='OIMMetadata', server='[OIM_MANAGED_SERVER_NAME]', fromLocation='/tmp/mds.zip');

6. Restart the OIM managed servers.