Performnace Monitoring Using ITCAM for RT

Introduction:

The information technology (IT) provides services for its end user in the form of facilities and application for the users to perform their day-to-day functions. These facilities and applications are in the form of IT services. They run transaction, perform workflows and execute processes.
With the emerging initiative for IT to perform as a business oriented entity, such as the implementation of IT infrastructure library (ITIL®) initiatives, the needs of having a formally defined Service Level Agreement (SLA) with the end-user groups are developed. The SLA would contain the services specifications that are provided by IT for the end-users.
To meet real-world business commitments, gathering detailed statistics of the transactions in your environment can be critical. This article describes how you can set up end-to-end performance monitoring with IBM Tivoli Composite Application manager for Response Time and see how to turn on Application Response Measurement (ARM) instrumentation for IBM HTTP Server, WebSphere® Application Server (Application Server), and DB2® UDB on Windows®.
Much IBM middleware has been enhanced in the past few years to exploit the Application Response Measurement (ARM) open standard as a way to provide internal transaction statistics to any external tool supporting this standard. Application Server (Express, Base, ND, XD) and DB2 UDB have been ARM instrumented, as well as the WebSphere plug-in for IBM HTTP Server (IHS) and Microsoft™ Internet Information Server (IIS). You can now monitor your Web application's performance from end to end if you're using this middleware. However, ARM instrumentation is disabled by default. Before starting to monitor with ITCAM RT we have to enable ARM in these applications.
You need an ITCAM for RT server, RT’s Client Application Time agent running on each server to collect ARM statistics. For detailed view its better to have a Dashboard Agent installed.

In ITCAM RT V 6.2 Release the ARM processing has been simplified and performance enhanced. Some of the main features are
· Shared memory no longer used
· Transaction records are written asynchronously directly to disk with minimal processing for later analysis


Prerequisites:
IBM Tivoli Monitoring V 6.1 FP 5 and above
IBM Tivoli Composite Application Manager for Response Time V 6.2
IBM HTTP Server 6.0
IBM WebSphere Application Server 6.0.2
IBM Db2 Universal Database 8.2

IBM Tivoli Monitoring v 6.1 fp 5 or above and ITCAM for RT server components should be installed. Once this in place you can deploy ITCAM RT, CAT agents on all the servers. RT agent installation is not covered in this article.








Enabling ARM instrumentation

Enable ARM Instrumentation in WebSphere
Enabling ARM in Application Server is a two-step process. First, you have to enable Request Metrics. Then, inside each AppServer setting, you specify where to look for the ARM implementation class and JNI library.

Enable request metrics
Monitoring and Tuning -> Request Metrics
In general properties select Enable request Metrics
Select ALL in components to be instrumented.
In the Trace level select :
i. HOPS – Entry and exit only. Lowest overhead and only gives overall transaction performance.
ii. Performance_DEBUG – Medium overhead and higher detail. Includes Session Beans etc.
iii. DEBUG – Highest overhead and the most detail. Gives Entity bean get/set timings
Set the Request metrics destination to ARM agent
Set the agent type to ARM4
Set the ARM factory to com.ibm.tivoli.transperf.arm4.transaction.Arm40TransactionFactory
2. Now you must add custom properties to the Java™ Virtual Machine (JVM) in each AppServer. Application servers > yourAppServer > Process Definition > Java Virtual Machine > Custom Properties, create the following properties:

a. java.library.path
b. ws.ext.dirs
java.library.path should point to where the libarm4.dll is located and to have ws.ext.dirs point to where the armjni4.jar file is located at.
For ITCAM for RT 6.2 and above – $ITMHOME defaults to c:\ibm\itm for windows and /opt/IBM/ITM/ for unix.
Windows - $ITMHOME\tmaitm6\ for java.library.path and set $ITMHOME\tmaitm6\t4\lib for ws.ext.dirs
Linux - $ITMHOME/li6243/t4/lib/
AIX - $ITMHOME/aix51/t4/lib/
HPUX - $ITMHOME/ hpx1111/t4/lib/
If you want to pass ARM corelator to DB2 (required for DB2 to issue ARM calls ) you should define a custom property
com.ibm.websphere.pmi.reqmetrics.PassCorrelatorToDB to true
Restart WebSphere Application Server
Now WebSphere should be issuing ARM calls, also it can pass ARM corelator to DB2.

Enable ARM instrumentation In IHS
Technically, you're going to enable ARM in the Application Server plug-in running inside IHS, not in IHS itself. After you enable request metrics in WebSphere, you just have to regenerate the plug-in configuration file to enable ARM instrumentation. Look for the following lines in your new plugin-cfg.xml:



Enable ARM instrumentation in DB2 UDB

Beginning with Version 8.2 onwards ARM is enabled by default. But you have to configure DB2, where to look for ARM library files. DB2 registers with ARM if it finds a libarm4.dll file. To configure DB2 lib path to include ARM library issue the following command from db2 command line.

db2set DB2LIBPATH=

For ITCAM RT V 6.2 above libarm4.dll location will be
1. Windows - $ITMHOME\tmaitm6\
2. Linux - $ITMHOME/li6243/t4/lib/
3. AIX - $ITMHOME/aix51/t4/lib/
4. HPUX - $ITMHOME/ hpx1111/t4/lib/ Where $ITMHOME defaults to C:\IBM\ITM for windows and /opt/IBM/ITM for unix

Viewing transaction statistics using Tivoli Enterprise Portal

Open the TEP console and expand the Client Response Time agent. You can see CRT specific work Spaces.

Below is the screen shot of Application workspace. This will give you over all performance details of an application like IHS. You can easily measure the parameters like percent failed, percent slow, percent Available, Average response time etc. Since here we are using ARM we won’t be able to track Network time and Server Time, only Client time can be measured.



Transaction work Space give detailed information of each transaction. Check the below screenshot.


Troubleshooting

No ARM data in the TEP workspace

Verify that the application is properly configured to generate ARM data.

Check the /tmaitm6/arm/log/cat for *.dat files. ARM writes these .dat files when it is collecting data. Make a copy of the *.dat file and view the copy to see what data is collected.

Check the Agent Configuration workspace (right-click Client Response Time -> Workspaces -> Agent Configuration) to make sure the application and transaction patterns are correct. The application and transaction patterns can be modified in the CRT_Client_Transactions situation.

Enable ARM debug

Goto the $ITMHOME/tmaitm6/arm/log/cat directory do one of the following:
Create a file named debug_all.txt - This turns on DLL level tracing for all processes.
Create a file named debug_.txt – This turns on DLL level tracing only for the specified process.
This turns on debugging until you delete the file, and then the tracing turns off
The ARM debug log is written to c:\program files\ibm\tivoli\common\bwm\logs\trace-armdebug.log on Windows and /var/ibm/tivoli/common/bwm/logs/trace-armdebug.log. on unix