Issue

Reporting and investigating issues related to performance

Summary of Issue

This article lists the initial information that must be collected in order to investigate issues related to performance.

Analysis

Specific answers to the following questions should be provided when reporting performance issues, along with the supporting artifacts listed in the Resolution section of this article:

  • Was the performance ever good?
    • If yes, quantify by providing relevant logs and reports that show how much time it was taking before, and how much time it currently takes.
    • What is the expected or acceptable result?
    • When did it start slowing down?  What was the exact date and time?
  • Were there any related changes to the environment that coincided with the bad performance?
  • What was the sequence of events leading to the problem?
  • Where/how was it noticed?
  • What is the significance or impact of the problem?  For example:
    • Is it affecting all instances (AIS/ActOne), or limited only to some?
    • Is it affecting all transaction types?
    • Is it affecting all work item types?
  • Are additional volumes involved (e.g., transactions, work items, ActOne users)?
  • Is the problem consistent or sporadic?
  • Does the problem occur at specific periods of the day?
  • Are there any problematic use cases?

 

Resolution

Artifacts listed under AIS must be collected if the investigation is related to issues such as solution detection performance, AIS hanging, and AIS system resource utilization.

Artifacts listed under ActOne must be collected if the investigation is related to issues such as UI performance or unresponsiveness.

Artifacts listed under both AIS and ActOne must be collected if the issue involves both components, for example, slowness in distributing work items, calls between ActOne and AIS, refreshing platform lists, and deploying/refreshing policy manager rules.

Database reports must be collected if the issue involves database queries.

Collecting initial AIS information

  • AIS logs with profiler enabled from the relevant instances KB0032350   
  • If hanging is involved, enable the AIS flow trace and collect multiple thread dumps KB0032750 (Linux), KB0032650 (Windows)
  • Enable relevant AIS traces in either aislog.config (regular loggers) or aislog4j.confg (Java-related loggers) 
  • The Actimize Support Tool can be used to collect logs KB0032406
  • If the issue involves JDBC, enable AIS JDBC tracing KB0102924
  • If the issue involves Java
    • Enable Garbage Collection logging and collect the GC log KB0090142
    • If an OutOfMemory error is occurring within the JVM, add the following arguments to "Additional JVM Settings" in ais_config.xml:
      -XX:+HeapDumpOnOutOfMemoryError;-XX:HeapDumpPath=<path>
    • Monitoring Java processes using JMX-compliant tools KB0061806, KB0050081 (over SSL) 
  • If the issue involves a solution maintenance job, the contents of the relevant SQL_FOR_BATCH_LOG table

 

Collecting initial ActOne information

 

Collecting initial DB information

  • SQL Server
    • Utility for extracting SQL Server information KB0077713
    • Gathering SQL Server information KB0032343
    • Checking data distributions in table partitions KB0077831
    • Reduce deadlock errors KB0032734
    • If blocking is currently observed, gather blocking information KB0090131
  • Oracle
    • Oracle reports
      • AWR, ADDM, ASH
      • For specific SQL IDs: AWR SQL, SQL Tuning Advisor, SQL Monitor
    • Utility for extracting Oracle information KB0072573
    • Diagnostic queries for investigating the performance of a specific SQL ID KB0090135
    • Getting the full AWR history report for specific wait events KB0081294

 

Collecting hardware information