Overview

It was noted that the response time from the WLF solution in RT is especially slow when processing SWIFT messages of the 7xx “family”. This article describes several options to handle this situation and the DB-Q implementation solution which addressed the issue at the DBS Singapore project.

Background

WLF performance, as any other analytical solution, depends on many factors, including aspects specific to each implementation.

The main factors impacting performance of WLF message filtering are:

• The type of messages scanned 
• The length of the message text 
• Which fields in the message are scanned 
• The size of the lists 
• The type of lists 
• Matching algorithm parameters 
• Filtering rules applied 
• Infrastructure (hardware, software) 

The expected and normal behavior of the system is to have a different processing time based on these factors. In particular, data size (the length of the message scanned, and the size of list), have a dramatic impact on the filtering time. This behavior would be expected on any system that scans a text - the longer the text, the longer the scan takes. From analysis of several cases, there is an almost linear correlation between the length of the message and the processing time. 

MT-7xx messages

In a typical SWIFT implementation, all the messages that take significantly longer to process are very long MT 7XX messages (Documentary Credits & Guarantees). The reason these 7XX messages take longer to process is because of the unusually long message length (5-10K characters). Scanning those messages is like scanning every word in a long contract narrative for matches against thousands on names. 

In the extra long messages, the long texts are in the following tags:

·         45 - Description of Goods and/or Services

·         46 - Documents Required

·         47 - Additional Conditions for the documentary credit

·         71 - Charges

·         78 - Instructions to the Paying/Accepting/Negotiating Bank

Possible Solutions

The main challenge is ensuring reasonable performance to the WLF RT solution when processing MT-7xx messages. Several solutions are possible; most of them involve a change to the standard implementation architecture:

1. Since the MT 7XX messages are not money transfers, only documentary credits & guarantees, compliance can consider filtering those messages in batch instead of RT. Compliance should consider if blocking these transactions is required, since they are only documents, not actual movement of funds or securities, and any related fund transfers would be processed and filtered as a separate transaction.

2. It is possible to set up the solution to ignore the specific tags that are long texts and causing the long processing time. As far as we can see in the sample messages provided, these tags contain only contractual narratives that do not contain names, unless they are already included in other fields.

3. It is possible to adjust the filtering algorithm to reduce the run time, but this will probably effect the matching results. In order to do this we would need access to the real data and lists, since it is not possible to tune this on unrealistic sample data.

4. We can recommend upgrades to hardware, which should improve performance.

5. The client can release those messages automatically on the Integration Layer application side if the response is not returned within 60 seconds, and the filtering will continue after the message is released.

6. The client can accept that less then 1% of the messages may longer to scan.

In any of these options, we cannot guarantee an absolute upper limit to the run time that is not dependent on message size.

So even if the run time is reduced for these messages, if we get a message that is much bigger, it may go over the expectation.

Option 5 is the only way to guarantee that no message will be delayed more then a given response time limit (e.g 1 minute), although some messages will be released before the scanning is completed, and alerts would be generated only after the fact.

Suggested Solution: Monitor 7XX messages in a-synchronic mode

A new process is being created to cater for processing long messages splitting the long messages from original flow. This will in turn help all other messages to be processed quickly. This was implemented in the DBS Singapore project (MQ integration).

Before the fix, all messages are being processed sequentially as they are read from the MQ.

When a long message (example: 700 series message) arrives, the processing took longtime causing subsequent messages to be timed out.

The fix is to create a function in ACM where user can define which messages can be screened separately. For reference they are called Excludemsgs.

When Excludemsgs arrive those messages are stored in DB table (db_queue table created in Application DB) and sent a “PEND” response to the IP system.

The other kind of messages which do not fall under Excludemsgs will follow the previous approach of processing and send PEND or PASS response depending upon the Alert generation.

A separate process (start_wlf_messageprocessedlistener) is created to read the ExcludeMsgs from DB table and process them. This process runs every 1 second. If alert is raised then alert is distributed to ACM. If there is no alert generated a final response of “PASS” is sent to IP. Final Response is necessary since a PEND response is sent earlier for ExcludeMsgs.  A final response for Alerted messages will be sent when compliance changes the Alert status to “BLOCK” or “Release”.  After the processing of Excludemsgs, the message is deleted from the DB table.

There can be a retry of up to 10 times for the message to be processed. Each retry will update the num_tries in the DB table. If the max retries (10) are updated the message will not be picked up again. Support is required to investigate the root cause. The application log for the above process will be written in server.log.

Design Notes:

1. Create dbQueue table in app db.
2. Main flow:
   1. When a message type 7XX arrives – save in the db queue table.
   2. Return PENDING
3. New process:
   1. Monitors the dbQueue table and do the load and detect logic:  (look for more details in Apendix)
      1. If there is a hit: it will create alert and will NOT send anything to any MQ.
      2. If there is not hit: is will send RELEASE to the a-synchronic MQ (the one that is being populated during the block-release process)
   2. Additional abilities:
      1. This process should be run with the attribute: ‘Process stops on first error’ disabled. (Meaning – we never want it to fail).
      2. The process have 2 parameters:
         1. Sleep time: indicated how often the flow fetches an entry from the dbQueue table and run the process.
         2. Num of retries: if the process fails it will try again until it reaches the threshold.

Open Issues:

This solution was successfully implemented in an MQ integration, but it needs to be re-designed for an ADK integration.