Adding Message Persistence to WebSphere MQ Application

Why persistence?

The PET lab had a requirement to enhance the data integrity of the book buy processing in the bookstore application. The bookstore application processes a book buy by sending a WebSphere MQ message from WebSphere Application Server nodes to a separate node containing a C program that listens to a WebSphere MQ queue named BOOK.BUY. The C program pulls messages off of the queue, parses them, then reads and updates several DB2 tables using two-phase commit to process a book buy. The C program then sends a message to a reply queue named BOOK.BUY.REPLY on the WebSphere Application Server node acknowledging the buy transaction and giving details. The application also uses WebSphere MQ to browse books from a database containing over 4 million books, obtained from weekly feeds from the Library of Congress. Any messages on the queues were lost when WebSphere MQ was stopped for any reason. A new requirement specified that no messages should be lost during the book buy process when WebSphere MQ is stopped and restarted either manually or because of a system failure. WebSphere MQ can satisfy this requirement through persistent messages by writing the message to disk on the WebSphere MQ log before putting the message on the queue. During a WebSphere MQ restart, any persistent message not on the queue will be recovered from the log. Figure 1 shows the bookstore application flow:

Figure 1. Bookstore application flow

Adding message persistence to an application requires an understanding of WebSphere MQ queue attributes, queue manager log settings, and the programming parameters that set persistence values. You must also understand WebSphere MQ logging, because persistence results in additional message lengths for each message, and each message is written to disk at the transaction commit point.

Queue attributes and programming changes

There is no such thing as a persistent or non-persistent queue. A default persistence queue setting DEFPSIST is defined at the queue level, but persistence is set at the message level. The programs doing the MQPUT call set the message descriptor field Persistence to persistent ornon-persistent, or use the default persistence setting of the queue. If the option is not specified, the default persistence setting for the queue is used. The MQGET operation receives the Persistence field in the message descriptor as either persistent or non-persistent. A queue can contain both persistent and non-persistent messages. If WebSphere MQ fails or is restarted, persistent messages are kept while non-persistent messages in the same queue are lost.

The DEFPSIST value is defined with an alter queue command with a default value of NO. You can change it like this:

runmqsc
alter ql(BOOK.BUY) defpsist(YES)

For testing purposes, we left the BOOK.BUY.REPLY queue on the sender node as DEFPSIST(NO). The Persistence field in the MQ message descriptor MQMD is used to define the message persistence for the MQPUT operation. Values and descriptions from the WebSphere MQ Programming Reference are shown below:

MQPER_PERSISTENT

Message is persistent, which means that the message survives system failures and restarts of the queue manager. Once the message has been put, and the putter's unit of work committed (if the message is put as part of a unit of work), the message is preserved on auxiliary storage. It remains there until the message is removed from the queue, and the getter's unit of work committed (if the message is retrieved as part of a unit of work). When a persistent message is sent to a remote queue, a store-and-forward mechanism is used to hold the message at each queue manager along the route to the destination, until the message is known to have arrived at the next queue manager. Persistent messages cannot be placed on:

• Temporary dynamic queues
• Shared queues that map to a CFSTRUCT object at CFLEVEL(2) or below, or where the CFSTRUCT object is defined as RECOVER(NO). Persistent messages can be placed on permanent dynamic queues and predefined queues.

MQPER_NOT_PERSISTENT

Message is not persistent, which means that the message does not normally survive system failures or restarts of the queue manager, even if an intact copy of the message is found on auxiliary storage during restart of the queue manager. In the special case of shared queues, non-persistent messages do survive restarts of queue managers in the queue-sharing group, but do not survive failures of the coupling facility used to store messages on the shared queues. On VSE/ESA, MQPER_NOT_PERSISTENT is not supported.

MQPER_PERSISTENCE_AS_Q_DEF

Message has default persistence:

If the queue is a cluster queue, the persistence of the message is taken from the DefPersistence attribute defined at the destination queue manager that owns the particular instance of the queue on which the message is placed. Usually, all of the instances of a cluster queue have the same value for the DefPersistence attribute, although this is not required.

The value of DefPersistence is copied into the Persistence field when the message is placed on the destination queue. IfDefPersistence is changed later, messages that have already been placed on the queue are not affected.

If the queue is not a cluster queue, the persistence of the message is taken from the DefPersistence attribute defined at the local queue manager, even if the destination queue manager is remote. If there is more than one definition in the queue-name resolution path, the default persistence is taken from the value of this attribute in the first definition in the path. This could be:

• An alias queue
• A local queue
• A local definition of a remote queue
• A queue-manager alias
• A transmission queue (for example, the DefXmitQName queue)

The value of DefPersistence is copied into the Persistence field when the message is put. If DefPersistence is changed subsequently, messages that have already been put are not affected. On VSE/ESA, this value is not supported.

Both persistent and non-persistent messages can exist on the same queue.

As explained in the WebSphere MQ Programming Reference, when replying to a message, you should normally use the same persistence type as the request message.

For an MQGET call, the value returned is either MQPER_PERSISTENT or MQPER_NOT_PERSISTENT.

This is an output field for the MQGET call, and an input field for the MQPUT and MQPUT1 calls. The initial value of this field isMQPER_PERSISTENCE_AS_Q_DEF.

The MQ constants used for setting the persistence level are defined as:

Constant	Value	Hex Value
MQPER_NOT_PERSISTENT	0	X'00000000'
MQPER_PERSISTENT	1	X'00000001'
MQPER_PERSISTENCE_AS_Q_DEF	2	X'00000002'

In the bookstore application, the programs already used the MQPER_PERSISTENCE_AS_Q_DEF constant when setting the persistence field for theMQPUT operation. At MQOPEN time, the DefPersistence field reflects the queue's DEFPSIST value. If MQPER_PERSISTENCE_AS_Q_DEF is specified on the MQPUT, the DefPersistence field is copied to the message descriptor's Persistence field. Since this was the case we were able to change the DEFPSIST attribute of the destination queue and not make any programming changes. The listening program sees thePersistence field as MQPER_PERSISTENT or MQPER_NOT_PERSISTENT and uses the same value in the reply message. In our case, theMQMD Persistence value was received as MQPER_PERSISTENT. The listener program used this field in its reply MQMD structure and sent a persistent message back to the reply queue, even though we left the reply queue DEFPSIST attribute as NO.

WebSphere MQ logging considerations

Persistent messaging adds to message sizes and records all messages to disk for the WebSphere MQ queue manager logs. The log is written before the queue is updated. Persistence adds 750 bytes to each message on the MQPUT call. In a high-volume system, you must manage the logs. We striped our logs over four logical RAID5 ESS LUNs with each RAID5 LUN containing eight disks. Several parameters in the log stanza of the configurations should be changed from their defaults to handle the greater write activity and additional log space. Key parameters to consider in the queue manager's qm.ini file:

Logpath

We set up a symbolic link to have the default log path /var/mqm/log/qmgrname point to a large file system striped over 4 ESS luns.

LogType

Choices are CIRCULAR and LINEAR. Linear is needed for media and forward recovery, which was not part of our test. If you choose linear, you must have a process to manage the movement or deletion of the unneeded logs off your disk. Circular reuses logs that are no longer needed for crash recovery. Circular was sufficient for our test as long as we increased the number of logs and log sizes. You cannot change this parameter after you create the queue manager.

LogPrimaryFiles and LogSecondary Files

Defaults are 3 and 2 respectively. The sum of these two parameters cannot exceed 63. The higher level of logging activity with persistent messaging requires a change to the defaults. As a starting point, we set LogPrimaryFiles to 32 and LogSecondayFiles to 16. You can change these values after creating the queue manager. Stopping and restarting MQ will put these changes into effect. We did notice that no files are allocated at MQ startup. We never did get to the 32 primary log files. WebSphere MQ allocates new log files only when needed.

LogFilePages

The default is 1024 4K pages, or 4 MB on AIX. The maximum value is 16,384 pages, or 64 MB. In a high-volume system, use the maximum. WebSphere MQ manuals recommend a large LogFilePage size with a small number of files over many smaller files. Total file system space must be a little more than this value multiplied by the number of primary and secondary log files. You cannot change these values after creating the queue manager. If the queue manager already exists, you must dropped and recreate it. To specify LogFilePages at create time, usecrtmqm -lf logfilesize QMname.

LogBufferPages

This is in 4K pages and sets the amount of memory for log write buffers. The default of 0 specifies that the queue manager will select the value, usually 64, or 256KB in MQ V5.3. Higher sizes will result in better write throughput, with a maximum value of 512 pages, which you should use in a high-volume transaction system. You can change this values after creating the queue manager by stopping and restarting WebSphere MQ.

Performance tests

The following configuration was used in the persistence tests in the lab environment:

Buy node (1):

• 4 332 3 MHz POWER3 processors, 3 GB RAM
• 2 FC adapters, SAN attached to ESS F20, 8 vpaths
• AIX V5.2
• WebSphere MQ V5.3
• DB2 V8.1

WebSphere Application Server nodes (4):

• 4 375 MHz POWER3 processors, 2 GB RAM
• AIX V5.2
• GPFS V2.3.0.6 file systems
• WebSphere Application Server V5.1.1.4
• WebSphere MQ V5.3

Support nodes:

• IBM HTTP Server node with WebSphere plug-in for load balancing
• AKStress node (Robot)
• WebSphere MQ node for bookstore browsing

Tests were run with a varying number of users simulating the book buys against two to four WebSphere Application Server nodes and one back-end buy node with WebSphere MQ, DB2, and user-written WebSphere MQ listener code. The queue attribute DEFPSIST was flipped betweenYES and NO to compare elapsed transaction times with persistence and non-persistence. The listener program was set up to take the persistence setting of the request message and use it in the reply message. Key measurement points:

• Elapsed time from the WebSphere Application Server node PUT statement to the GET of the reply message
• Number of buy transactions in one hour
• Write rate on the buy node queue manager logs measured by the AIX filemon command
• CPU time of the buy node

Non-persistent vs. persistent message comparisons

DEFPSIST	# WebSphere Nodes	Users/#threads	Buys/hour	Avg elapsed time (ms)	Buy Node writes/sec (KB)	Buy Node CPU %	WebSphere Node CPU%
NO	2	20/5	158656	52	243	40	35/35
YES	2	20/5	112665	148	499	37	30/30
NO	2	40/10	267298	67	393	64	45/46
YES	2	40/10	181476	243	722	55	45/44
NO	2	50/10	294864	89	455	73	55/55
YES	2	50/10	204460	305	772	60	57/57
NO	4	50/10	320571	130	453	75	38/24/30/29
YES	4	50/10	269856	163	949	78	31/39/30/33
NO	4	2 robots at 40/10	390952	52	243	91	33/43/35/27
YES	4	2 robots at 40/10	358889	52	243	97	35/43/52/48

The write rate on the logs roughly doubled on the buy node when running at the lower CPU utilization rates once persistence was used.

Changing the queue attribute DEFPSIST from NO to YES increased average elapsed times from 300% to 400% when running two WebSphere Application Server nodes. The tests with the two WebSphere Application Server nodes also had a 30% decrease in transaction throughput, measured in buys per hour in the table.

Tests with four application server nodes sending messages to the back-end buy node had less of an impact: a 10-25% increase in elapsed times and an 8-16% decrease in transaction throughput, because the four application server nodes put greater CPU stress on the back-end buy node, even without persistence.

Conclusion

This article reviewed the basic steps for adding message persistence to an existing application using WebSphere MQ, and offered a starting point for adding message persistence to any size application. Topics included queue administration, logging considerations, and a programming overview for message persistence.