Fundraising tech/Pending queue consumers
This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on any information on this page. This work is done! Pending messages are now all stored in the pending table in the SmashPig database |
This page exists to document where we read messages from the 'pending' queues, prior to consolidation into a better key-value store.
Pending queues edit
queue2civicrm_import_completion_message edit
This runs inside the queue2civicrm completed donation queue import job. When a message comes in with a 'completion-message-id', the job calls this function to retrieve a message from the pending queue with missing donor details.
This is used for importing messages from Amazon and AstroPay.
Amazon message ID format is 'amazon-' + Amazon's CaptureReferenceId. The CaptureReferenceId is also stored as the gateway_txn_id for Amazon donations.
AstroPay message ID format is 'astropay-' + our order ID, which is the contribution tracking ID, a period, and the API sequence number. The order ID is included in AstroPay IPN messages as x_invoice and in audit files under the Invoice column. We don't have access to the AstroPay-side transaction ID when we enqueue the pending message at the payments frontend.
Adyen IPN message processing jobs edit
ProcessCaptureRequestJob reads from the pending queue, updates the message, and writes back. RecordCaptureJob reads from the pending queue, deletes that message, and adds a message to the completed donation queue.
Message ID format is 'adyen-' + our order ID, which is the ct_id + '.' + sequence number as with AstroPay
Expired message removal edit
This job removes all messages over a certain age from queues.
Reads by date
Paypal IPN edit
Written to and read from by standalone IPN listener, within the same process. Correlation ID is randomly generated by the IPN listener. Looks like it's supposed to hold messages in case of failure, but nothing picks up messages except for the expiry job.
Limbo queues edit
DI frontend edit
Writes to 'limbo' when sending donor to processor, deletes when they come back. Could be used to accept successful donations after sessions expire, but isn't. Also writes to 'globalcollect-cc-limbo' when sending donor to Ingenico. All IDs are gateway name + '-' + order_id.
Orphan rectification job edit
Reads from 'globalcollect-cc-limbo' by date. Using the pending queue for this would mean selecting by date and by gateway name.
Conclusions edit
Some queue usage can simply be removed: nothing useful happens with Paypal IPN messages or with any limbo queue messages besides globalcollect-cc-limbo
The pending queue will need to make messages available by several different selectors: date, gateway name, gateway txn id, and order ID. Adding gateway account would be prudent in case we need multiple accounts per gateway in the future.
Indexes required edit
- idx_pending_date (date) : Expired message removal
- idx_pending_date_gateway (date, gateway) : Orphan rectification, potentially also expired message removal if we wanted gateway-specific expiration dates)
- idx_pending_order_id_gateway (order_id, gateway) : Adyen & AstroPay IPN message processing
- idx_pending_gateway_txn_id_gateway (gateway_txn_id, gateway) : Amazon message processing