Fundraising tech/Components

Long-term goals for our components

• We have started to push CiviCRM customizations upstream or into CiviCRM extensions.
• CentralNotice and A/B testing should be librarized and ideally supported by dedicated teams.
• Banner impressions parsing is being rewritten from scratch.
• Payments audit scripts should be ported to PHP.
• DonationInterface payments API integration should be librarized and decoupled from MediaWiki.
• DonationInterface UI is being ported to a unified Mustache form.

Drupal and CiviCRM

Software	Description	Source	Documentation
Audit modules (drupal)	Compares automatically-downloaded nightly audit files against the CRM database, and logs discrepancies.	wikimedia/fundraising/crm
CiviCRM	Customer relationship management (CRM) layer for managing donor information.	wikimedia/fundraising/crm/civicrm	Installation | Upgrade | Testing
Communications	Generalized mailing job management and templating. Wraps the sendmail library (currently PHPMailer).	wikimedia/fundraising/crm
Contribution Tracking	(Redundant) applies the contribution_tracking table schema. This should be done using the MediaWiki extension instead.	wikimedia/fundraising/crm	Wikitech:Fundraising
Drupal	CMS framework our stuff runs on, manages users and permissions, provides a hook-based API.	wikimedia/fundraising/crm/drupal	Installation | Upgrade
Drush	Commandline frontend for Drupal	wikimedia/fundraising/drush
Exchange Rates	This module handles converting foreign currencies into US Dollars. It consists of two major components. One function handles doing the conversions based on exchange rates stored in the drupal.exchange_rates table. This function is called from the queue2civicrm module when it is processing donations from the queue. Another function, which is activated by a cron job, looks up the exchange rates on the web and updates the exchange_rates table. The exchange rates are currently pulled from oanda.com, with http://www.ecb.int/stats/eurofxref/eurofxref-daily.xml used as a back-up source.	wikimedia/fundraising/crm
Fredge queue consumer		wikimedia/fundraising/crm
Metrics Reporter	Exposes per-gateway throughput counts and processing time statistics as seen on https://grafana.wikimedia.org/dashboard/db/fundraising-overview.	wikimedia/fundraising/crm
Large Donation	Send an email to the Major Gifts team when any online donation exceeds a configured dollar amount threshold.	wikimedia/fundraising/crm
Offline to CiviCRM	Import check and chargeback spreadsheets into the CRM.	wikimedia/fundraising/crm	Importing donation files
Donations queue consumer	Takes donations added to the queue by IPN listeners or the front-end payments servers and inserts them into CiviCRM	wikimedia/fundraising/crm	Fundraising tech/Message queues
Recurring queue consumer	Same as the donations queue consumer, but for recurring donations.	wikimedia/fundraising/crm	Fundraising tech/Recurring
Recurring Ingenico	Ingenico does not actively recur subscriptions. This module is run daily, and will query the database for recurring donations which should be charged that day.	wikimedia/fundraising/crm	Fundraising tech/Recurring
Thank You	This module sends a thank you message to online donors. There's a job polling for new donations with a null no_thank_you and an empty thankyou_date field. The body of the thank you message is taken from the modules/thank_you/templates directory, and an interface to configure or test mailings can be found in Administration » Configuration » Thank you settings.	wikimedia/fundraising/crm	Generate new Thank You email templates

MediaWiki

Software	Description	Source	Documentation
MediaWiki for donations	This is the version we use to accept donations.	mediawiki/core in Gerrit (fundraising/* branches)]	https://mediawiki.org | Upgrade
Extension:CentralNotice	Display and manage banners	CentralNotice in Gerrit	Extension:CentralNotice
Extension:ContributionTracking	Unified, lifecycle tracking for donations.	ContributionTracking in Gerrit	Extension:ContributionTracking
Extension:DonationInterface	This generates forms and handles payment processing.	DonationInterface in Gerrit	Extension:DonationInterface
Extension:LandingCheck	This handles landing pages for banners.	LandingCheck in Gerrit	Extension:LandingCheck

Other

Software	Description	Source	Documentation
Audit modules (python)	Audit modules built on a Python stack. Sadly, deprecated.	wikimedia/fundraising/tools
Stats parser	Written in Django for some reason, has 3 tasks Loads banner impressions from the sampled beacon files, parsing out banner, campaign, project, country, language, and banner display result (show/hide, hide reason), and rounding off the timestamp to the nearest minute Load landing page impressions from the unsampled donatewiki and foundationwiki logs, selecting certain pages and parsing out the utm_* variables, project, language, and country Count unique email clicks. For landing page impressions with a contact_id,insert utm_source, utm_ampaign, contact_id, and link_id into a table with a unique constraint on those columns, using 'ON DUPLICATE KEY' to discard clicks after the initial one for a given donor and email. For all the tasks, it filters out certain IPs, UAs, and response codes.	wikimedia/fundraising/tools/DjangoBannerStats
Mustache	Recommended templating language for all new components.		TODO: MediaWiki+zordius flavored docs
Twig	Legacy templating language, used in the CRM Thank You module and some minor mailing stuff.
Minor tools	Currently houses: auditing, banner screenshotter, statistical queries (see also "br"), bulk mail	wikimedia/fundraising/tools	Fundraising_tech/tools
SmashPig	Real-time listener framework and nascent payment processing library	wikimedia/fundraising/SmashPig
PHPMailer	Native PHP SMTP client used by some CRM modules.
Maxmind GeoIP	Library we use to resolve IP addresses to a place.
Maxmind Minfraud	Fraud solution we use for front-line defense.

Server applications

Software	Description	Source	Documentation
Apache httpd 2.2	The httpd web server	http://httpd.apache.org/	Configuring httpd
MariaDB	RDBMS backend for CRM, MediaWiki, and tools.		TODO
Redis	Data store for transient or high-availability information
rsyslog	All persistent logging must be done through syslog.
process-control	Schedules fundraising cron jobs and collects output.	wikimedia/fundraising/process-control

See also

A quick pre-campaign checklist.

A selection of essential systems and interactions.