Manual:Fuzz-tester.php

Fuzz-tester.php is a maintenance script that performs fuzz-style testing of MediaWiki's parser and forms by generating lots of nasty wiki text, to ask the Parser to render that wiki text to HTML, or ask MediaWiki's forms to deal with that wiki text, to check MediaWiki's output for problems and to repeat those tasks, to help find bugs and find security issues, or potential security issues.

Requirements

Command-line PHP5, with PHP-curl enabled (not all installations have this enabled, try "apt-get install php5-curl" if you're on Debian to install) the Tidy standalone executable ("apt-get install tidy").

Optional

If you want to run the curl scripts, you'll need standalone curl installed ("apt-get install curl").
For viewing the W3C validator output on a command line, the "html2text" program may be useful ("apt-get install html2text").

Types of problems are being checked for

Unclosed tags
Errors or interesting warnings from Tidy
PHP errors/warnings/notices
MediaWiki internal errors
Very slow responses.
No response from apache.
Optionally checking for malformed HTML using the W3C validator.

Saving tests and test results

Any of the fuzz tests which find problems are saved for later review. In order to help track down problems, tests are saved in a number of different formats. The default filename extensions and their meanings are:

".test.php" : PHP script that reproduces just that one problem using PHP-Curl.
".curl.sh" : Shell script that reproduces that problem using standalone curl.
".data.bin" : The serialized PHP data so that this script can re-run the test.
".info.txt" : A human-readable text file with details of the field contents.

Wiki configuration for testing

You should make some additions to LocalSettings.php in order to catch the most errors. Note this configuration is for **TESTING PURPOSES ONLY**, and is IN NO WAY, SHAPE, OR FORM suitable for deployment on a hostile network. That said, personally I find these additions to be the most helpful for testing purposes:

// --------- Start ---------
// Everyone can do everything. Very useful for testing, yet useless for deployment.
$wgGroupPermissions['*']['autoconfirmed']   = true;
$wgGroupPermissions['*']['block']           = true;
$wgGroupPermissions['*']['bot']             = true;
$wgGroupPermissions['*']['delete']          = true;
$wgGroupPermissions['*']['deletedhistory']  = true;
$wgGroupPermissions['*']['deleterevision']  = true;
$wgGroupPermissions['*']['editinterface']   = true;
$wgGroupPermissions['*']['hiderevision']    = true;
$wgGroupPermissions['*']['import']          = true;
$wgGroupPermissions['*']['importupload']    = true;
$wgGroupPermissions['*']['minoredit']       = true;
$wgGroupPermissions['*']['move']            = true;
$wgGroupPermissions['*']['patrol']          = true;
$wgGroupPermissions['*']['protect']         = true;
$wgGroupPermissions['*']['proxyunbannable'] = true;
$wgGroupPermissions['*']['renameuser']      = true;
$wgGroupPermissions['*']['reupload']        = true;
$wgGroupPermissions['*']['reupload-shared'] = true;
$wgGroupPermissions['*']['rollback']        = true;
$wgGroupPermissions['*']['siteadmin']       = true;
$wgGroupPermissions['*']['unwatchedpages']  = true;
$wgGroupPermissions['*']['upload']          = true;
$wgGroupPermissions['*']['userrights']      = true;
$wgGroupPermissions['*']['renameuser']      = true;
$wgGroupPermissions['*']['makebot']         = true;
$wgGroupPermissions['*']['makesysop']       = true;

// Enable weird and wonderful options:
                            // Increase default error reporting level.
error_reporting (E_ALL);    // At a later date could be increased to E_ALL | E_STRICT
$wgBlockOpenProxies = true; // Some block pages require this to be true in order to test.
$wgEnableUploads = true;    // enable uploads.
$wgDBerrorLog = "/root/mediawiki-db-error-log.txt";  // log DB errors, replace with suitable path.
$wgShowSQLErrors = true;    // Show SQL errors (instead of saying the query was hidden).
$wgShowExceptionDetails = true;  // want backtraces.
$wgEnableAPI = true;        // enable API.
$wgEnableWriteAPI = true;   // enable API.

// Install & enable Parser Hook extensions to increase code coverage. E.g.:
require_once("extensions/ParserFunctions/ParserFunctions.php");
require_once("extensions/Cite/Cite.php");
require_once("extensions/inputbox/inputbox.php");
require_once("extensions/Sort/Sort.php");
require_once("extensions/wikihiero/wikihiero.php");
require_once("extensions/CharInsert/CharInsert.php");
require_once("extensions/FixedImage/FixedImage.php");
// Install & enable Special Page extensions to increase code coverage. E.g.:
require_once("extensions/Cite/SpecialCite.php");
require_once("extensions/Renameuser/SpecialRenameuser.php");
// --------- End ---------

If you want to try E_STRICT error logging, add this to the above:

// --------- Start ---------
error_reporting (E_ALL | E_STRICT);
set_error_handler( 'error_handler' );
function error_handler ($type, $message, $file=__FILE__, $line=__LINE__) {
   if ($message == "var: Deprecated. Please use the public/private/protected modifiers") return;
   print "<br />\n<b>Strict Standards:</b> Type: <b>$type</b>:  $message in <b>$file</b> on line <b>$line</b><br />\n";
}
// --------- End ---------

Also add/change this in LocalSettings.php:

// --------- Start ---------
$wgEnableProfileInfo = true;
$wgDBserver = "localhost"; // replace with DB server hostname
// --------- End ---------

Usage

Run with "php fuzz-tester.php". To see the various command-line options, run "php fuzz-tester.php --help". To stop the script, press Ctrl-C.

Console output:

If requested, first any previously failed tests will be rerun.
Then new tests will be generated and run. Any tests that fail will be saved, and a brief message about why they failed will be printed on the console.
The console will show the number of tests run, time run, number of tests failed, number of tests being done per minute, and the name of the current test.

TODO:

Some known things that could improve this script:

Logging in with cookie jar storage needed for some tests (as there are some pages that cannot be tested without being logged in, and which are currently untested - e.g. Special:Emailuser, Special:Preferences, adding to Watchist).
Testing of Timeline extension (I cannot test as ploticus has/had issues on my architecture).

Option/Parameter	Description
--quiet	Hides passed tests, shows only failed tests.
--base-url	URL to a wiki on which to run the tests.
--directory	Full path to directory for storing failed tests. Will be created if it does not exist.
--include-binary	Includes non-alphanumeric characters in the tests.
--w3c-validate	Validates pages using the W3C's web validator. Slow. Currently many pages fail validation.
--user	Login name of a valid user on your test wiki.
--password	Password for the valid user on your test wiki.
--delete-passed-retests	Will delete retests that now pass.
--rerun-failed-tests	Whether to rerun any previously failed tests.
--max-errors	Maximum number of errors to report before exiting. Does not include errors from `--rerun-failed-tests`.
--max-runtime	Maximum runtime, in minutes, to run before exiting. Only applies to new tests, not `--rerun-failed-tests`.
--specific-test	Runs only the specified fuzz test. Only applies to new tests, not `--rerun-failed-tests`.
--keep-passed-tests	Saves all test files, even those that pass.
--help	Show the help message.

Example

If you wanted to fuzz test a nightly MediaWiki checkout using cron for 1 hour, and only wanted to be informed of errors, and did not want to redo previously failed tests, and wanted a maximum of 100 errors, then you could do: php {$_SERVER["SCRIPT_NAME"]} --quiet --max-errors=100 --max-runtime=60.