Open main menu

Extension:InterwikiExistence

MediaWiki extensions manual
OOjs UI icon advanced.svg
InterwikiExistence
Release status: experimental
InterwikiExistence.png
Implementation Special page
Description Allows interwiki links to be detected for existence
Author(s) Nathan Larson (Leucostictetalk)
Latest version 1.0.1 (2013-10-18)
MediaWiki 1.22+
Database changes Yes
Tables iwe_page
License GPL
Download See the code section
See the changelog
Parameters
  • $wgInterwikiExistenceNewSeconds
  • $wgInterwikiExistenceDeleteSecond
  • $wgInterwikiExistenceMinimumNewSeconds
  • $wgInterwikiExistenceMinimumDeleteSeconds
  • $wgInterwikiExistenceNewOdds
  • $wgInterwikiExistenceDeleteOdds
  • $wgInterwikiExistenceRemoteWikiUrl
  • $wgInterwikiExistenceApiNewArgs
  • $wgInterwikiExistenceApiDeleteArgs
  • $wgInterwikiExistenceApiAllPagesArgs
  • $wgInterwikiExistenceUserAgent
  • $wgInterwikiExistencePrefix
  • $wgInterwikiExistenceRecursionInProgress
  • $wgInterwikiExistenceLocalLinksGoInterwiki
Hooks used
ArticleSaveComplete
LinkEnd
LoadExtensionSchemaUpdates
Translate the InterwikiExistence extension if it is available at translatewiki.net
Check usage and version matrix.

The InterwikiExistence extension allows you to have existence detecting interwiki links. Normally, wikipedia:foo will always be a bluelink. This will make it a redlink if that page doesn't exist. This extension polls a remote API to get the data necessary for the integration.

Contents

InstallationEdit

Download InterwikiExistenceEdit

  • Download the latest version of InterwikiExistence (once it gets added to Git)
  • Create a folder in the extensions folder named InterwikiExistence
  • Move the files to the extensions/InterwikiExistence/ folder

Install InterwikiExistenceEdit

  • Edit LocalSettings.php in the root of your MediaWiki installation, and add the following line near the bottom:
require_once("$IP/extensions/InterwikiExistence/InterwikiExistence.php");

Create and populate tablesEdit

To create the necessary tables, you'll need to run update.php. Then go to /extensions/InterwikiExistence and run the maintenance script: php populateInterwikiExistencePageTable.php. That'll probably take a few hours to finish downloading all the data from Wikipedia's API:Allpages. If you have to abort midway, don't worry, it'll pick up where it left off when you restart it.

ConfigurationEdit

Change these settings to your liking, making sure that you include them in LocalSettings.php after the require_once line installing the extension.

// Update the interwiki existence table every 900 seconds (15 minutes).
$wgInterwikiExistenceNewSeconds = 900;
$wgInterwikiExistenceDeleteSeconds = 900;
// Absolute minimum seconds between refreshes triggered by special page;
// 'interwikiexistencenoratelimit' users are exempt
$wgInterwikiExistenceMinimumNewSeconds = 60;
$wgInterwikiExistenceMinimumDeleteSeconds = 60;
// x in 360 chance, upon an edit being saved, of checking to see whether it's time to update the
// iwe_page table; if your wiki gets a lot of edits (e.g. more than one every 24 hours), set this
// to less than 360 to reduce database load. E.g. 36 would be a 10% chance. Change to 0 if you
// want to make it never check; e.g. if you only want to manually trigger updates.
$wgInterwikiExistenceNewOdds = 360;
$wgInterwikiExistenceDeleteOdds = 360;
// This is the URL from which to obtain interwiki maps.
$wgInterwikiExistenceRemoteWikiUrl = 'https://en.wikipedia.org/w/api.php';
// Arguments to use when accessing the API to check for page creations
$wgInterwikiExistenceApiNewArgs =
	'?action=query&list=recentchanges&rctype=new&rclimit=500&rcstart=$1&rcdir=newer&format=json';
// Arguments to use when accessing the API to check for page deletions/restorations
$wgInterwikiExistenceApiDeleteArgs =
	'?action=query&list=logevents&letype=delete&lelimit=500&lestart=$1&ledir=newer&format=json';
$wgInterwikiExistenceApiAllPagesArgs =
	'?action=query&list=allpages&aplimit=500&apfrom=$1&format=json';
// The user-agent to use in requests to the wikis from which the interwiki maps are obtained
$wgInterwikiExistenceUserAgent = "User-Agent: $wgSitename's InterwikiExistence. Contact info: URL: "
        . $wgServer . $wgScriptPath . " Email: $wgEmergencyContact";
// This is the interwiki prefix to use for making blue links.
$wgInterwikiExistencePrefix = 'wikipedia';
// This is to prevent infinite loops from occurring
$wgInterwikiExistenceRecursionInProgress = false;
// Set to true to make local links go interwiki
$wgInterwikiExistenceLocalLinksGoInterwiki = false;
// Sysops can manually trigger the update
$wgGroupPermissions['sysop']['interwikiexistence'] = true;
// Bureaucrats can manually trigger the update without being throttled
$wgGroupPermissions['bureaucrat']['interwikiexistencenoratelimit'] = true;

Known issuesEdit

Red-linked interwiki links don't go to the right place; e.g. wikipedia:Adams Chapel will take you to wikipedia:Adams+Chapel. Also, redlinked interwiki links like [[wikipedia:Adams Chapel|foo]] don't display as foo, as they should. If you care to try to figure out what's wrong, you're more than welcome. The problem probably is in InterwikiExistenceHooks::interwikiExistenceLinkEnd().

Also, despite indexing of the iwe_page_title field, insertions to the table still slow populateInterwikiExistencePageTable.php to a crawl. Concurrent database transactions would probably be much faster than these consecutive ones. Have at it, if you want to do it.

There's also some code duplication between InterwikiExistenceHooks::interwikiExistenceUpdate() and PopulateInterwikiExistencePageTable::execute(). Had the database access been thought out better, a function would have been created that could be used not only for API:Recentchanges and API:Logevents but also API:AllPages. Also, someone who knows what they're doing should probably rewrite InterwikiExistenceHooks::interwikiExistenceLinkEnd().

At the moment, the extension is throwing "Notice: Undefined index" errors; the developer is working to resolve this issue.

TODOs:

  • Enable it to import enwiki-latest-all-titles. Only use AllPages if a config setting is switched on. (Unnecessary server load)
  • Enable PopulateInterwikiExistencePageTable to handle other namespaces besides the main one.
  • Enable handling of page restorations (leaction=delete/restore).
  • Implement some sort of algorithm (I thought it might be a divide and conquer algorithm, but now I'm not so sure) that lets the extension figure out where to pick up with the API polls. E.g., suppose you do an enwiki-latest-all-titles import, and don't know the timestamp of when the page listwas exported to that file. Select a date that is the earliest you think it could possibly have been exported, e.g. four months ago. InterwikiExistence should then, using API polls, be able to figure out when the export was done. Hmm, how exactly, though, to get to a high enough degree of certainty without doing a bunch of wasteful API pulls?

ChangelogEdit

  • 1.0.1: Introduced as highly experimental and with known glitches.

UsageEdit

If you want to bring your tables up to date, visit "Special:InterwikiExistence. Otherwise, the extension will occasionally poll the remote wiki's API for new data when pages are saved.

CodeEdit

InterwikiExistence.phpEdit

<?php
/**
 * InterwikiExistence extension by Leucosticte
 * URL: http://www.mediawiki.org/wiki/Extension:InterwikiExistence
 *
 * This program is free software. You can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version. You can also redistribute it and/or
 * modify it under the terms of the Creative Commons Attribution 3.0 license.
 *
 * This extension looks up all the wikilinks on a page that would otherwise be red and compares them
 * to a table of page titles to determine whether they exist on a remote wiki. If so, the wikilink
 * turns blue and links to the page on the remote wiki.
 */


/* Alert the user that this is not a valid entry point to MediaWiki if they try to access the
special pages file directly.*/

if ( !defined( 'MEDIAWIKI' ) ) {
	echo <<<EOT
		To install the InterwikiExistence extension, put the following line in LocalSettings.php:
		require( "extensions/InterwikiExistence/InterwikiExistence.php" );
EOT;
	exit( 1 );
}

$wgExtensionCredits['other'][] = array(
	'path' => __FILE__,
	'name' => 'InterwikiExistence',
	'author' => 'Leucosticte',
	'url' => 'https://www.mediawiki.org/wiki/Extension:InterwikiExistence',
	'descriptionmsg' => 'interwikiexistence-desc',
	'version' => '1.0.1',
);

$dir = dirname( __FILE__ ) . '/';
$wgAutoloadClasses['InterwikiExistenceHooks'] = $dir . 'InterwikiExistence.hooks.php';
$wgAutoloadClasses['SpecialInterwikiExistence'] = $dir . 'SpecialInterwikiExistence.php';
$wgExtensionMessagesFiles['InterwikiExistence'] = $dir . 'InterwikiExistence.i18n.php';
$wgExtensionMessagesFiles['InterwikiExistenceAlias'] = $dir . 'InterwikiExistence.alias.php';
$wgGroupPermissions['InterwikiExistence']['interwikiexistence']    = true;
$wgSpecialPages['InterwikiExistence'] = 'SpecialInterwikiExistence';
define( 'INTERWIKIEXISTENCE_SUCCESS', 1 );
define( 'INTERWIKIEXISTENCE_NOUPDATES', 2 );
define( 'INTERWIKIEXISTENCE_THROTTLED', 3 );
define( 'INTERWIKIEXISTENCE_NODECODE', 4 );
define( 'INTERWIKIEXISTENCE_NORETRIEVE', 5 );
define( 'INTERWIKIEXISTENCE_SKIPPED', 6 );
define( 'INTERWIKIEXISTENCE_NOTIMESTAMP', 7 );
define( 'INTERWIKIEXISTENCE_CONTINUING', 8 );

$wgHooks['LoadExtensionSchemaUpdates'][] = 'InterwikiExistenceHooks::InterwikiExistenceCreateTable';
$wgHooks['LinkEnd'][] = 'InterwikiExistenceHooks::interwikiExistenceLinkEnd';
$wgHooks['ArticleSaveComplete'][] =
	'InterwikiExistenceHooks::interwikiExistenceOnArticleSaveComplete';

// Update the interwiki existence table every 900 seconds (15 minutes).
$wgInterwikiExistenceNewSeconds = 900;
$wgInterwikiExistenceDeleteSeconds = 900;
// Absolute minimum seconds between refreshes triggered by special page;
// 'interwikiexistencenoratelimit' users are exempt
$wgInterwikiExistenceMinimumNewSeconds = 60;
$wgInterwikiExistenceMinimumDeleteSeconds = 60;
// x in 360 chance, upon an edit being saved, of checking to see whether it's time to update the
// iwe_page table; if your wiki gets a lot of edits (e.g. more than one every 24 hours), set this
// to less than 360 to reduce database load. E.g. 36 would be a 10% chance. Change to 0 if you
// want to make it never check; e.g. if you only want to manually trigger updates.
$wgInterwikiExistenceNewOdds = 360;
$wgInterwikiExistenceDeleteOdds = 360;
// This is the URL from which to obtain interwiki maps.
$wgInterwikiExistenceRemoteWikiUrl = 'https://en.wikipedia.org/w/api.php';
// Arguments to use when accessing the API to check for page creations
$wgInterwikiExistenceApiNewArgs =
	'?action=query&list=recentchanges&rctype=new&rclimit=500&rcstart=$1&rcdir=newer&format=json';
// Arguments to use when accessing the API to check for page deletions/restorations
$wgInterwikiExistenceApiDeleteArgs =
	'?action=query&list=logevents&letype=delete&lelimit=500&lestart=$1&ledir=newer&format=json';
$wgInterwikiExistenceApiAllPagesArgs =
	'?action=query&list=allpages&aplimit=500&apfrom=$1&format=json';
// The user-agent to use in requests to the wikis from which the interwiki maps are obtained
$wgInterwikiExistenceUserAgent = "User-Agent: $wgSitename's InterwikiExistence. Contact info: URL: "
        . $wgServer . $wgScriptPath . " Email: $wgEmergencyContact";
// This is the interwiki prefix to use for making blue links.
$wgInterwikiExistencePrefix = 'wikipedia';
// This is to prevent infinite loops from occurring
$wgInterwikiExistenceRecursionInProgress = false;
// Set to true to make local links go interwiki
$wgInterwikiExistenceLocalLinksGoInterwiki = false;
// Sysops can manually trigger the update
$wgGroupPermissions['sysop']['interwikiexistence'] = true;
// Bureaucrats can manually trigger the update without being throttled
$wgGroupPermissions['bureaucrat']['interwikiexistencenoratelimit'] = true;

InterwikiExistence.hooks.phpEdit

<?php
/**
 * InterwikiExistence.hooks.php
 *
 * @author Leucosticte <https://www.mediawiki.org/wiki/User:Leucosticte>
 * @copyright GPL
*/

class InterwikiExistenceHooks {
	public static function InterwikiExistenceCreateTable( $updater = null ) {
		if ( $updater === null ) {
			global $wgExtNewTables;
			$wgExtNewTables[] = array(
				'iwe_page',
				dirname( __FILE__ ) . '/iwetable.sql'
			);
		} else {
			$updater->addExtensionUpdate( array( 'addTable', 'iwe_page',
				dirname( __FILE__ ) . '/iwetable.sql', true ) );
		}
		return true;
	}

	public static function interwikiExistenceLinkEnd( $dummy, Title $target, array $options,
		&$html, array &$attribs, &$ret ) {
		global $wgInterwikiExistencePrefix, $wgInterwikiExistenceIwUrl,
			$wgInterwikiExistenceRecursionInProgress,
			$wgInterwikiExistenceLocalLinksGoInterwiki;
		$dbr = wfGetDB( DB_MASTER );
		$title = $target->getFullText();
		$isInterwiki = false;
		$encodedTitle = $dbr->strencode ( $title );
		#$encodedTitle = $title;
		if ( !$wgInterwikiExistenceRecursionInProgress && $target->getInterwiki ()
			== $wgInterwikiExistencePrefix ) {
			#$encodedTitle = $dbr->strencode ( substr ( $title, strlen (
			$encodedTitle = substr ( $title, strlen (
				$wgInterwikiExistencePrefix ) + 1, strlen ( $title )
				#- $wgInterwikiExistencePrefix ) );
				- $wgInterwikiExistencePrefix );
			$isInterwiki = true;
		}
		$wgInterwikiExistenceRecursionInProgress = false;
		$result = $dbr->selectRow ( 'iwe_page', array ( 'iwe_page_created',
			'iwe_page_deleted' ), array (
				'iwe_page_title' => $encodedTitle ) );
		if ( $result ) {
			if ( $result->iwe_page_created > $result->iwe_page_deleted &&
				( $isInterwiki || $wgInterwikiExistenceLocalLinksGoInterwiki ) ) {
				$wgInterwikiExistenceRecursionInProgress = true;
				if ( !$html ) {
					$html = $title;
				}
				$ret = Linker::link ( Title::newFromText( $wgInterwikiExistencePrefix . ':'
					. $encodedTitle ), $html );
				return false;
				}
		}
		if ( $isInterwiki ) {
			$interwiki = Interwiki::fetch ( $wgInterwikiExistencePrefix );
			$ret = Html::rawElement ( 'a', array ( 'href' => $interwiki->getURL (
				$encodedTitle ), 'class' => 'new' ),
				#$title ), 'class' => 'new' ),
			        $title );
			return false;
		}
		return true;
	}

	public static function interwikiExistenceOnArticleSaveComplete ( &$article, &$user, $text,
		$summary, $minoredit, $watchthis, $sectionanchor, &$flags, $revision, &$status, $baseRevId ) {
		global $wgInterwikiExistenceSeconds, $wgInterwikiExistenceNewOdds,
			$wgInterwikiExistenceDeleteOdds,
			$wgInterwikiExistenceRemoteWikiUrl, $wgInterwikiExistenceApiNewArgs,
			$wgInterwikiExistenceApiDeleteArgs, $wgInterwikiExistenceUserAgent,
			$wgInterwikiExistenceMinimumSeconds, $wgContLang, $wgVersion;
		// Do recentchanges API pull to check for new pages
		InterwikiExistenceHooks::interwikiExistenceUpdate ( $wgInterwikiExistenceNewOdds,
			'interwikiexistencenewtimestamp', 'interwikiexistencenewconttimestamp',
			true, $wgInterwikiExistenceRemoteWikiUrl, $wgInterwikiExistenceApiNewArgs,
			'recentchanges', 'iwe_page_created', 'rccontinue',
			$wgInterwikiExistenceSeconds, $wgInterwikiExistenceMinimumSeconds, true,
			$wgInterwikiExistenceUserAgent );
		// Do logevents API pull to check for deletions/restorations
		InterwikiExistenceHooks::interwikiExistenceUpdate ( $wgInterwikiExistenceDeleteOdds,
			'interwikiexistencedeletetimestamp',
			'interwikiexistencedeleteconttimestamp',
			true, $wgInterwikiExistenceRemoteWikiUrl, $wgInterwikiExistenceApiDeleteArgs,
			'logevents', 'iwe_page_deleted', 'lecontinue',
			$wgInterwikiExistenceSeconds, $wgInterwikiExistenceMinimumSeconds, true,
			$wgInterwikiExistenceUserAgent );
		return true;
	}

	public static function interwikiExistenceUpdate ( $odds, $lastRunProperty,
		$continueProperty, $rollDice, $apiUrl, $apiArgs, $submodule, $iwepField,
		$continueType, $skipSeconds, $throttleSeconds, $checkMinimumTime,
		$userAgent ) {
		// Roll the dice to decide whether to check whether it's time for an update.
		if ( $rollDice ) {
			if ( rand ( 0, 360 ) > $odds ) {
				return INTERWIKIEXISTENCE_SKIPPED;
			}
		}
		$dbw = wfGetDB( DB_MASTER );
		// Last-run timestamp stored in the user_properties table.
		$lastRunResult = $dbw->selectrow( 'user_properties', 'up_value', array (
			'up_user' => '1',
			'up_property' => $lastRunProperty
		) );
		// QueryCont timestamp stored in the user_properties table.
		$continueResult = $dbw->selectrow( 'user_properties', 'up_value', array (
			'up_user' => '1',
			'up_property' => $continueProperty
		) );
		// If no timestamp is in the table, abort; that indicates that update.php hasn't
		// been run yet.
		$timestamp = wfTimestampNow();
		if ( !$lastRunResult || !$continueResult ) {
			return INTERWIKIEXISTENCE_NOTIMESTAMP;
		} else {
			// If it hasn't been a long enough time, don't do the update yet.
			if ( $rollDice && $timestamp - $lastRunResult->up_value
				< $skipSeconds ) {
				return INTERWIKIEXISTENCE_SKIPPED;
			}
			if ( $checkMinimumTime && $timestamp - $lastRunResult->up_value
				< $throttleSeconds ) {
				return INTERWIKIEXISTENCE_THROTTLED;
			}
		}
		// Poll
		$apiUrl = str_replace ( '$1', $continueResult->up_value, $apiUrl . $apiArgs );
		$apiPull = InterwikiExistenceHooks::InterwikiExistenceApiPull ( $apiUrl,
			$userAgent );
		// No data? Bad data?
		if ( isset ( $apiPull['status'] ) ) {
			return $apiPull;
		}
		// See what titles are in the database
		$firstElement = true;
		$cond = '';
		$titles = array();
		foreach ( $apiPull['query'][$submodule] as $apiPullElement ) {
			$cleanTitle = $dbw->strencode ( $apiPullElement['title'] );
                        if ( !$firstElement ) {
                                $cond .= " OR ";
                        }
                        $cond .= "iwe_page_title=" . $dbw->addquotes ( $cleanTitle );
			$firstElement = false;
			$titles[$cleanTitle] =
				InterwikiExistenceHooks::removeUndesirables (
				$apiPullElement['timestamp'] );
			$lastTimestamp = $titles[$cleanTitle];
                }
		$result = $dbw->select( 'iwe_page', array ( 'iwe_page_title', $iwepField ),
			$cond );
		if ( $result ) {
			foreach ( $result as $row ) {
				$timestamp = $titles[$row->iwe_page_title];
				unset ( $titles[ $row->iwe_page_title ] );
				if ( $iwepField = 'iwe_page_created' ) {
					$compare = $row->iwe_page_created;
				} else {
					$compare = $row->iwe_page_deleted;
				}
				if ( $timestamp > $compare ) {
					$dbw->update ( 'iwe_page', array (
						$iwepField => $timestamp ),
						array ( 'iwe_page_title' => $row->iwe_page_title,
							"$timestamp > $iwepField" )
					);
				}
			}
		}
		// Insert rows
		$insertArray = array();
		foreach ( $titles as $title => $timestamp ) {
			if ( $title != 'updated' ) {
				$insertArray[] = array (
					'iwe_page_title' => $title,
					$iwepField => $timestamp
				);
			}
		}
		$dbw->insert( 'iwe_page', $insertArray );

		// Update lastrun timestamp if we've reached the end of the queries
		if ( !isset ( $apiPull['query-continue'] ) ) {
			$upsert = array (
				'up_user' => '1',
				'up_property' => $lastRunProperty,
				'up_value' => InterwikiExistenceHooks::removeUndesirables
					( $timestamp )
			);
			$dbw->upsert(
				'user_properties',
				$upsert,
				array ( 'up_user', 'up_property' ),
				$upsert
			);
			$timestamp = $lastTimestamp;
			$status = true;
		} else {
			$timestamp = InterwikiExistenceHooks::removeUndesirables (
				$apiPull['query-continue'][$continueType] );
			$status = INTERWIKIEXISTENCE_CONTINUING;
		}
		// Either way, update the continue timestamp
		$upsert = array (
				'up_user' => '1',
				'up_property' => $continueProperty,
				'up_value' => $timestamp
			);
		$dbw->upsert(
			'user_properties', $upsert, array ( 'up_user', 'up_property' ),
			$upsert );
		return $status;
	}


	public static function interwikiExistenceApiPull ( $url, $userAgent ) {
		// Set up file_get_contents options
		$opts = array(
			'http'=>array(
				'method' => "GET",
				'header' => $userAgent
			)
		);
		$streamContext = stream_context_create( $opts );
		// Do the API pull
		$contents = file_get_contents ( $url, false, $streamContext );
		if ( !$contents ) {
			wfDebugLog( 'InterwikiExistence', "Retrieval from $url failed\n" );
			return array ( 'url' => $url, 'status' =>
				INTERWIKIEXISTENCE_NORETRIEVE ); // Abort update
		}
		// Decode and return API contents
		$apiPull = json_decode ( $contents, true );
		if ( !$apiPull ) {
			wfDebugLog( 'InterwikiExistence', "json decode of $url failed\n" );
			return array ( 'url' => $url, 'status' =>
				INTERWIKIEXISTENCE_NODECODE ); // Abort update
		}
		return $apiPull;
	}

	public static function removeUndesirables ( $timestamp ) {
		return str_replace ( array ( '-', ':', 'T', 'Z' ), '', $timestamp );
	}
}

SpecialInterwikiExistence.phpEdit

<?php
/**
 * SpecialInterwikiExistence.php
 * This is a special page to poll the API manually.
 *
 * @author Leucosticte <https://www.mediawiki.org/wiki/User:Leucosticte>
 * @copyright GPL
*/

if ( !defined( 'MEDIAWIKI' ) ) {
	die( 'This file is a MediaWiki extension. It is not a valid entry point' );
}

class SpecialInterwikiExistence extends SpecialPage {
	function __construct() {
		parent::__construct( 'InterwikiExistence', 'interwikiexistence' );
	}

	public function userCanExecute( User $user ) {
		return true;
	}

	function execute( $par ) {
                global $wgInterwikiExistenceNewSeconds,
			$wgInterwikiExistenceDeleteSeconds,
			$wgInterwikiExistenceNewOdds,
			$wgInterwikiExistenceDeleteOdds,
			$wgInterwikiExistenceRemoteWikiUrl, $wgInterwikiExistenceApiNewArgs,
			$wgInterwikiExistenceApiDeleteArgs, $wgInterwikiExistenceUserAgent,
			$wgInterwikiExistenceMinimumNewSeconds,
			$wgInterwikiExistenceMinimumDeleteSeconds;
		$user = $this->getUser();
		if ( !$user->isAllowed( 'interwikiexistence' ) ) {
                        throw new PermissionsError( null, array( array(
                                'interwikiexistence-notallowed' ) ) );
		}
		$this->setHeaders();
		$output = $this->getOutput();
                $context = $this->getContext();
                $isNotExempt = !$user->isAllowed( 'interwikiexistencenoratelimit' );
                // Don't roll dice or check time
		SpecialInterwikiExistence::reaction (
			InterwikiExistenceHooks::interwikiExistenceUpdate (
				$wgInterwikiExistenceNewOdds,
				'interwikiexistencenewtimestamp',
				'interwikiexistencenewconttimestamp',
				false, $wgInterwikiExistenceRemoteWikiUrl,
				$wgInterwikiExistenceApiNewArgs,
				'recentchanges', 'iwe_page_created', 'rccontinue',
				$wgInterwikiExistenceNewSeconds,
				$wgInterwikiExistenceMinimumNewSeconds,
				true, $wgInterwikiExistenceUserAgent ), $output,
				'interwikiexistence-creation',
				$wgInterwikiExistenceMinimumNewSeconds );
		SpecialInterwikiExistence::reaction (
			InterwikiExistenceHooks::interwikiExistenceUpdate (
				$wgInterwikiExistenceDeleteOdds,
				'interwikiexistencedeletetimestamp',
				'interwikiexistencedeleteconttimestamp',
				false, $wgInterwikiExistenceRemoteWikiUrl,
				$wgInterwikiExistenceApiDeleteArgs,
				'logevents', 'iwe_page_deleted', 'lecontinue',
				$wgInterwikiExistenceDeleteSeconds,
				$wgInterwikiExistenceMinimumDeleteSeconds,
				true, $wgInterwikiExistenceUserAgent ), $output,
				'interwikiexistence-deletion',
				$wgInterwikiExistenceMinimumDeleteSeconds );
	}

	public static function reaction ( $input, $output, $action, $seconds ) {
		if ( is_array ( $input ) ) {
                        if ( $input['status'] == INTERWIKIEXISTENCE_NORETRIEVE ) {
                                $output->addWikiMsg ( 'interwikiexistence-trigger-noretrieve',
                                        $result['url']);
                        }
                        if ( $input['status'] == INTERWIKIEXISTENCE_NODECODE ) {
                                $output->addWikiMsg ( 'interwikiexistence-trigger-nodecode',
                                        $result['url'] );
                        }
                } else switch ( $input ) {
                        case INTERWIKIEXISTENCE_SUCCESS:
                                $output->addWikiMsg ( 'interwikiexistence-trigger-success',
					wfMessage ( $action ) );
                                break;
			case INTERWIKIEXISTENCE_CONTINUING:
                                $output->addWikiMsg ( 'interwikiexistence-trigger-continuing',
					wfMessage ( $action ) );
                                break;
                        case INTERWIKIEXISTENCE_THROTTLED:
                                $output->addWikiMsg ( 'interwikiexistence-trigger-throttled',
					$seconds, wfMessage ( $action ) );
                                break;
			case INTERWIKIEXISTENCE_NOTIMESTAMP:
                                $output->addWikiMsg ( 'interwikiexistence-trigger-notimestamp',
					wfMessage ( $action ) );
                                break;
		}
		return;
	}

	protected function getGroupName() {
		return 'other';
	}
}

InterwikiExistence.alias.phpEdit

<?php
/**
 * Aliases for InterwikiExistence
 *
 * @file
 * @ingroup Extensions
 */
$specialPageAliases = array();
/** English
 * @author Leucosticte
 */
$specialPageAliases['en'] = array(
    'InterwikiExistence' => array( 'InterwikiExistence', 'Trigger interwiki existence update' ),
);

/** German (Deutsch)
 * @author Kghbln
 */
$specialPageAliases['de'] = array(
    'InterwikiExistence' => array( 'Interwikilinktabelle', 'Aktualisierung der Interwikilinktabelle starten' ),
);

InterwikiExistence.i18n.phpEdit

<?php
/**
 * Internationalisation for InterwikiExistence
 *
 * @file
 * @ingroup Extensions
 */
$messages = array();

/** English
 * @author Leucosticte
 * @author Kghbln
 */
$messages['en'] = array(
	'interwikiexistence' => 'Trigger interwiki existence table update',
	'interwikiexistence-desc' => 'Updates the local interwiki existence table with the contents of an interwiki map retrieved from another wiki',
	'interwikiexistence-trigger-success' => "The $1 part of the interwiki existence table is now completely up to date!",
        'interwikiexistence-trigger-throttled' => 'You do not have permission to manually trigger an update of the $2 part of the interwiki existence table after it has already been triggered within the last {{PLURAL:$1|second|seconds}}.',
	'interwikiexistence-trigger-noretrieve' => "An error retrieving data from the remote wiki prevented the interwiki table from being updated. The URL was:
$1",
        'interwikiexistence-trigger-nodecode' => "An error decoding data from the remote wiki prevented the interwiki table from being updated. The URL was:
$1",
        'interwikiexistence-notallowed' => 'Your account does not have permission to trigger interwiki existence table updates.',
        'interwikiexistence-notimestamp' => 'No interwiki existence timestamp was found for the $1 part of the interwiki existence table. Perhaps a system administrator needs to run <code>update.php</code>.',
        'interwikiexistence-continuing' => 'Progress was made in updating the $1 part of the interwiki existence table, but more API polls are needed to bring it completely up to date.',
        'interwikiexistence-creation' => 'creation',
        'interwikiexistence-deletion' => 'deletion'
);

/** Message documentation
 * @author Leucosticte
 */
$messages['qqq'] = array(
	'interwikiexistence-desc' => '{{desc}}',
	'interwikiexistence-trigger-success' => 'This is the message the user gets after he triggers a successful interwiki existence table update.',
        'interwikiexistence-trigger-throttled' => "This is the message the user gets when he tries to update the interwiki table too soon after the last triggering.",
	'interwikiexistence-trigger-noretrieve' => 'This is the message the user gets if an error retrieving data from the remote wiki prevents the interwiki existence table from being
updated. The parameter is the URL of the remote wiki.',
        'interwikiexistence-trigger-nodecode' => 'This is the message the user gets if an error decoding data from the remote wiki prevents the interwiki table from being updated. The
parameter is the URL of the remote wiki.',
        'interwikiexistence-notallowed' => 'This is the message the user gets when he tries to trigger interwiki table updates from an account without the interwiki right.',
        'interwikiexistence-notimestamp' => 'This is the message that is displayed when no interwiki existence timestamp is found.',
        'interwikiexistence-continuing' => 'This is the message displayed with progress was made in updating the $1 part of the interwiki existence table, but more API polls are needed
to bring it completely up to date.',
        'interwikiexistence-creation' => 'This is the name of the creation timestamp.',
        'interwikiexistence-deletion' => 'This is the name of the deletion timestamp.'
);

/** German (Deutsch)
 * @author Kghbln
 */
$messages['de'] = array(
	'interwikiexistence' => 'Aktualisierung der Interwikilinktabelle starten',
	'interwikiexistence-desc' => 'Aktualisiert die lokale Interwikitabelle mit dem Inhalt einer Interwikiübersicht eines anderen Wikis',
	'interwikiexistence-trigger-success' => "Der Teil $1 der Interwikilinktabelle ist nun aktuell.",
        'interwikiexistence-trigger-throttled' => 'Du bist nicht berechtigt die Aktualisierung des Teils $2 der Interwikilinktabelle zu starten, da während der vergangenen {{PLURAL:$1|Sekunde|Sekunden}} bereits eine Aktualisierung begonnen wurde.',
	'interwikiexistence-trigger-noretrieve' => "Ein Fehler beim Abrufen der Daten von einem fremden Wiki hat die Aktualisierung der Interwikilinktabelle verhindert. Die URL des Wikis lautet: „$1“,
        'interwikiexistence-trigger-nodecode' => "Ein Fehler beim Dekodieren der Daten von einem fremden Wiki hat die Aktualisierung der Interwikilinktabelle verhindert. Die URL des Wikis lautet: $1,
        'interwikiexistence-notallowed' => 'Du bist nicht berechtigt die Aktualisierung der Interwikilinktabelle zu starten.',
        'interwikiexistence-notimestamp' => 'In der Interwikilinktabelle konnte zu Teil $1 kein Zeitstempel ermittelt werden. Wohlmöglich muss das Skript <code>update.php</code> noch ausgeführt werden.',
        'interwikiexistence-continuing' => 'Teil $1 der Interwikilinktabelle wurde teilweise aktualisiert. Indes muss die API weitere Datenabrufe vornehmen, um die Aktualisierung abschließen zu können.',
        'interwikiexistence-creation' => 'erstellt',
        'interwikiexistence-deletion' => 'gelöscht'
);

/** German (formal address) (Deutsch (Sie-Form)‎)
 * @author Kghbln
 */
$messages['de-formal'] = array(
        'interwikiexistence-trigger-throttled' => 'Sie sind nicht berechtigt die Aktualisierung des Teils $2 der Interwikilinktabelle zu starten, da während der vergangenen {{PLURAL:$1|Sekunde|Sekunden}} bereits eine Aktualisierung begonnen wurde.',
        'interwikiexistence-notallowed' => 'Sie sind nicht berechtigt die Aktualisierung der Interwikilinktabelle zu starten.',
);

populateInterwikiExistencePageTable.phpEdit

<?php

/**
 * populateInterwikiExistencePageTable.php
 * This script polls AllPages on the remote wiki until it has completely populated the iwe_page
 * table.
 *
 * @author Leucosticte <https://www.mediawiki.org/wiki/User:Leucosticte>
 * @copyright GPL
*/

require_once( dirname( __FILE__ ) . "/../../maintenance/Maintenance.php" );

class PopulateInterwikiExistencePageTable extends Maintenance {
    public function execute() {
        global $wgInterwikiExistenceRemoteWikiUrl, $wgInterwikiExistenceApiAllPagesArgs,
            $wgInterwikiExistenceUserAgent;
        require_once 'setInterwikiExistenceTimestamp.php';
        SetInterwikiExistenceTimestamp::setTimestamp ( 'new', 'now' );
        SetInterwikiExistenceTimestamp::setTimestamp ( 'delete', 'now' );
        $dbw = wfGetDB( DB_MASTER );
        $result = $dbw->selectrow( 'user_properties', 'up_value', array (
            'up_user' => '1',
            'up_property' => 'interwikiexistenceallpagescont',
	) );
        if ( $result ) {
            $apiStart = $result->up_value;
            if ( !$apiStart ) {
                $apiStart = '!';
            }
        } else {
            $apiStart = '!';
        }
        echo "Starting with $apiStart ...\n";
        while ( $apiStart != null ) {
            $apiUrl = $wgInterwikiExistenceRemoteWikiUrl . str_replace (
                '$1', $apiStart, $wgInterwikiExistenceApiAllPagesArgs );
            $apiPull = InterwikiExistenceHooks::InterwikiExistenceApiPull ( $apiUrl,
                $wgInterwikiExistenceUserAgent );
            if ( isset ( $apiPull['status'] ) ) {
                echo "There was a problem polling the API. URL = $apiUrl\n";
                die();
            }
            // See what titles are in the database
                    $firstElement = true;
                    $cond = '';
                    $titles = array();
                    foreach ( $apiPull['query']['allpages'] as $apiPullElement ) {
                            $cleanTitle = $dbw->strencode ( $apiPullElement['title'] );
                            if ( !$firstElement ) {
                                    $cond .= " OR ";
                            }
                            $cond .= "iwe_page_title=" . $dbw->addquotes ( $cleanTitle );
                            $firstElement = false;
                            $titles[$cleanTitle] = wfTimestampNow();
                    }
                    $result = $dbw->select( 'iwe_page', array ( 'iwe_page_title', 'iwe_page_created' ),
                            $cond );
                    if ( $result ) {
                            foreach ( $result as $row ) {
                                    $timestamp = $titles[$row->iwe_page_title];
                                    unset ( $titles[ $row->iwe_page_title ] );
                                    if ( $timestamp > $row->iwe_page_created  ) {
                                        $dbw->update ( 'iwe_page', array (
                                                'iwe_page_created' => $timestamp ),
                                                array ( 'iwe_page_title' => $row->iwe_page_title )
                                        );
                                    }
                            }
                    }
                    // Insert rows
                    $insertArray = array();
                    foreach ( $titles as $title => $timestamp ) {
                            if ( $title != 'updated' ) {
                                    $insertArray[] = array (
                                            'iwe_page_title' => $title,
                                            'iwe_page_created' => $timestamp
                                    );
                            }
                    }
                    $dbw->insert( 'iwe_page', $insertArray );
                    // Update interwikiexistenceallpagescont timestamp if we've reached the end of the queries
                    if ( !isset ( $apiPull['query-continue'] ) ) {
                            echo "Population complete!\n";
                            $apiStart = null;
                    } else {
                            $apiStart = $apiPull['query-continue']['allpages']['apcontinue'];
                            $upsert = array (
                                    'up_user' => '1',
                                    'up_property' => 'interwikiexistenceallpagescont',
                                    'up_value' => $apiStart
                            );
                            $dbw->upsert(
                                    'user_properties',
                                    $upsert,
                                    array ( 'up_user', 'up_property' ),
                                    $upsert
                            );
                            echo "Continuing with $apiStart ...\n";
                    }
        }
    }
}

$maintClass = 'PopulateInterwikiExistencePageTable';
if( defined('RUN_MAINTENANCE_IF_MAIN') ) {
    require_once( RUN_MAINTENANCE_IF_MAIN );
} else {
    require_once( DO_MAINTENANCE ); # Make this work on versions before 1.17
}

setInterwikiExistenceTimestamp.phpEdit

<?php

/**
 * setInterwikiExistenceTimestamp.php
 * This script sets the "new" and "delete" timestamps for the InterwikiExistence extension.
 *
 * @author Leucosticte <https://www.mediawiki.org/wiki/User:Leucosticte>
 * @copyright GPL
*/

require_once( dirname( __FILE__ ) . "/../../maintenance/Maintenance.php" );

class SetInterwikiExistenceTimestamp extends Maintenance {
            public function __construct() {
                    parent::__construct();
                    $this->addOption( 'timestamp', 'Timestamp in yyyymmddhhmmss format, or "now"',
                        true, true );
                    $this->addOption( 'type', 'Type of timestamp ("new" or "delete")', true, true);
                    $this->addOption( 'overwrite', 'Overwrite existing timestamp ("yes" or "no"',
                        false, true );
            }

            public function execute() {
                    $timestamp = $this->getOption( 'timestamp' );
                    $type = strtolower ( $this->getOption( 'type' ) );
                    $overwrite = $this->getOption( 'overwrite' );
                    setTimestamp ( $type, $timestamp, $overwrite );
            }

            public static function setTimestamp ( $type, $timestamp, $overwrite = null ) {
                    if ( $type != 'new' && $type != 'delete' ) {
                        echo 'Allowable types: "new" or "delete"' . "\n";
                        die();
                    }
                    if ( strtolower ( $timestamp == 'now' ) ) {
                        $timestamp = wfTimestampNow();
                    }
                    if ( !is_numeric ( $timestamp ) || strlen ( $timestamp ) != 14 ) {
                        echo 'The timestamp must be a 14-digit integer (or use "now")' . "\n";
                        die();
                    }
                    if ( $type == 'new' ) {
                        SetInterwikiExistenceTimestamp::setProperty (
                                'interwikiexistencenewtimestamp', $timestamp,
                                $overwrite );
                        SetInterwikiExistenceTimestamp::setProperty (
                                'interwikiexistencenewconttimestamp', $timestamp,
                                $overwrite );
                    } else {
                        SetInterwikiExistenceTimestamp::setProperty (
                                'interwikiexistencedeletetimestamp', $timestamp,
                                $overwrite );
                        SetInterwikiExistenceTimestamp::setProperty (
                                'interwikiexistencedeleteconttimestamp', $timestamp,
                                $overwrite );
                    }
            }


        public static function setProperty ( $property, $timestamp, $overwrite = null ) {
                #$dbr = wfGetDB( DB_SLAVE );
                $dbr = wfGetDB( DB_MASTER );
                $dbw = wfGetDB( DB_MASTER );
                $result = $dbr->selectrow( 'user_properties', 'up_value', array (
                    'up_user' => '1',
                    'up_property' => $property
                ) );
                if ( !$result ) {
                    $dbw->insert ( 'user_properties', array (
                        'up_user' => '1',
                        'up_property' => $property,
                        'up_value' => $timestamp
                    ) );
                } elseif ( strtolower ( $overwrite ) === 'yes' ) {
                        $dbw->update ( 'user_properties', array (
                                'up_value' => $timestamp
                        ), array (
                                'up_user' => 1,
                                'up_property' => $property,
                        ) );
                }
        }
}

$maintClass = 'SetInterwikiExistenceTimestamp';
if( defined('RUN_MAINTENANCE_IF_MAIN') ) {
    require_once( RUN_MAINTENANCE_IF_MAIN );
} else {
    require_once( DO_MAINTENANCE ); # Make this work on versions before 1.17
}

iwetable.sqlEdit

BEGIN;

CREATE TABLE iwe_page(
-- Primary key
iwe_page_id                     int NOT NULL PRIMARY KEY AUTO_INCREMENT,
-- Page title
iwe_page_title                  varchar(255) binary NOT NULL,
-- Timestamp of when page was either created or added to this table by update.php
iwe_page_created                binary(14) NOT NULL default '',
-- Timestamp of when page was deleted
iwe_page_deleted                binary(14) NOT NULL default ''
);

CREATE INDEX iwe_page_title ON       iwe_page (iwe_page_title);

COMMIT;