Extension:VisualData/Maintenance scripts

Currently VisualData offer the following maintenance scripts:

RebuildData
ImportData
DeleteRegex
ProcessData
Uninstall

RebuildData

RebuildData can be used in the following way:

php extensions/VisualData/maintenance/RebuildData.php

it drops all the VisualData tables and completely rebuilds them based on the contents of wiki articles. As mentioned above it is particularly important that the "source of truth" are the data stored in wiki articles rather than in the tables of the extension, since the data-structure of the tables of the extensions may change at any time, and there is no guarantee that the data are preserved in the original form. It is also particularly important that the data are stored in wiki articles (as json content model) so that sysops or authorized users have direct access to them.

Within such framework, protection of sensitive data may be achieved either using preventing access extensions, or more adequately using a private wiki where to store sensitive information and a public wiki where to store public information. By this point of view future versions of the extensions may allow inter-wiki queries and/or form submissions.

allowed paramaters
parameter	description	required
exclude-prefix	exclude pages with prefixes (comma separated)	no
recast-data	recast data	no

ImportData

ImportData allows to import either json or csv (comma separated values) data to your wiki. The command can be used in the following way:

php extensions/VisualData/maintenance/ImportData.php --file "[path-to-the-following-file]/countries+states+cities.json" --schema Country --pagename-formula "Data:countries/<name>" --main-slot

it allows to bulk-import json data to your wiki and to create an article for each of them based on the pagename-formula. For instance the previous command creates 250 wiki pages based on this data each of them with all the data related to each country, for a total of more than 620,000 imported entries.

The data are created in the main slot (with the parameter --main-slot) because no wikitext is expected to be associated to the article.

This way the wiki becomes a repository for all your data, and the specific VisualData database only a way to query the data that can be rebuilt anytime, where the "source of truth" is the standard Mediawiki database and the wiki article itself.

Since version 1.09 the pagename-formula can include a #count suffix to enable a counter (integer increment, starting from 1).

allowed paramaters
parameter	description	required
file	filename (complete path)	yes
schema	schema registered on the wiki	yes
pagename-formula	pagename formula	no
main-slot	whether to save to main slot	no
limit	limit pages to be imported	no
category-field	field to be used to assign categories (at root level)	no
csv-array-field-separator	separator for array fields (only csv)	no (default comma)

DeleteRegex

DeleteRegex can be used in the following way:

php extensions/VisualData/maintenance/DeleteRegex.php --regex "Data:countries/.+" --delete

As the name and the example suggest, it allows to bulk-delete wiki articles based on a regex.

allowed paramaters
parameter	description	required
regex	regex applied to page_title	yes
delete	do delete (true/false)	no
namespace	limit to specific namespace (literal)	no
limit	limit results (numeric)	no

ProcessData

ProcessData is a "boilerplate" script intended to advanced users who need to create their own scripts to handle json-data.

Here is an example method with a query and a callback to edit and save the returned data for each article.

	private function myScript() {

		// schema name
		$schema = '';

		// e.g. [[name::+]]
		$query = '';

		// leave empty to retrieve all printouts, if
		// they are too many use the params printouts-from-conditions = true
		$printouts = [];

		// use an high limit (default is 100)
		$params = [
			'limit' => 5000
		];

		$callback = static function ( $title, $data, $categories ) {
			// edit returned data, they will be automatically saved
			// in the same article if edited
			return $data;
		};

		$updated = \VisualData::editDataCallback( $this->user, $schema, $query, $printouts, $params, $callback );

		foreach ( $updated as $titleText ) {
			echo "$titleText has been updated" . PHP_EOL;
		}
	}

After saving the updated script it can be called with

php extensions/VisualData/maintenance/ProcessData.php --script myScript

myScript is the method name, the class may contain a method for each required operation.

Uninstall

See here for more information: since VisualData stores articles' metadata within a separated slot with its content model (derived from json), such slots/slot roles and content model have to be deleted or converted before removing the extension.

The uninstall script first exports all json data in a file saved in the user's home folder, and then removes all slots with json-data, converts to json the json-data content model (used by VisualData) if saved in the main slot (or removes it if the script is called with the remove-main-slot-jsondata option) and finally converts to text the html content model which is also provided by VisualData.

Call the script this way to test it out:

php extensions/VisualData/maintenance/Uninstall.php --limit 10

and like so to perform the actual deletion:

php extensions/VisualData/maintenance/Uninstall.php --uninstall

After that the extension can be commented out/removed from LocalSettings.php

// wfLoadExtension( 'VisualData' );

allowed paramaters
parameter	description	required
limit	set limit	no
remove-main-slot-jsondata	remove main slot if it contains json-data	no
uninstall	perform actual deletion