Manual:Pywikibot/MySQL

Pywikibot includes basic support for SQL queries on MySQL-like database or replica.

Setting up

Pywikibot

To make Pywikibot work with SQL-based database or its replica, make sure you have pymysql library installed. Probably you already installed it when installing Pywikibot. Otherwise, run:

$ pip install pymysql

Second you should update the DATABASE SETTINGS section in your user-config.py file:

Parameter	Value
Parameter	locally	remotely	Toolforge
`db_hostname_format`	`localhost`	IP address or host	`{0}.analytics.db.svc.wikimedia.cloud`^[1]
`db_port`	identical to the server port in `my.cnf` file (default: `3306`)
`db_name_format`	identical to database name (default: `{0}`)^[1]		`{0}_p`^[1]
`db_connect_file`	path to the `my.cnf` file (default: `/etc/mysql/my.cnf` or `~/.my.cnf`)		`~/replica.my.cnf`
`db_username`	the credentials to connect to the database, if no `db_connect_file` provided		Do not use
`db_password`			Do not use

↑ ^1.0 ^1.1 ^1.2 {0} here stands for dbName value in current pywikibot.Family.

Database

The database also have to be ready and have to contain corresponding set-up and structure.

On Toolforge everything should be up and running with credentials specified in the tool's replica.my.cnf, see wikitech:Help:Toolforge/Database.

From dump

If you want to run your local instance (e.g. a copy loaded from Wikimedia dump), follow the following steps:

For Wikimedia dump first download your chosen SQL dump from: https://dumps.wikimedia.org/backup-index.html.

Second install and configure mariadb or other preferred MySQL-like database on your local machine. Follow your OS distribution manual (e.g. https://wiki.archlinux.org/index.php/MySQL).

Once ready, start MySQL terminal/command line interface:

$ mysql -u yourusename -p

and create new database using the database name chosen before:

> create database yourdbname;

In case of Wikimedia dump finally you have everything prepared for the final step: importing the downloaded sql table dump to your prepared local database:

> quit
$ mysql -u yourusername yourdbname < /path/to/sql/dump/file/xywiki-20180601-sometable.sql

or you can declare your own database containing page namespace, page title and some additional rows of your choice:

> create table yourtable (page datatype, namespace datatype, ...)
> set yourtable.page = somepage
> set yourtable.namespace = somens
> quit

Usage

Terminal/command line

If the desired script supports pagegenerators.py , you can run your script with -mysqlquery:'query' generator. You will be prompted for the query if no query specified. The query should return two columns, page namespace and page title pairs from some table, e.g.:

$ python pwb.py some_script -mysqlquery:'select page_namespace, page_title from page where ...;'

Inside your script

If your script does not support page generators yet, you can import MySQLPageGenerator from pagegenerators.py:

from pywikibot import pagegenerators

Bot.__init__(generator=pagegenerators.MySQLPageGenerator('select page_namespace, page_title from page where ...', site=self.site))

But you can also get other information from database using pywikibot.data.mysql library:

from pywikibot.data import mysql

some_value = mysql.mysql_query('select page_id from page where ...')