API:Client Code/Access Library Comparison

Initial microtask for Evaluating MediaWiki web API client libraries.

The Python libraries listed on API:Client Code fall into two groups: libraries that offer a layer of abstraction that makes it easy for programmers to take advantage of the internal structure of the Mediawiki instance, and simple API wrappers that handle functions like logging in and cookies but do not otherwise mask the API structure.

Available Python MediaWiki API libraries

Libraries that provide layers of abstraction over the MediaWiki API:

Pywikipediabot (IRC)
mwclient (PyPI)
wikitools (PyPI)
Wikipedia (PyPI)
- Documentation warns not to use this for bots; does not support rate-limiting and other important features.

Simple wrappers for the MediaWiki API:

simplemediawiki (PyPI)
- Handles cookies, login
- Requires Python 2.6+ or 3.3+
- Includes a documentation build configuration file, but no separate documentation
- Provides usage examples in comments
- Includes unit tests
- Last updated: March 2014

supersimplemediawiki
- Handles login, cookies, get, post, and API calls
- Python 2.6-3.3 compatible
- Has an example script, but no other documentation.
- No tests provided
- Last updated: Nov 2013

python-mwapi (PyPI)
- Handles login, get, post
- Minimal documentation; no usage examples provided
- No tests provided
- Last updated: Dec 2012

The simplemediawiki is the only one of these basic API client libraries actively maintained (as of March 2014) and it is now compatible with Python 3. It is the best documented and is the only one that includes unit tests. The documentation could be improved by writing a more in-depth introduction to the library, making the example code easier to find and better documented, and including a separate documentation file. Simplemediawiki makes it easy to create a user_agent string, handles login and cookies, and provides a Python interface for making API calls (including the relatively recent Wikibase extension wbsearchentities).

Tests for simplemediawiki

First tests: getting data, no login/token needed

Make sure that the API call you're making is supported on the wiki you're interacting with. wbsearchentities is available on Wikidata, not on Mediawiki.

 
import simplemediawiki 

wiki = simplemediawiki.MediaWiki('http://www.mediawiki.org/w/api.php')

# sample query suggested by Wikidata API sandbox
print wiki.call({'action': 'wbsearchentities', 'search': 'abc',
                 'language': 'en', 'limit': 10, 'continue': 10})

# result: {u'servedby': u'mw1200', u'error': {u'info': u"Unrecognized
#         value for parameter 'action': wbsearchentities", u'code':
#         u'unknown_action'}}

# Wikidata API Sandbox request: /w/api.php?action=wbsearchentities&format=json&
#                  search=abc&language=en&limit=10&continue=10
#
# Wikidata API Sandbox result:
## {
##    "searchinfo": {
##        "search": "abc"
##    },
##    "search": [
##        {
##            "id": "Q4650112",
##            "url": "//www.wikidata.org/wiki/Q4650112",
##            "description": "album by Chinese-American rapper Jin",
##            "label": "ABC"
##        },
##        {
##            "id": "Q4650107",
##            "url": "//www.wikidata.org/wiki/Q4650107",
##            "description": "American high wheeler automobile",
##            "label": "ABC"
##        },
##        ...etc...
##    ],
##    "search-continue": 20,
##    "success": 1
##}

Get calls work now that they are being called on the correct API! For instance:

import simplemediawiki 

#generating a user-agent string 
myuseragent = simplemediawiki.build_user_agent('fhocutt_bot_API_test',
                 0.1, 'frances.hocutt+wikibot@gmail.com')

wikidata = simplemediawiki.MediaWiki('http://www.wikidata.org/w/api.php')
print wikidata.call({'action': 'wbsearchentities', 'search': 'abc', 'language':
                     'en'})

##result: {u'search-continue': 7, u'searchinfo': {u'search': u'abc'}, u'search':
##         [{u'url': u'//www.wikidata.org/wiki/Q169889', u'description':
##           u'American broadcast television network', u'label':
##           u'American Broadcasting Company', u'id': u'Q169889', u'aliases':
##           [u'ABC']}, {u'url': u'//www.wikidata.org/wiki/Q286874',
##           u'description': u'Disambiguation page', u'label': u'ABC', u'id':
##           u'Q286874', u'aliases': [u'Abc', u'ABC (broadcasting)',
##           u'ABC (channel)', u'ABC (disambiguation)', u'ABC (network)',
##           u'ABC (TV channel)', u'ABC (TV)', u'ABCs']}, {u'url':
##           u'//www.wikidata.org/wiki/Q781365',
##           u'label': u'Australian Broadcasting Corporation', u'id': u'Q781365',
##           u'aliases': [u'ABC']}, ...etc...

Corresponding API request in browser:

http://www.wikidata.org/w/api.php?action=wbsearchentities&format=json&search=abc&language=en

Result:

  {"searchinfo":{"search":"abc"},"search":[{"id":"Q169889","url":
  "//www.wikidata.org/wiki/Q169889","aliases":["ABC"],"description":
  "American broadcast television network","label":"American Broadcasting Company"},
  {"id":"Q286874","url":"//www.wikidata.org/wiki/Q286874","aliases":
  ["Abc","ABC (broadcasting)","ABC (channel)","ABC (disambiguation)","ABC (network)",
  "ABC (TV channel)","ABC (TV)","ABCs"],"description":"Disambiguation page","label":"ABC"},
  {"id":"Q781365","url":"//www.wikidata.org/wiki/Q781365","aliases"
  ["ABC"],"label":"Australian Broadcasting Corporation"}
  {"id":"Q1057802","url":"//www.wikidata.org/wiki/Q1057802", ...etc...

action=query&prop=info results for a missing page:

 
import simplemediawiki 

#generating a user-agent string 
myuseragent = simplemediawiki.build_user_agent('fhocutt_bot_API_test',
                 0.1, 'frances.hocutt+wikibot@gmail.com')

wikidata = simplemediawiki.MediaWiki('http://www.wikidata.org/w/api.php')

# getting information on 'Main Page'
print wikidata.call({'action': 'query', 'titles': 'Main Page', 'prop':
                    'info'})

# result = {u'query': {u'pages': {u'-1': {u'contentmodel': u'wikibase-item',
#                      u'ns': 0, u'pagelanguage': u'en', u'missing': u'',
#                      u'title': u'Main Page'}}}}

# Sandbox request: /w/api.php?action=query&prop=info&format=json&titles=Main%20Page
#
# Sandbox result:
##{
##    "query": {
##        "pages": {
##            "-1": {
##                "ns": 0,
##                "title": "Main Page",
##                "missing": "",
##                "contentmodel": "wikibase-item",
##                "pagelanguage": "en"
##            }
##        }
##    }
##}

Testing login, tokens, editing

Testing login capabilities and namespaces():

import simplemediawiki
 
myuseragent = simplemediawiki.build_user_agent('fhocutt_bot_API_test',
                 0.1, 'frances.hocutt+wikibot@gmail.com')

mediawiki = simplemediawiki.MediaWiki('http://www.mediawiki.org/w/api.php')
mediawiki.login('fhocutt_bot', ' . . . ')
print 'Logged in '+ str(mediawiki.login('fhocutt_bot', '  . . . '))

namespaces = mediawiki.namespaces()
print namespaces

result = mediawiki.call({'action': 'query', 'meta': 'userinfo', 'uiprop':
                         'rights'})
print result

##Results in: 
##Logged in True
##{0: u'', 1: u'Talk', 2: u'User', 3: u'User talk', 4: u'Project',
## 5: u'Project talk', 6: u'File', 7: u'File talk', 8: u'MediaWiki',
## 9: u'MediaWiki talk', 10: u'Template', 11: u'Template talk', 12: u'Help',
## 13: u'Help talk', 14: u'Category', 15: u'Category talk', 1198:
## u'Translations', 1199: u'Translations talk', 828: u'Module', 829:
## u'Module talk', 2500: u'VisualEditor', 2501: u'VisualEditor talk',
## 90: u'Thread', 91: u'Thread talk', 92: u'Summary', 93: u'Summary talk',
## 100: u'Manual', 101: u'Manual talk', 102: u'Extension', 103:
## u'Extension talk', 104: u'API', 105: u'API talk', 106: u'Skin',
## 107: u'Skin talk', -1: u'Special', -2: u'Media'}
##{u'query': {u'userinfo': {u'rights': [u'createaccount', u'read', u'edit',
##u'createpage', u'createtalk', u'writeapi', u'editmyusercss', u'editmyuserjs',
##u'viewmywatchlist', u'editmywatchlist', u'viewmyprivateinfo',
##u'editmyprivateinfo', u'editmyoptions', u'centralauth-merge',
##u'codereview-use', u'abusefilter-view', u'abusefilter-log', u'translate',
##u'vipsscaler-test', u'reupload-own', u'move-rootuserpages', u'minoredit',
##u'purge', u'sendemail', u'lqt-split', u'lqt-merge', u'lqt-react',
##u'translate-messagereview', u'translate-groupreview', u'flow-hide',
##u'mwoauthmanagemygrants'], u'id': 1401507, u'name': u'Fhocutt bot'}}}

Conclusion: namespaces() works, login returns True, and the userinfo query would not be possible without the logged in bot.

Testing the 'edit' API call:

import simplemediawiki
 
myuseragent = simplemediawiki.build_user_agent('fhocutt_bot_API_test',
                 0.1, 'frances.hocutt+wikibot@gmail.com')
 
print myuseragent
 
mediawiki = simplemediawiki.MediaWiki('http://www.mediawiki.org/w/api.php')
mediawiki.login('fhocutt_bot', ' . . . ')
print mediawiki.login('fhocutt_bot', '  . . . ')
 
#info on logged in user
result = mediawiki.call({'action': 'query', 'meta': 'userinfo', 'uiprop':
                         'rights'})
print result

token = mediawiki.call({'action': 'tokens'})
print token['tokens']['edittoken']

print mediawiki.call({'action': 'query', 'prop': 'revisions', 'rvprop':
                      'content', 'titles': 'Project:Sandbox'})

# [[editing Project:Sandbox]]
mediawiki.call({'action': 'edit', 'title': 'Project:Sandbox', 'section':'new',
                'text': 'hello to you too', 'token': token['tokens']['edittoken']})

print mediawiki.call({'action': 'query', 'prop': 'revisions', 'rvprop':
                      'content', 'titles': 'Project:Sandbox'})

##Result:
##fhocutt_bot_API_test/0.1 python-simplemediawiki/1.1.1 (+frances.hocutt+wikibot@gmail.com)
##True
##{u'query': {u'userinfo': {u'rights': [u'createaccount', u'read', u'edit',
##u'createpage', u'createtalk', u'writeapi', u'editmyusercss', u'editmyuserjs',
##u'viewmywatchlist', u'editmywatchlist', u'viewmyprivateinfo',
##u'editmyprivateinfo', u'editmyoptions', u'centralauth-merge', u'codereview-use',
##u'abusefilter-view', u'abusefilter-log', u'translate', u'vipsscaler-test',
##u'reupload-own', u'move-rootuserpages', u'minoredit', u'purge', u'sendemail',
##u'lqt-split', u'lqt-merge', u'lqt-react', u'translate-messagereview',
##u'translate-groupreview', u'flow-hide', u'mwoauthmanagemygrants'],
##u'id': 1401507, u'name': u'Fhocutt bot'}}}
##575f150e71c7c6f03c772cb7138cf9a5+\
##{u'query': {u'normalized': [{u'to': u'User:Fhocutt bot', u'from':
##u'User:fhocutt_bot'}], u'pages': {u'202248': {u'ns': 2, u'pageid': 202248,
##u'revisions': [{u'*': u'hello everyone', u'contentmodel': u'wikitext',
##u'contentformat': u'text/x-wiki'}], u'title': u'User:Fhocutt bot'}}}}
##{u'query': {u'normalized': [{u'to': u'User:Fhocutt bot', u'from':
##u'User:fhocutt_bot'}], u'pages': {u'202248': {u'ns': 2, u'pageid': 202248,
##u'revisions': [{u'*': u'hello everyone else', u'contentmodel': u'wikitext',
##u'contentformat': u'text/x-wiki'}], u'title': u'User:Fhocutt bot'}}}}

The existing text of Project:Sandbox of "hello everyone" has been replaced with "hello everyone else", as directed.

Note that cache issues may mean that these changes do not appear when reloading the page in the browser.

Tests for mwclient

First tests

import mwclient

# create a site for Wikipedia
mysite = mwclient.Site('en.wikipedia.org')
print 'host is ' + mysite.host + ', path is ' + mysite.path

# make a page for [[Wikipedia:Sandbox]]
sandboxpage = mysite.Pages['Wikipedia:Sandbox']
print str(sandboxpage.site) + ', page name is ' + sandboxpage.name
text = sandboxpage.edit() #fetches text, does not change it

print 'Page is ' + str(len(text)) + ' chars long'

print 'Text in sandbox: ', text.encode('utf-8')

## Result:
##host is en.wikipedia.org, path is /w/
##<Site object 'en.wikipedia.org/w/'>, page name is Wikipedia:Sandbox
##Page is 332 chars long
##Text in sandbox:  {{Please leave this line alone (sandbox heading)}}<!--
##*               Welcome to the sandbox!              *
##*            Please leave this part alone            *
##*           The page is cleared regularly            *
##*     Feel free to try your editing skills below     *
##â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– 
##â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– 
##â– â– â– -->

The save() method requires logging in. Adding the following lines to the end of the above script

sandboxpage.save(text + u'\nwoohoo!', summary = 'Test edit')
print sandboxpage.edit()

leads to the following error message, rather than the text of the sandbox page with an added "woohoo!":

  Traceback (most recent call last):
    File "C:\Python27\mwclientfirsttest.py", line 17, in <module>
      sandboxpage.save(text + u'\nwoohoo!', summary = 'Test edit')
    File "C:\Python27\lib\site-packages\mwclient\page.py", line 114, in save
      raise errors.LoginError(self.site)
  LoginError: <Site object 'en.wikipedia.org/w/'>

The login functions have not been tested yet but are the next step. mwclient has also not been tested to determine whether the more recent wikibase-related API calls are made accessible.