User:Want/Is ifexist expensive function?

Author: Aleš Kapica (Want), published in July 2024

Last updated: 2024-07-25 by Want

I recall, MediaWiki.org originally was intended to serve as a source of usable templates and help pages for administrators of independent MediaWiki instances. Today exists here 12,860 templates, but most of them practically impossible usable.

Anyone who has tried it knows that it's not easy. You need to manually copy the template, find the dependencies, copy the missing templates and modules, and finally fix the code according to the situation. Here most templates uses modules of the Extension:Scribunto which complicated it. – Why?

MediaWiki.org does not use the advanced ParserFunctions, loops, user variables, etc. provided by native PHP extensions because someone in the past thought they were a risk. I understand it. The complexly template in the pure wikicode are not very clear (brackets, brackets, brackets,..) and a small error in the code of the frequentaly uses template, as {{Ll }} for example, can cause the entire website to be unavailable. From this perspective, it's posed a risk.

Scribunto (development start in 2012), allows you to write scripts directly into wiki pages (modules) and processed this code through an independent server-side interpreter. But isn't it a ticking time bomb in terms of MediaWiki development?

The advantage of native PHP extensions is that they are not dependent on other software. If the performance of PHP improves, their performance will automatically improved too. In 2019, at the Wikimedia Hackathon in Prague, I learned that the deployment of a new version of PHP (from 5.x to 7.x), which according to tests was supposed to bring a significant improvement in performance, had practically no effect. – Why?

Problem of the Main page Mediawiki.org

I recently stirred up the murky waters here by trying to solve a problem that unnecessarily discourages beginners who don't speak English well. The language bar on the main page does not switch languages ​​subpages as usual, but do links to the translatable template. Moreover, it does not apply to all translations. I've been trying to figure out where this is coming from to possibly fix it, but I haven't been able to find it.

But that's not the only problem of this language bar. If you have a small display (e.g. on a mobile phone) it's not visible because it is at the very bottom, outside the screen. So I used the {{Language }} template, which although switches languages ​​correctly, but also doesn't refer to all the languages ​​into which the content of the main page is translated and started working on a new {{Mt }} template that would she solved it.

My change sparked the discussion (see Topic:Y8l01lc8uein5fng), that showed that is there a long-term unresolved problem. And I was surprised at how expensive templates MediaWiki.org uses.

Comparation of the {{Ll }} template

The following table compares the performance of my Template:ll and local template {{Ll }}. I'm not using caching and yet the values ​​are very similar. However, my template has a much wider use, because its output can be used as the input of another template. Real examples of use are described in documentation, which is of course translatable.

	Limit	Template:Main page	MediaWiki	Woodcraft	Hlavní strana
Templates		{{Ll }}, {{Localized link }}, {{Pagelang }}, {{TNTN }}, {{Translatable }}, {{Translatable/en }}, {{Translatable template name }} Module:Template translation and message with title	{{Ll }}, {{Localized link }}, {{Pagelang }}, {{Translatable }}, {{Translatable/en }} and Module:Template translation	Template:ll, Template:pagelang	Template:ll, Template:pagelang
CPU time usage		0.028 seconds	0.035 seconds	0,023 sekund	0,043 sekund
Real time usage		0.041 seconds	0.051 seconds	0,049 sekund	0,053 sekund
Preprocessor visited node count	1,000,000	57	57	58	58
Post-expand include size	2,097,152 bytes	253 bytes	253 bytes	304	414
Template argument size	2,097,152 bytes	93 bytes	93 bytes	27 bajtů	42 bajtů
Highest expansion depth	100	11	11	9	9
Expensive parser function count	500	1	1	2	2
Unstrip recursion depth	20	0	0	0	0
Unstrip post-expand size	5,000,000 bytes	0 bytes	0 bytes	0 bajtů	0 bajtů
Number of Wikibase entities loaded	400	0	0	unused	unused
Lua time usage	10.000 seconds	0.003 seconds	0.004 seconds	unused	unused
Lua memory usage	52,428,800 bytes	662,697 bytes	662,697 bytes	unused	unused
ExtLoops	2600	unused	unused	0	0

Languagebar

I've written a lot of templates for my multilingual wiki, but most of them can't be used on MediaWiki.org because used functions unavailable there. But the {{Mt }} template can be compared because was wroted for using on MediaWiki.org.

First tested on my wiki. After placing it on MediaWiki.org, I noticed that when applied to all language codes, page languagebar exceeds the Expensive parser function count, which limited to 500. So I was looking for another way to solve it and found Manual:Checking for page existence.

A redesigned template is embedded here as {{Mu }}. It uses the #ifeq function instead of the expensive #ifexists function. And since it does the same thing and uses the same parameters, it was possible to do following test pages:

• User:Want/languagebar
• User:Want/Mt af
• User:Want/Mu af

You can do yourself test, if you do any change in code and use only the preview.

Test existence by #ifeq is wrong!

When I tried the {{Mu }} template, I was surprised that the processing of the User:Want/languagebar page code crashed after 15 iterations, because limit the post-expand include size was full. This is really proof that using an expensive function #ifexist is a more reasonable choice than check existence of the page by #ifeq function, which does not count towards the number of expensive parser functions!

	Limit	{{Mu }}
Status		stopped after 'ang' (15 items done on my wiki done 156 items)
CPU time usage		25.294 seconds
Real time usage		27.051 seconds
Preprocessor visited node count	1,000,000	584,178
Post-expand include size	2,097,152 bytes	2,097,152 bytes
Template argument size	2,097,152 bytes	1,752,121 bytes
Highest expansion depth	100	35
Expensive parser function count	500	14
Unstrip recursion depth	20	0
Unstrip post-expand size	5,000,000 bytes	30,660 bytes
Number of Wikibase entities loaded	400	0
Lua time usage	10.000 seconds	7.247 seconds
Lua memory usage	52,428,800 bytes	1,394,903 bytes

Here is output Lua Profile returned after processing crash:

recursiveClone	<mwInit.lua:45>	2820 ms	38.7%
MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::getExpandedArgument		1300 ms	17.9%
MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::newTitle		980 ms	13.5%
?		500 ms	6.9%
type		380 ms	5.2%
(for generator)		360 ms	4.9%
	<mwInit.lua:45>	160 ms	2.2%
MediaWiki\Extension\Scribunto\Engines\LuaSandbox\LuaSandboxCallback::expandTemplate		120 ms	1.6%
newFrame	<mw.lua:153>	120 ms	1.6%
tostring		80 ms	1.1%
[others]		460 ms	6.3%

Is the function #ifexist expensive?

So I did one more test. For one item only.

	Limit	#ifexist (User:Want/Mt af)	#ifeq (User:Want/Mu af)
CPU time usage		0.010 seconds	0.194 seconds
Real time usage		0.014 seconds	0.248 seconds
Preprocessor visited node count	1,000,000	42	2,629
Post-expand include size	2,097,152 bytes	220 bytes	109,357 bytes
Template argument size	2,097,152 bytes	85 bytes	5,838 bytes
Highest expansion depth	100	10	31
Expensive parser function count	500	2	12
Unstrip recursion depth	20	0	0
Unstrip post-expand size	5,000,000 bytes	0 bytes	0 bytes
Number of Wikibase entities loaded	400	0	0
Lua time usage	10.000 seconds	unused	0.080 seconds
Lua memory usage	52,428,800 bytes	unused	1,303,599 bytes
Lua Profile		unused	unused

Because the User:Want/languagebar page has same effect as the {{Language }} template. I can do next test. Difference of it, that the User:Want/languagebar page works with all 452 language codes. However the {{Language }} handles only 72 of them. That was the reason to do two tests: First for full set of language codes and second for set language codes as used by the {{Language }} template.

	Limit	{{Language }}	{{Mt }}
Note		only 72 items done	all 452 languages (same set language codes as {{Language }})
CPU time usage		0.695 seconds	0.903 seconds (0.141 seconds)
Real time usage		0.960 seconds	1.225 seconds (0.224 seconds)
Preprocessor visited node count	1,000,000	10,126	29,261 (6,651)
Post-expand include size	2,097,152 bytes	164,062 bytes	106,155 bytes (23,426 bytes)
Template argument size	2,097,152 bytes	43,298 bytes	60,846 bytes (11,396 bytes)
Highest expansion depth	100	18	13 (13)
Expensive parser function count	500	243	548 (134)
Unstrip recursion depth	20	0	0 (0)
Unstrip post-expand size	5,000,000 bytes	7,685 bytes	0 bytes (0 bytes)
Number of Wikibase entities loaded	400	0	0 (0)
Lua time usage	10.000 seconds	0.410 seconds	unused
Lua memory usage	52,428,800 bytes	1,692,425 bytes	unused

You can see for yourself that my template, which does not use the Lua interpreter, is 5x faster and much less memory intensive. The only parameter that limits its use for all language codes used by MediaWiki is more lower the limit for expensive parser functions. My wiki use 2000 for it. Here is set only 500. It is not possible to process all language codes without exceeding of it!

Template as a tool

My own wiki use templates very extensively. Not only for dynamic content of my Main page, but also as an auxiliary tool.

For example, Template:transstat generate calculations for paid translators. For links to specific places in the content of a site, I widely use navigation templates, because my version of the Template:ll supports anchors across all languages of content.

I use specialized templates that allow the page content to be translated into other languages, although the page itself is not translatable because I need the links to have only one destination at any given time. See for example 3AA3 – There's a lot waiting to be translated, but I'm working for the future. Even a beginner can translate a text. But only an experienced user can write documentation on how to use the features of a new template or extension.

I continually write own translatable help pages, for functionalities what the MediaWiki.org not documented. Info about technical changes wiki give the page Current events. There you can find all the information about updates and installed extensions from November 2013, when the wiki was launched, until now. You can try my widget, which generate link and send page to google translate.

Unfortunately, nothing from it can be used on MediaWiki.org because not installed the necessary extensions. But anyone who maintains own wiki can use it, if installed needed extensions and register and log in to my wiki – that's the only way he can get access to the code, which he can copy. Anonymous visitors do not have access to it.

Note: The dependency of the template on the extension can be recognized even without logging out of the categorization.

Singlepages

I use expensive features quite heavily on my wiki. And you can test for yourself how fast the pages load. The most expensive page my wiki is "The Book of Woodcraft, 1922 (book)", a Singlepage for a book that has 624 pages. Due to these pages, I have increased some limits (see char *)

	Limit	The Book of Woodcraft, 1922 (book)
CPU time usage		3.943 seconds
Real time usage		4.707 seconds
Preprocessor visited node count	1,000,000	21,415
Post-expand include size*	6,266,880 bytes	5,020,421 bytes
Template argument size*	6,266,880 bytes	17,259 bytes
Highest expansion depth	100	13
Expensive parser function count*	2000	0
Unstrip recursion depth	20	1
Unstrip post-expand size	5,000,000 bytes	2,608 bytes
ExtLoops	2600	638

My experiences with MediaWiki

My first contact with Wikipedia website was begin of August 2004. I work only as user into 2008.

Since 2008 summer I am the administrator of the wiki https://wiki.control.fel.cvut.cz, which is still running today and has been gradually updated to the current version LTS 1.39.x. In the same year 2008, I created the first version of the Extension:AccessControl and started own experimenting with templates. That wiki has a total of 151 templates, of which 56 templates are unused. They were created for testing purposes when I was learning how to write templates. Most of the 52 open non-existent templates are a relic from June 2010 when I tried to import the first templates from wikibooks.

This is proven by calling the 'Template:Hiddenh3' in the template documentation Template:Ref/doc, which the wiki has, but you can find it at wikibooks. In total, this wiki contains 7431 pages (of which 2750 are content) and 3507 files. About 10 of the 206 registered users have been continuously creating content during that time. And thanks to the change history, we could effectively intervene in the event of a patent dispute, as we could prove that a patent application was filed up to 10 years after it was published on our wiki work on the subject.

User, sysop, translator, root, developer and maintainer

In 2013 I started my own wiki https://www.thewoodcraft.org

Content of it is multilingual from the very beginning and intensive dynamically generated. The content template that generates the content of the main page was created in April 2015. It's a sophisticate of codes, sometime inspired by templates from MediaWiki.org, as the ll template for example, that generates links to multilingual pages. Unlike the {{Ll }} template, support anchors and also responds to the currently selected interface language - if there is a translation of the article title in the relevant language, it will use it.

That wiki has a total of 654 templates. Half of it consists of translations of the specialized template concept, used in translations. 204 templates is by system MediaWiki marked as unused. In fact, it is the already mentioned translations that are called indirectly, through the concept template or the templates to insert code. Those templates can also be used in another instance of MediaWiki, it's pure wikicode, but some of them depend on extensions that MediaWiki.org doesn't support - Scribunto offers the same functionality as them. But is really effective? I don't know.

In total, this wiki contains 38650 pages (of which 7358 are content) and 1872 files. About a third of the 63 registered users continuously participated in the creation of content during that time.

Why I don't use Scribunto on wikis?

I don't use Scribunto because I see no reason why the wiki should run next interpreter of programming language on the server side. Widgets are something else. They work with javascript, which is interpreted on the user's side by the browser.

I have an advantage over other developers because I don't have to ask anyone if I want to try or test something.

The downside for me is updating the MediaWiki core. I maintain the code myself and the user base is too small to rely on the help of other developers. That's why I stick to code from the official git repositories as much as possible and that's why I'm also the maintainer of the AccessControl and EImage extensions.

My life is complicated only by changes between releases MW, that force changes in the code of the extensions used. I don't have time to continually keep track of partial changes and correct minor errors in the PHP code or in the code that Scribunto interprets.

The project I maintain is not as extensive as the WMF projects. I am mainly creater of content, not programmer or sysop. Every user of my wiki has absolutely freedom of creation. He must only respect others users and changes of their content do only after consultation with hims.

I do templates maximally effective, because the efficient code garantee a good speed of environment. And the host's performance is not unlimited too.

I wrote a lot of manuals to my wiki. And I also contribute to the translations of those on MediaWiki.org - it saves me work.

My work on MW

Through the page Project:translation/cs, I try to pass on my experience to other translators in order to make the journey as easy as possible for beginners and those who want to run their own multilingual wiki.

I am pleased that, thanks to the Special:ActiveLanguages/cs group of Czech translators, although it is not very numerous, Czech is in fourth place in the ranking of translated content here.