Extension:External Data/Throttling data retrievals

{{#get_web_data:}}, {{#get_soap_data:}}, {{#get_program_data:}}, mw.ext.externalData.getWebData, mw.ext.externalData.getSoapData and mw.ext.externalData.getProgramData can be throttled per data source, that is, a delay between calls to the same web service or program will be enforced. Meanwhile, a message informing that calls are throttled will be shown, or stale cache, if allowed and available will be used.

For web sites, soap services and server-side programs, throttling is configured by the settings 'throttle key' and 'throtlle interval':

  • 'throttle key' is a template generating a throttle key. Wildcards like $host$, $url$ or $2nd_lvl_domain$ (for web services) or $param$ (for parameters to programs) will be replaced with their corresponding values. The default value is $2nd_lvl_domain$ meaning that, for example, any call to any page in Wikipedia in any language will have the throttle key wikipedia.org;
  • 'throtlle interval' is a float holding the mimimal interval, in seconds, between calls to web services or server-side programs with the same throttle key.

As with other settings, these are per data source, which can be:

  • The full URL,
  • host,
  • second-level domain,
  • '*' for the default fallback for any site.

Throttling makes sense, when there are numerous calls of parser function (e.g., caused by a template embedded many times) addressing the same external service or program that either requires much computational resources or is, effectively, a call to an external service, like youtube-dl.

If there are no throttling key or throttling interval is zero or not set, there will be no throttling. This is the default setting.