I am trying to get rendered HTML for a bunch of pages. I am pretty sure I am limiting myself to 200 requests per second to page/html API endpoint, and I have User-Agent set, but very quickly I am starting to get 429 responses. Have I misunderstood something about the rate limit for that API endpoint? I am making those 200 requests in parallel though. Could this be a problem? (I use stash=false as well.)
Topic on Talk:Wikimedia REST API/Flow
It looks like limit of 100 requests works. (Hm, or not. It still starts returning 429 after some time.) So I tested now with 100 requests per second. I managed to make 45852 requests in 8 minutes and after that they started failing with 429. This is a bit less than 100 requests per second.
So example response:
If you report this error to the Wikimedia System Administrators, please include the details below.
Request from IP via cp3056 cp3056, Varnish XID 528119086
Upstream caches: cp3056 int
Error: 429, Too Many Requests at Wed, 04 May 2022 16:48:01 GMT
I am sad that expected rate limit from hyperswitch does not get reported: https://github.com/wikimedia/hyperswitch/blob/2d4480149ae4d1969eec83063ddec05c9021574d/lib/filters/ratelimit_route.js#L32-L37
Please create a Phabricator ticket and paste the information above so people can check this and help you. Mutante (talk) 17:11, 4 May 2022 (UTC)