User:NeilK/Worklog/2010-12-14 to 2010-12-20

Projects: Upload Wizard, MediaStorage, Resource Loader

Status: solid

Last week:

• Mostly OOO sick
• Worked on MediaStorage
• Working on UploadWizard patches for next week (Tuesday)
  • didn't have a patch ready, instead worked on the concurrency bugs since they indicate a deeper issue

This week:

• UploadWizard bugs
• Able to help with Resource Loader a bit

Tuesday Dec 14 2010

• Conference call re: MediaStorage
• Weekly Features meeting
• Committed some refactors to HTMLForm, along with a way to list and clear all stashed files from UploadStash. This was a feature I was trying to add Friday in an oh-wouldn't-this-be-helpful-while-debugging mood and it took way too long. Upside, at least now I understand MediaWiki's HTMLForm and other HTML classes, since I was trying to do everything the right way. Now, back to Roan's concurrency patch, which was the whole point of this.

Wednesday Dec 15 2010

• Figured out the concurrency issue and decided that we cannot actually fix this problem at the moment. I fiddled with Roan's idea of using separate memcache items to store everything and it works, although that switches us over to a completely new (and even more fragile) way of storing temporary upload data. That wouldn't be so bad, but we also rely on existing public API methods for the final steps of publishing, which all assume we are stashing files in the old way. So: we lose, unless we also rewrite those API methods, which aren't *quite* expired...

The right thing is to rearchitect all of MediaWiki to use a database table for stashed upload data. For many reasons -- reliability, concurrency, cross-colo functionality. Sessions and memcache are for stashing stuff that's slightly expensive to calculate, not for building apps on top of. Also, we can't impose any policy on sessions or memcache, like "must retain this data for as long as it takes to upload the file" -- for all we know the upload could take hours.

Deferring that until later. In the meantime we can do one upload at a time. (We could also rejigger the front end so that no upload was ever started within one or two seconds of each other, but that mitigates the problem, doesn't solve it -- you can't control timing with HTTP APIs).

Thursday Dec 16 2010

Worked on the highest priority frontend bugs. Fixed Safari 3.2 (and probably a lot of other legacy browsers in the process; there was a javascript backwards incompatibility). Oddly I don't have any environment where I can run Mac Safari 3.2 (Snow Leopard), but I realized it's also available for Windows. I hope the issue was the same.

It would have been nice if Calcey had noted which Safari was actually failing. I couldn't replicate the issue in 5, for obvious reasons. Then I remembered to check their test plan; I forgot they were going to test in a slightly outmoded version of Safari.

Set up a good environment to test IE8, and also got a good handle on why IE8 fails. No fix for the latter yet.

Friday Dec 17 2010

Bryan Tong Minh noticed my uploadstash-backend branch and said that if I'm rewriting it, it should be kicked out of 1.17. Makes sense, but perhaps I could accelerate writing the db version instead.

Dropped into the MogileFS IRC channel to ask a couple of questions, turned into a bit longer session with main developer (Dormando). Added a lot of info to MediaStorage Distributed Filesystem Choices.

Worked on this IE7/8 issue and tracked it down to the 'file' parameter being curiously missing on the PHP side. Although after installing IE8 dev tools (IE8 development now merely sucks, instead of sucking HARD) it appears to be fully part of the form as far as javascript is concerned on IE's end. For some reason I can't config Wireshark to read incoming network data from my IE8 VM, so I don't *really* know what the hell IE is sending. Lots of fruitless fiddling with config. Maybe should get some help here.

Some people debugging on IE use an HTTP proxy to capture WTH it is sending. Annoying that I can't do this with Wireshark.

Monday Dec 20 2010

Moved desks.

• Brandon (acting in Alolita's stead) asked me if I was pushing a patch today... should have consulted with Roan. I had just one bugfix and some other cleanups, and wanted to get a few more fixes in before a patch. Unfortunately it's way too late (in Amsterdam time) to consult with Roan. Will try to email him at end of business and see if he wants to do a quick patch tomorrow. Update Roan says he's good to go if I am (it's midnight in England, where he just arrived to hang with Reedy)

• Added a bit more data to the Distributed FS page, re: extensibility and how to handle multiple colos in MogileFS. Russell agrees that extensibility is useful to have as another criterion.

• IE8 issues with UploadWizard

Consulted Ryan Lane on the Wireshark issue when debugging my IE8 inside VMWare. We tried a number of things (even going to raw tcpdump) and he is stumped too. Somehow VMWare is arranging that Apache gets the data, but we can't figure out how it's getting there. Will try IRC or scouring the internets, if not will give up and ... try running wireshark within windows? ...use proxy?

AHA. VMware's impl of devices is different on MacOS. Although somehow it shows up in ifconfig's list of interfaces, it does not have a /dev entry. You need to use the utility vmnet-sniffer to capture a .pcap file which Wireshark will happily read.

OKAY, so back to the actual problem. (bug 26099). Somehow, with IE, we get the file as a filename, with urlencoding the usual key-value pairs of parameters. But not the uploaded file. Safari uses a boundary-style encoding, for each param. Somehow we are not creating a form with the right disposition or encoding or whatever. What does IE8 do under normal circumstances, with an ordinary posted file?

IE8 posting to UW - Content-Type: application/x-www-form-urlencoded

IE8 posting to Flickr - Content-Type: multipart/form-data; boundary=---------------------------7da1f11f2014a

In ApiUploadHandler.js we are saying to set enctype = 'multipart/form-data'. But this doesn't work on IE?

Wait a minute, there is code here copied (in ancient times) from mwEmbed, along with a note that changing ACTION doesn't work in IE. :( However, I have tested this before and it used to work... maybe something else has changed. Clearly we are POSTing, so changing the action DID work. But the enctype didn't.

Wow, easiest bugfix ever (given a couple of days of work just to get good debugging...). IE wants the attr to be "encType" and won't accept "enctype". No other browser I'm testing seems to care so we're good to go.

Worked a bit on IE6, IE7 issues. IE6 errors were impenetrable, so switched to IE7 in the hope this would be more debuggable.