Architecture meetings/RFC review 2014-10-01

Oct 02 07:02:55 <TimStarling>    #startmeeting
Oct 02 07:02:56 <wm-labs-meetbot>    TimStarling: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee'
Oct 02 07:03:02 *    rfarrand has quit (Quit: Computer has gone to sleep.)
Oct 02 07:03:12 <TimStarling>    #topic API roadmap | | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE). | Logs:
Oct 02 07:03:21 <TimStarling>    #link
Oct 02 07:03:47 *    lbenedix has quit (Quit: Leaving.)
Oct 02 07:04:07 *    DarTar has quit (Quit: DarTar)
Oct 02 07:04:31 <TimStarling>    do we have anomie and yurikR?
Oct 02 07:04:33 *    rdaiccherlb has quit (Quit: Computer has gone to sleep.)
Oct 02 07:04:38 <anomie>    TimStarling: I'm here
Oct 02 07:04:49 <yurikR>    yep
Oct 02 07:05:02 *    ryasmeen is now known as ryasmeen|Away
Oct 02 07:05:38 *    J-Mo (~jtmorgan@ has joined #wikimedia-office
Oct 02 07:05:42 *    rfarrand (~rfarrand@ has joined #wikimedia-office
Oct 02 07:06:09 *    DangSunM|cloud has quit (Ping timeout: 260 seconds)
Oct 02 07:06:09 *    Revi has quit (Ping timeout: 260 seconds)
Oct 02 07:06:57 <TimStarling>    can you tell us what has been done on this API work since the architecture summit?
Oct 02 07:07:03 *    alantz ( has joined #wikimedia-office
Oct 02 07:07:56 <anomie>    I've started working on the stuff in the document. I added Gerrit links to each item as patches got submitted, and moved a few things to a "completed" section.
Oct 02 07:08:44 <anomie>    Since we're taking things slow as far as deprecation, some have their patches merged but need an analysis of whether people have actually changed their code.
Oct 02 07:09:00 *    TrevorParscal is now known as TrevorP|Away
Oct 02 07:09:39 *    Guest24138 (sid13042@gateway/web/ has joined #wikimedia-office
Oct 02 07:10:11 <TimStarling>    you mean like token handling?
Oct 02 07:10:23 *    rdaiccherlb (~rdaiccher@wikimedia/rdicerb-wmf) has joined #wikimedia-office
Oct 02 07:10:36 <anomie>    Yes
Oct 02 07:10:43 *    James_F|Away is now known as James_F
Oct 02 07:10:58 *    Revi (sid12940@wikimedia/Hym411) has joined #wikimedia-office
Oct 02 07:11:11 *    ryasmeen|Away is now known as ryasmeen
Oct 02 07:11:26 *    alantz has quit (Ping timeout: 258 seconds)
Oct 02 07:12:11 *    alantz ( has joined #wikimedia-office
Oct 02 07:12:53 <TimStarling>    it looks like you need code review on some changes
Oct 02 07:12:59 <anomie>    Yes, I do
Oct 02 07:15:02 <TimStarling>    is there anything else you need?
Oct 02 07:15:12 *    alantz has quit (Client Quit)
Oct 02 07:16:39 <anomie>    Not really.
Oct 02 07:16:47 <anomie>    I'm still not too fond of the decision to go with format=json2 for, but I still agree with the points you made at Wikimania that clean breaking is better than random mystery breaking.
Oct 02 07:16:47 <TimStarling>    I have a set of API feature requests from Tomasz that he sent in June
Oct 02 07:17:09 <anomie>    I would like to see those
Oct 02 07:17:16 <TimStarling>    I'll forward
Oct 02 07:17:39 *    alantz ( has joined #wikimedia-office
Oct 02 07:17:51 <TimStarling>    the main one that is relevant is a request for "chain queries"
Oct 02 07:18:13 <TimStarling>    "The fever queries we have to send the better it gets for our users batteries."
Oct 02 07:19:00 <TimStarling>    so I suppose we are talking about doing multiple actions in a single POST request
Oct 02 07:19:19 *    InezK_away is now known as InezK
Oct 02 07:19:38 <anomie>    We already have generators for a common instance in action=query. Details on what other "chains" he's thinking of would be useful.
Oct 02 07:19:58 <TimStarling>    yeah, he didn't give details, but I assume he knows about generators already
Oct 02 07:20:20 *    DarTar (~DarTar@wikimedia/DarTar) has joined #wikimedia-office
Oct 02 07:21:23 *    wisdom is now known as alpha
Oct 02 07:22:24 <gwicke>    don't forget that SPDY / HTTP2 is around the corner
Oct 02 07:22:37 *    alantz has quit (Quit: Computer has gone to sleep.)
Oct 02 07:23:34 <TimStarling>    well, if we are just talking about doing several unconnected API queries in a row, that could be done with pipelining, if the client supported that
Oct 02 07:23:42 <gwicke>    which eliminates some of the issues that generators are designed to address
Oct 02 07:23:59 <TimStarling>    but what if you are taking some data from one query and using it in the next query?
Oct 02 07:24:00 <anomie>    gwicke: No it doesn't.
Oct 02 07:24:03 *    DanielK_WMDE (~daniel@wikipedia/duesentrieb) has joined #wikimedia-office
Oct 02 07:24:06 *    alantz ( has joined #wikimedia-office
Oct 02 07:24:17 <TimStarling>    then it could be arbitrarily complicated
Oct 02 07:24:22 <gwicke>    TimStarling: right, that is the bit that isn't addressed
Oct 02 07:25:07 <gwicke>    security is another relevant aspect to consider
Oct 02 07:25:15 <gwicke>    DOS in particular
Oct 02 07:25:51 *    alantz has quit (Client Quit)
Oct 02 07:25:52 *    Jyothis has quit (Remote host closed the connection)
Oct 02 07:26:23 *    Jyothis (~Jyothis@wikipedia/Jyothis) has joined #wikimedia-office
Oct 02 07:26:25 <TimStarling>    you mean DOS by means of an expensive query batch?
Oct 02 07:26:26 <gwicke>    we shouldn't provide entry points that allow somebody to take down the API cluster by visiting some static web page with their cell phone
Oct 02 07:27:03 <gwicke>    there is a security bug with an example page
Oct 02 07:28:10 *    TrevorP|Away is now known as TrevorParscal
Oct 02 07:28:18 <gwicke>    #62615
Oct 02 07:28:49 <yurikR>    i would actually prefer to keep queries separate too
Oct 02 07:28:53 *    parent5446 (parent5446@mediawiki/parent5446) has joined #wikimedia-office
Oct 02 07:29:08 <yurikR>    if you want to chain requests, lets rely on http-level protocol
Oct 02 07:29:11 *    alantz ( has joined #wikimedia-office
Oct 02 07:29:13 *    DarTar has quit (Quit: DarTar)
Oct 02 07:29:36 <TimStarling>    but splitting it up implies duplicated overhead
Oct 02 07:29:48 <yurikR>    if some data is needed for consequent request, we either create specific api that understands that (e.g. - generators for query and other)
Oct 02 07:29:57 <cscott>    or rely on gzip compression to take care of it
Oct 02 07:30:16 *    alantz has quit (Client Quit)
Oct 02 07:30:30 <yurikR>    well, the overhead will be negligent if they reuse the same connection, plus caching might make it much more efficient
Oct 02 07:30:47 <yurikR>    with combining done on the api level, caching is totally busted
Oct 02 07:30:52 *    Jyothis has quit (Ping timeout: 240 seconds)
Oct 02 07:30:56 *    aharoni has quit (Remote host closed the connection)
Oct 02 07:30:59 <TimStarling>    I mean in varnish, apache and HHVM
Oct 02 07:31:03 <gwicke>    I don't think anybody is proposing to get rid of generators or chaining in general altogether -- it's just that we should be careful about what we use them for, and keep in mind how HTTP/2 affects the trade-offs
Oct 02 07:31:15 <TimStarling>    there is per-request overhead at each level
Oct 02 07:31:22 <TimStarling>    especially in HHVM/MW
Oct 02 07:31:43 <JetLaggedPanda>    re: Tomasz's requests, I think the problem there is action=mobileformat, which apps use (this was the reason for asking about pipelining, IIRC)
Oct 02 07:31:44 <TimStarling>    also in MySQL
Oct 02 07:31:48 <JetLaggedPanda>    and that doesn't support generators or anything
Oct 02 07:31:58 <JetLaggedPanda>    so over time slowly things have been tacked on to it
Oct 02 07:32:16 *    anomie sees no action=mobileformat on enwiki
Oct 02 07:32:18 <TimStarling>    there's a big difference in MySQL CPU usage between doing a single query that gets information about 100 pages, and doing 100 queries, one for each page
Oct 02 07:32:28 <JetLaggedPanda>    anomie: gah, action=mobileview
Oct 02 07:32:29 *    flyingclimber has quit (Remote host closed the connection)
Oct 02 07:32:48 <gwicke>    TimStarling: the same is not necessarily true if each of those pages is stored on a different node
Oct 02 07:33:03 *    mhurd has quit (Quit: mhurd)
Oct 02 07:33:05 <yurikR>    JetLaggedPanda, there was a big change a while ago that allowed any module to use generators
Oct 02 07:33:15 <anomie>    JetLaggedPanda: I'd have to look at what exactly action=mobileview is doing, but offhand it sounds like it needs any unique bits rolled into core. Much like a lot of MobileFrontend.
Oct 02 07:33:22 <yurikR>    so now the mobileview simply needs to be updated to use generators
Oct 02 07:33:25 <JetLaggedPanda>    anomie: i agree, yeah
Oct 02 07:33:29 <TimStarling>    we're not going to split storage across hundreds of nodes
Oct 02 07:33:54 <JetLaggedPanda>    yurikR: yeah, that would be good too, although perhaps it needs general query prop= as well
Oct 02 07:34:01 <gwicke>    perhaps not hundreds, but we already use dozens
Oct 02 07:34:25 <JetLaggedPanda>    yurikR: *also*, perhaps this could be solved by simply making mobileview html a prop= for action=query, but I guess that'll have caching implications
Oct 02 07:34:28 <TimStarling>    I don't think so
Oct 02 07:34:46 <yurikR>    JetLaggedPanda, yes, i think it should have been done that way :)
Oct 02 07:35:13 <gwicke>    TimStarling: I agree with your general point, it's just that it might not be an eternal truth to the same degree it's right now
Oct 02 07:35:14 *    tfinc (~tfinc@wikipedia/Tfinc) has joined #wikimedia-office
Oct 02 07:36:42 *    alantz ( has joined #wikimedia-office
Oct 02 07:37:58 <TimStarling>    #info implementation by anomie is proceeding, some changes just need code review and merge
Oct 02 07:37:59 <gwicke>    there are for example wins in making more API requests static by storing or caching them; combined with different cost structures in HTTP/2 some applications might actually perform better when they do a few parallel requests vs. hitting a custom, uncached entry point
Oct 02 07:38:41 <TimStarling>    #info Tomasz requested a "chain query" feature, but we need specific requirements
Oct 02 07:39:06 <gwicke>    I see it more as a gradual shift
Oct 02 07:39:49 <TimStarling>    you can't cache API responses
Oct 02 07:40:33 <gwicke>    it's not technically impossible
Oct 02 07:40:40 <TimStarling>    maybe you could if it were REST, but it is too difficult to invalidate the multiple URL variants enabled by the action API
Oct 02 07:40:44 <anomie>    Some API response can be cached, mostly action=query. We already emit cache-control headers indicating what MediaWiki thinks about cacheability.
Oct 02 07:41:04 <anomie>    True, people might have stale caches then.
Oct 02 07:41:12 <TimStarling>    the client requests cache-control headers
Oct 02 07:41:37 <TimStarling>    the client is explicitly requesting a stale cache since there is no way to update those caches once they are generated
Oct 02 07:41:40 *    James_F is now known as James_F|Away
Oct 02 07:42:25 *    gwicke nods
Oct 02 07:42:29 *    bearND has quit (Remote host closed the connection)
Oct 02 07:42:44 <TimStarling>    maybe we could normalize requests in varnish...
Oct 02 07:42:46 <DanielK_WMDE>    there's a lot of stuff on that rfc page. perhaps it would be good to split it to ease discussion.
Oct 02 07:42:48 <anomie>    The major opportunity for caching is revision content, for which gwicke is already working on a REST API specifically intended for heavy caching.
Oct 02 07:43:12 <DanielK_WMDE>    the way things a structured now, i'm afraid some high profile discussions may drown out talk about some finer points
Oct 02 07:43:28 <gwicke>    I think it might be worth looking for other resources that could potentially be cacheable with the right URL structure
Oct 02 07:43:50 <gwicke>    and have the right granularity / access pattern for this to make sense
Oct 02 07:43:59 <TimStarling>    even with normalization, you still have things like rvprop
Oct 02 07:44:25 *    bearND ( has joined #wikimedia-office
Oct 02 07:44:27 <TimStarling>    with REST, you just send all the data, but with api.php, each application will request a different rvprop
Oct 02 07:44:41 *    TrevorParscal is now known as TrevorP|Away
Oct 02 07:45:29 <TimStarling>    so even in that simple case, you multiply the cache space requirement by several
Oct 02 07:46:13 *    alantz has quit (Quit: Computer has gone to sleep.)
Oct 02 07:46:20 <gwicke>    yeah, it only makes sense if the number of variants is more limited
Oct 02 07:46:38 <gwicke>    which is something we could try to move towards for newer modules
Oct 02 07:46:55 <gwicke>    where the trade-offs make sense
Oct 02 07:47:02 <TimStarling>    for purging, imagine if you had to send an HTCP purge request for each rvprop combination
Oct 02 07:47:13 *    alantz ( has joined #wikimedia-office
Oct 02 07:48:00 <anomie>    DanielK_WMDE: There's basically no discussion happening there at the moment, so I doubt anything is being drowned out. Although at some point (not now) I'd still like to hear your thoughts on what makes things like ApiResult::setIndexedTagName hard for you to use (without getting into redesigning the whole thing around a forest of objects, that was discussed enough at Wikimania IMO).
Oct 02 07:48:06 <gwicke>    returning more props by default would probably not make a big difference in request size, and could still result in a faster response if the response is cached in exchange
Oct 02 07:49:13 *    alantz has quit (Read error: Connection reset by peer)
Oct 02 07:49:21 *    Guest24138 has quit (Changing host)
Oct 02 07:49:21 *    Guest24138 (sid13042@wikimedia/DangSunM) has joined #wikimedia-office
Oct 02 07:49:31 *    alantz ( has joined #wikimedia-office
Oct 02 07:49:36 *    Guest24138 is now known as DangSunM|cloud
Oct 02 07:49:39 <gwicke>    there are some entry points where the choices culd perhaps be reduced a bit without major ill effects
Oct 02 07:50:22 <TimStarling>    #info gwicke suggests we consider a gradual shift towards greater edge caching coupled with the use of SPDY, as a replacement for batches embedded in single queries (incl. generators)
Oct 02 07:50:53 <gwicke>    that's overstating it quite a bit
Oct 02 07:51:24 <TimStarling>    the meetbot command is unprivileged, you can do your own #info if you like
Oct 02 07:52:57 *    alantz has quit (Read error: Connection reset by peer)
Oct 02 07:52:59 <gwicke>    #info s/as a replacement for batches/as a replacement for *some* batches and expensive generators/
Oct 02 07:53:21 *    alantz ( has joined #wikimedia-office
Oct 02 07:53:47 *    PPena has quit (Quit: Computer has gone to sleep.)
Oct 02 07:54:05 <TimStarling>    should we mark this RFC as approved?
Oct 02 07:54:14 <gwicke>    I also think that we could do some of the assembly and orchestration in an intermediate layer
Oct 02 07:54:18 *    JetLaggedPanda is now known as YuviPanda|zzz
Oct 02 07:54:51 <gwicke>    netflix of example has been doing something like that:
Oct 02 07:54:52 *    awight has quit (Remote host closed the connection)
Oct 02 07:55:11 *    Krenair thinks we should
Oct 02 07:55:15 <TimStarling>    I think the reason RFCs don't get approved is that we worry that by marking an RFC approved, we are approving every little aspect
Oct 02 07:55:59 <anomie>    It's fine with me to mark it as approved; I've been treating it that way for a while now.
Oct 02 07:56:22 <anomie>    The only drawback might be that it might discourage further discussion and further things for my "TODO" list.
Oct 02 07:56:25 *    mhurd ( has joined #wikimedia-office
Oct 02 07:56:28 *    moizsyed ( has joined #wikimedia-office
Oct 02 07:57:13 *    kaity|away is now known as kaity
Oct 02 07:57:15 <TimStarling>    yeah, maybe it makes sense for something this complex to be a living document
Oct 02 07:57:44 <anomie>    We could move the living document portion of it out of the RFC, although I'm not sure what would be left in the RFC then.
Oct 02 07:57:45 <DanielK_WMDE>    ...or factor out some parts that can be considered agreed on and treated as a "plan".
Oct 02 07:57:50 <yurikR>    gwicke and I just spoke about caching a bit, and it seems ideally we should somehow cache certain requests, and devise a well established way to flush them when they become obsolete
Oct 02 07:57:54 *    jhobs has quit (Ping timeout: 246 seconds)
Oct 02 07:57:59 <TimStarling>    it suggests a status flow "in draft" -> "archived complete" for big RFCs
Oct 02 07:58:17 *    jhobs ( has joined #wikimedia-office
Oct 02 07:58:25 <yurikR>    this caching won't apply to every api request, but we really ought to move in that direction
Oct 02 07:58:33 <DanielK_WMDE>    what does "archived complete" mean?
Oct 02 07:58:43 <DanielK_WMDE>    "we are done talking"?
Oct 02 07:59:00 <TimStarling>    it means it will be listed at
Oct 02 07:59:15 <TimStarling>    yes, which means we are done talking
Oct 02 07:59:19 *    alantz has quit (Quit: Computer has gone to sleep.)
Oct 02 07:59:23 *    Jeff_Green (~jgreen@wikipedia/jgreen) has left #wikimedia-office
Oct 02 08:00:01 <TimStarling>    we presumably won't discuss archived RFCs in public IRC meetings or architecture committee meetings
Oct 02 08:00:33 *    alantz ( has joined #wikimedia-office
Oct 02 08:00:43 <TimStarling>    for the API roadmap, the work could theoretically be eternal
Oct 02 08:00:54 <DanielK_WMDE>    yea, makes sense
Oct 02 08:00:58 *    ori (~ori@wikipedia/ori-livneh) has joined #wikimedia-office
Oct 02 08:01:00 <TimStarling>    but I prefer to see RFCs as change requests that can be approved and completed
Oct 02 08:01:08 *    zz_MissGayle ( has joined #wikimedia-office
Oct 02 08:01:10 *    zz_MissGayle is now known as MissGayle
Oct 02 08:01:14 <DanielK_WMDE>    in such a case, the goal of the rfc is not to implement a feaqture, but to agree on a general plan
Oct 02 08:01:31 *    MissGayle has quit (Changing host)
Oct 02 08:01:31 *    MissGayle (~gyoung@wikimedia/gyoung) has joined #wikimedia-office
Oct 02 08:01:38 *    PPena ( has joined #wikimedia-office
Oct 02 08:01:45 <TimStarling>    maybe the RFC should be called "API roadmap 1"
Oct 02 08:01:51 <anomie>    DanielK_WMDE: I think that's a good summary of RFCs in general.
Oct 02 08:01:54 <TimStarling>    which can be marked approved
Oct 02 08:02:20 <TimStarling>    then while that is being implemented, an "API roadmap 2" RFC can be the parking lot for design of the next batch of features
Oct 02 08:03:09 <TimStarling>    then we can schedule a meeting to discuss "API roadmap 2" and we will know that that means we are looking forward not back
Oct 02 08:03:22 <AaronS>    heh
Oct 02 08:03:41 <TimStarling>    you know it is nice when people don't have to read so much
Oct 02 08:03:51 *    James_F|Away is now known as James_F
Oct 02 08:03:53 *    mhurd has quit (Quit: mhurd)
Oct 02 08:04:14 <TimStarling>    Daniel complained about the RFC being big already, but it has a lot of complete stuff mixed with plans for the near future, plus a few plans for the somewhat more distant future
Oct 02 08:04:34 <DanielK_WMDE>    anomie: that's my understanding too, but the final status is currently called "implemented". That'S a lot more than "agreed on a plan".
Oct 02 08:05:08 <TimStarling>    we have "accepted" also
Oct 02 08:05:14 *    Ltrlg has quit (Quit: Leaving.)
Oct 02 08:05:18 <anomie>    So, to summarize: RFC is approved, the "living document" aspect should be abstracted out into a project page of some sort (I'll do that), and when we have enough of a backlog of non-trivial changes we'll make a new RFC (I'll probably do that too when the time comes).
Oct 02 08:05:27 *    parent5446 (parent5446@mediawiki/parent5446) has left #wikimedia-office ("wikimedia-office")
Oct 02 08:05:52 <TimStarling>    yeah, makes sense I think
Oct 02 08:06:24 *    moizsyed has quit (Remote host closed the connection)
Oct 02 08:07:02 *    kaity is now known as kaity|away
Oct 02 08:07:05 *    alantz has quit (Quit: Computer has gone to sleep.)
Oct 02 08:07:06 *    kaity|away is now known as kaity
Oct 02 08:07:23 <TimStarling>    #action anomie to abstract the "living document" aspect of the RFC out to a project page
Oct 02 08:07:36 <TimStarling>    ok, anything else before I end the meeting?
Oct 02 08:07:53 *    kristenlans has quit (Quit: kristenlans)
Oct 02 08:07:54 <anomie>    Not from me, I was about to leave the meeting anyway
Oct 02 08:08:07 *    bearND has quit (Remote host closed the connection)
Oct 02 08:08:26 <TimStarling>    #endmeeting