Architecture meetings/Performance guidelines discussion 2014-05-14

21:00-22:00 UTC, in #wikimedia-office.

We discussed the Performance guidelines draft, and secondarily the architecture guidelines and security guidelines.

Meeting summary and logs

 * LINK: https://www.mediawiki.org/wiki/Architecture_meetings/Performance_guidelines_discussion_2014-05-14 (sumanah, 21:01:19)
 * Performance guidelines (sumanah, 21:01:42)
 * LINK: https://www.mediawiki.org/wiki/Performance_guidelines (sumanah, 21:01:49)
 * We chatted about this during a discussion at the Zurich hackathon and it sounds like this document is about 80-90% there (sumanah, 21:02:37)
 * I already have a big list of TODOs that I want to hit today and tomorrow https://www.mediawiki.org/wiki/Talk:Performance_guidelines#TODO_for_Sumana.2C_14_May_2014 before circulating this on #wikitech-l (sumanah, 21:02:51)
 * The performance guidelines, along with the upcoming security guidelines https://www.mediawiki.org/wiki/Security_for_developers/Architecture, interrelate with https://www.mediawiki.org/wiki/Architecture_guidelines - the idea is for new developers to look at those 3 docs which describe how we do things (sumanah, 21:03:07)
 * they are descriptive, not hypothetical/aspirational (sumanah, 21:03:19)
 * ACTION: parent5446 to write "picking the right cache: a guide for MW developers" (sumanah, 21:11:11)
 * ACTION: sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores! (sumanah, 21:14:29)
 * IDEA: I want to check whether we think it's a first-class performance goal to make anonymous browsing and logged-in browsing the same speed. (sumanah, 21:14:36)
 * AGREED: that the goal of making anonymous browsing and logged-in browsing the same speed goes into the future architecture vision document (sumanah, 21:18:22)
 * LINK: https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers - I have 3 examples right now listed for the caching layers stuff. I think in context as 1 of those 3 it's probably fine? (sumanah, 21:26:42)
 * Slightly OT, but on the topic of breaking the cache-- can we mention that mediawiki has pretty sane default in OutputPage, and if disable it and add your own caching headers, please don't cache if the user is logged in. That broke lots of things last fall. (sumanah, 21:27:31)
 * ACTION: sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example (sumanah, 21:29:07)
 * HELP: would be cool for someone to write an automated test to " naively check for introduced calls to WebResponse::setcookie and call attention to them at least." to notify us of potential upcoming cookie issues in new code (sumanah, 21:34:09)
 * LINK: https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Critical_paths This looks good to me and I want to put it into the "how to think about performance" section (the second one, not the intro bullet points) (sumanah, 21:34:27)
 * "memcached and object cache are listed both as layers, but aren't they the same thing?" - bawolff (sumanah, 21:34:59)
 * ACTION: sumanah to clarify object cache vs memcached in Caching Layers section (sumanah, 21:35:22)
 * AGREED: https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Critical_paths looks good to add to "how to think about perf" (sumanah, 21:38:45)
 * LINK: https://wikitech.wikimedia.org/wiki/Incident_documentation/20120607-LastModifiedExtension "LastModified extension was causing every page view to an attempted POST request to http://en.wikipedia.org/w/api.php. " (sumanah, 21:43:23)
 * in production, we implement the object cache (ephemeral storage for serialized PHP objects) in memcached. But in MediaWiki-Vagrant installs, on Tool Labs, and in some other contexts, we use Redis for that instead. The object cache is an interface or layer that we can implement in various ways. (sumanah, 21:48:42)
 * LINK: https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency I think this looks good and am trying to figure out where to put it (sumanah, 21:49:07)
 * LINK: http://fabien.potencier.org/article/72/the-rise-of-composer-and-the-fall-of-pear - I need to read this thoroughly (sumanah, 21:51:08)
 * ACTION: sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section (sumanah, 22:01:37)

Meeting ended at 22:02:21 UTC.

Action items

 * parent5446 to write "picking the right cache: a guide for MW developers"
 * sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores!
 * sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example
 * sumanah to clarify object cache vs memcached in Caching Layers section
 * sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section

Action items, by person

 * parent5446
 * parent5446 to write "picking the right cache: a guide for MW developers"
 * sumanah
 * sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores!
 * sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example
 * sumanah to clarify object cache vs memcached in Caching Layers section
 * sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section

People present (lines said)

 * sumanah (114)
 * RoanKattouw (34)
 * parent5446 (27)
 * ^d (22)
 * superm401 (21)
 * ebernhardson (19)
 * jeremyb (15)
 * bawolff (14)
 * bd808 (14)
 * wm-labs-meetbot (8)
 * AndyRussG (6)
 * robla (4)
 * StevenW (2)
 * csteipp (2)
 * Krenair (1)
 * marktraceur (1)
 * Krinkle (1)
 * TimStarling (0)
 * paravoid (0)
 * ori (0)
 * brion (0)

Generated by MeetBot 0.1.4 (http://wiki.debian.org/MeetBot)

Full log
21:00:09 #startmeeting Discussion of Performance guidelines | Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE). https://meta.wikimedia.org/wiki/IRC_office_hours 21:00:09  Meeting started Wed May 14 21:00:09 2014 UTC and is due to finish in 60 minutes. The chair is sumanah. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:09  Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:09  The meeting name has been set to 'discussion_of_performance_guidelines___channel_is_logged_and_publicly_posted__do_not_remove_this_note___https___meta_wikimedia_org_wiki_irc_office_hours' 21:00:16 #chair sumanah brion TimStarling paravoid ori 21:00:16  Warning: Nick not in channel: brion 21:00:16  Warning: Nick not in channel: TimStarling 21:00:16  Warning: Nick not in channel: paravoid 21:00:16  Warning: Nick not in channel: ori 21:00:16  Current chairs: TimStarling brion ori paravoid sumanah 21:00:32 hmmm 21:00:36 lol 21:00:52 It's the Zurich meeting all over again 21:00:56 ha! 21:01:04 it is ok since this is not a "please approve this" meeting 21:01:19 #link https://www.mediawiki.org/wiki/Architecture_meetings/Performance_guidelines_discussion_2014-05-14 21:01:42 #topic Performance guidelines 21:01:42 Krinkle is here this time at least! 21:01:45 True! 21:01:49 #link https://www.mediawiki.org/wiki/Performance_guidelines 21:02:10 (As I said in my email to wikitech-l: 21:02:11 We discussed this some in Zurich but I'd love a 21:02:12 chance to ask some followup questions to firm everything up. I'd also 21:02:12 welcome the chance to explain the two similar documents I'm working on: 21:02:12 architecture and security guidelines. 21:02:13 ) 21:02:36  I am 21:02:37 #info We chatted about this during a discussion at the Zurich hackathon and it sounds like this document is about 80-90% there 21:02:51 #info I already have a big list of TODOs that I want to hit today and tomorrow https://www.mediawiki.org/wiki/Talk:Performance_guidelines#TODO_for_Sumana.2C_14_May_2014 before circulating this on #wikitech-l 21:03:07 #info The performance guidelines, along with the upcoming security guidelines https://www.mediawiki.org/wiki/Security_for_developers/Architecture, interrelate with https://www.mediawiki.org/wiki/Architecture_guidelines - the idea is for new developers to look at those 3 docs which describe how we do things 21:03:19 #info they are descriptive, not hypothetical/aspirational 21:04:35 If anyone has any questions for me - "what's the point", "why are these 2 principles separate", etc., I am happy to answer 21:05:06 I can also use this time to ask a few questions :-) for instance: should I list memcache as a persistence layer? maybe I should say, "maybe use memcache (and/or other caches) INSTEAD OF a persistence layer!" 21:05:57 Items in memcached can be removed randomly at the will of the server 21:06:07 So I'd recommend not listing it as persistence 21:06:19 Even if, in practice, WMF memcache servers tend not to delete anything unless they expire 21:06:24 redis would be the 'in-memory' persistence store 21:06:27 [citation needed] 21:07:02 So, in https://www.mediawiki.org/wiki/Performance_guidelines#Persistence_layer right now I say: "memcached cache - storage for things that persist between requests, and that you don't need to keep - you're fine with losing any one thing. Use memcached to store objects if the database COULD recreate them but it would be computationally expensive to do so." 21:07:04 Note, however, that Redis when used as $wgMemc should not be considered persistence 21:07:21 memcache also deletes in case of server reboot and doesn't gracefully handle outage of a node (in case you have a memcache cluster with a number of nodes) 21:07:41 sumanah: yeah that is good, but I'd recommend not mentioned memached specifically 21:07:45 Why not? 21:07:47 you can't shrink or grow memcache node count without pain 21:07:55 In reality, it's $wgMemc (or any BagOStuff object) that we should talk about 21:08:04 $wgMemc can be memcached 21:08:07 Or it can be another backend 21:08:12 It's abstracted 21:08:17 so you always have extra memcache hot spare not in use and then can substitute in for a dead host's slot 21:08:49 parent5446: I am willing to, in this document, often say something like "a cache (such as memcached)" or "a unique identifier (such as a primary key)". Some people learn better by thinking of general classes, and some through concrete examples 21:09:36 Yeah that's fine 21:09:41 Thanks 21:09:53 I mean, in the code we make the same mistake all the time anyway 21:09:56 i suppose my worry is more about using the words memcache and persistence anywhere close to eachother :) memcache should be treated as something more ephemeral than a persistence layer 21:09:57 parent5446: if you feel like taking a crack at writing "picking the right cache: a guide" then I would love that 21:10:22 ebernhardson: Roan said (quoting from notes): If something is very expensive to recompute ... use something closer to a store. (we use backend Varnish, which we call a cache, which is kind of closer to store on the cache-to-store spectrum.) 21:10:25 Yeah sure, put that an #action or whatever 21:11:11 sumanah: i imagine hes talking about the parser cache there?  I suppose we then need to declare levels of 'computationally expensive' 21:11:11 #action parent5446 to write "picking the right cache: a guide for MW developers" 21:11:18 sumanah: right, but "store" or "persistence" can also be for canonical (permanent storage). memcache definitely isn't that 21:11:21 because what (i think) roan is talking about there is things that can take multiple seconds 21:11:51 in which case we dont just want them in an ephemeral cache, but also (in the case of parser) writen to a database 21:11:58 ebernhardson: jeremyb if there is an existing guide out there to our caching layers and which ones are for which purposes, that's better than https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers, please tell me so I can link to it 21:13:21 not aware of anything better 21:13:37 sumanah: For what it's worth, I think the section you have for "persistence" looks good. Taken in context you are saying the right things. 21:13:56 Taken out of context there is ambiguity in the wording 21:14:00 nod, nod 21:14:19  sumanah: I'm here 21:14:29 #action sumanah to improve the cache-to-store spectrum distinction, make sure no one thinks any of the caches are actually persistent stores! 21:14:34 Thank you RoanKattouw 21:14:36 #idea I want to check whether we think it's a first-class performance goal to make anonymous browsing and logged-in browsing the same speed. 21:14:48  I think it is 21:14:59 That'd be amazing if we could achieve that 21:15:02 RoanKattouw: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140514.txt This meeting: Starts 2100. 21:15:06  Some people have been talking about this but there are no real efforts to do that yet 21:15:18 OK, so, would you put this goal into the aspirational bucket rather than the descriptive bucket? 21:15:19 So... make the app servers as fast as Varnish? Seems unlikely. 21:15:35 I think it means using more Varnish caching for logged-in users. 21:15:39  Yes, exactly 21:15:41 Caching as much as possible 21:15:59 bd808: more like turn off user specialization so everyone can get the same varnish output (with some ESI) 21:16:07  Which would also make things faster for logged-in users in, say, Europe 21:16:20  And exactly, we'd need techniques like ESI 21:16:27 * bd808 nods 21:16:33  Right now we don't use ESI at all but we would want serious ESI magic to pull this off 21:16:40 OK, so it sounds like this goes into the vision doc, agreed? 21:16:42 <RoanKattouw> So it's probably a ways out to actually do this 21:16:56 <RoanKattouw> sumanah: I assume the vision doc is for pie in the sky stuff? 21:17:02 ("vision doc" is what I am calling the "what we want to do in the future" thing that will probably be across security, perf, and architecture) 21:17:07 <RoanKattouw> Or rather, is where pie in the sky stuff goes among other things 21:17:28 Mmmmm... mysterious sky pie 21:17:41 So, agreed? 21:17:44 <RoanKattouw> Right 21:17:47 <^d> The other option is slowing down anonymous users to be more like logged in users. Might be less work than the other way-round. 21:17:49 <^d> 21:17:54 * RoanKattouw agrees 21:17:56 ouch :-) 21:17:59 ok 21:18:00 <RoanKattouw> (with sumanah, not ^d ) 21:18:13 <^d> Aw, I was hoping you agreed with me there :p 21:18:22 #agreed that the goal of making anonymous browsing and logged-in browsing the same speed goes into the future architecture vision document 21:18:48 btw RoanKattouw, Tyler agreed to write a "so you need to pick the right cache for your app needs" doc :D 21:18:54 <RoanKattouw> Sweet 21:19:27 So, another minitopic 21:19:29 I have in my notes: 21:19:30 "Maybe going too much into details: in context of frontend caching & Varnish - do not serve different content from the same URL. Diff content, diff URL. Must also be true for anon users! URLs must be deterministic - with proviso that not if you are logged in! Then wrinkles - cookies submitted ..... caching layers..... should strive to cache logged in requests..... longstanding problem." 21:19:54 <RoanKattouw> This sounds like a "how to not break things in the face of caching" thing? 21:20:09 The fact that something here is called a "longstanding problem" makes my ears perk up .... what bit here is the problem? That is, what bit should be going in the vision doc? 21:20:27 I presume it's "we want to someday cache logged-in users". Am I right? 21:20:32 <RoanKattouw> Yes I think so 21:20:39 <RoanKattouw> Which is what we just agreed to put in the vision doc already 21:20:55 i could be off base, but i think the 'longstanding problem' is that we vary by query string instead of the main url. its not an end of the world type thing though 21:21:03 OK! Just wanted to confirm and make sure I'm understanding right 21:21:12 It's very easy to break the ability of Varnish to cache content. 21:21:20 (with this stuff I am very aware that I might think "that is the same thing" when actually there is an important distinction I am missing) 21:21:43 This might be the biggest WTF for new developers in our stack. 21:22:01 bd808: How do developers tend to do it, when they do break things accidentally? 21:22:03 <RoanKattouw> sumanah: They're not exactly the same thing if you wanna be pedantic (one is a goal, the other is an implementation), but to most people said implementation is the de facto way to achieve said goal 21:22:07 nod, nod 21:22:40 sumanah: One way is by adding a new cookie. That happened just a couple weeks ago from ... somebody. 21:22:49 Yup. Matt, GettingStarted. 21:22:57 It's a "bad example" in that section. 21:23:05 but not all new cookies, just cookies that match a particular regex that no developer would know exists 21:23:07 But are there others I should mention explicitly? 21:23:19 nod, nod. ebernhardson is that regex available somewhere that I should link to? 21:23:28 <RoanKattouw> We tend to have things relatively well tied up such that you're unlikely to break Varnish's ability to cache content (except for the GettingStarted thing, but that had never happened before), it's much more likely that Varnish's success in caching content will break what you're trying to do 21:23:57 heh 21:24:00 sumanah: not sure, someone in ops(or maybe roan ;) would know 21:24:16 the varnish config is so far out of an average developers perview 21:24:27 RoanKattouw: doesn't that just lead to the developer "fixing" the bug by breaking caching? 21:24:30 this sounds like a lint/unit test thing 21:24:37 <StevenW> I'm not really sure that the GettingStarted issue is really a good generalized performance example 21:24:44 RoanKattouw: Agreed. Scholarships needed soe work when first put behind the misc-varnish cluster to actually function. 21:24:52 s/soe/some/ 21:24:53 StevenW: ah - Matt suggested it to me as one of the examples for that particular section 21:24:58 <RoanKattouw> robla: Depends on the caching layer. If the caching layer is internal to MW, that is likely; if it's Varnish, it's unlikely 21:25:18 <RoanKattouw> We simply don't give you a lot of power to break Varnish 21:25:39 <StevenW> It didn't cause a performance problem just because of adding a new cookie. It caused it because if you just happen to use a new cookie name that matches "Token" our regex-based system for cookie matching causes Varnish to go haywire 21:25:42 <RoanKattouw> sumanah: Re the cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue 21:25:54 <RoanKattouw> And yes what Steven said 21:26:01 * ^d noms on some cookies 21:26:15 <RoanKattouw> It was an unfortunate combination of GS naming their cookie badly and the Varnish config being extremely naïve 21:26:16 fair enough....rather flukey example then 21:26:42 #link https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers - I have 3 examples right now listed for the caching layers stuff. I think in context as 1 of those 3 it's probably fine? 21:27:18 Slightly OT, but on the topic of breaking the cache-- can we mention that mediawiki has pretty sane default in OutputPage, and if disable it and add your own caching headers, please don't cache if the user is logged in. That broke lots of things last fall. 21:27:31 #info Slightly OT, but on the topic of breaking the cache-- can we mention that mediawiki has pretty sane default in OutputPage, and if disable it and add your own caching headers, please don't cache if the user is logged in. That broke lots of things last fall. 21:27:54 superm401: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140514.txt This meeting: Starts 2100. 21:27:54 on the other hand, when i hear about us having any kind of big enough perforamnce problem that its discussed on the lists, its always some outlandish hard to predict thing like a regex inside varnish matching a new cookie 21:28:09 > doing your own caching headers 21:28:18 ^probably something that shouldn't be done 21:28:19 would it actually make sense to write a Jenkins test looking for things that are gonna break that regex? 21:28:40 RoanKattouw: ok, I will chase those down somehow, thanks 21:28:42 sumanah: no, because its not the specific regex, its more the class of problem 21:29:07 #action sumanah Re the GettingStarted cookie regex, there are some Gerrit changes linked from the ops-l thread about this issue - grab them to link to if we go forward with this as an example 21:29:13 <^d> You could add a jenkins test for any time someone calls WebResponse::setcookie in a patch. That'd at least call attention to it. 21:29:18 sumanah, that would be a hard test to write, since cookies are usually set conditionally. 21:29:21 [sorry just got here]: memcached and object cache are listed both as layers, but aren't they the same thing? 21:29:22 <^d> But ebernhardson is right, it's hard to check for programmatically. 21:29:32 ^d's idea is doable, though. 21:29:34 <RoanKattouw> It's theoretically impossible, even 21:29:36 bawolff: happy to fix that, or to ask you to fix it :) 21:29:45 <AndyRussG> sumanah: I'd be happy for to see detailed info, or links to detailed info about about performance, cookies and caching. Haven't looked carefully at that section yet (I came here out of interest on query tuning) but there is some new-cookie-setting happening in the Campaigns extension, which I'm working on. 21:29:54 Kind of like how jenkins-job-builder warns you if the XML changes. Not an indication of a problem, just something to look at. 21:29:59 <^d> RoanKattouw: $foo = 'bar'; $setcookie = 'setcookie'; $setcookie( $foo ); 21:30:03 <^d> Hard to check for :) 21:30:08 bawolff: I'd like to clarify that Caching Layers section (the description) and break it out onto wikitech.wikimedia.org at some point, and welcome help on it 21:30:16 ok 21:30:23 <RoanKattouw> ^d: Exactly. Also, halting problem etc. Strictly speaking checking for this is literally impossible without running the code 21:30:23 * bawolff is still reading through the document 21:30:33 Yes to ebernhardson-- it's the idea that special cookie imply you are logged in (ones that contain the strings "Session", "Token", or "forceHTTPS"). If varnish thinks you're logged in it won't use the cache. 21:30:45 $cookie = 'setcookie'; $varname = 'cookie'; $$varname( $cookie ); 21:30:52 AndyRussG: https://www.mediawiki.org/wiki/Performance_guidelines and https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code may come in handy for you. I don't think we have anything more comprehensive than those drafts 21:31:25 <AndyRussG> sumanah: OK, thanks! 21:31:26 <^d> RoanKattouw: Indeed. But see what I said above. You could naively check for introduced calls to WebResponse::setcookie and call attention to them at least. 21:31:43 <RoanKattouw> Yup 21:31:45 AndyRussG: if you want more specifics on specific topics, please feel free to ask on the talkpage or create redlinks 21:32:15 Would someone like to be responsible for following up on this automated test idea? I can #action it and we can move on with talking about the guidelines 21:32:17 <AndyRussG> sumanah: OK, will do, thanks 21:33:43 ok, no one has volunteered, so I'll ask for #help 21:34:09 #help would be cool for someone to write an automated test to " naively check for introduced calls to WebResponse::setcookie and call attention to them at least." to notify us of potential upcoming cookie issues in new code 21:34:27 OK, now 2 quick-ish questions for y'all 21:34:27 #link https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Critical_paths This looks good to me and I want to put it into the "how to think about performance" section (the second one, not the intro bullet points) 21:34:59 #info "memcached and object cache are listed both as layers, but aren't they the same thing?" - bawolff 21:35:22 #action sumanah to clarify object cache vs memcached in Caching Layers section 21:35:25 <RoanKattouw> bawolff is right, memcached is a possible implementation of the object cache 21:35:42 <RoanKattouw> Other implementations being APC, database, etc. 21:35:56 <RoanKattouw> (or nothing) 21:36:36 Yes. But this document is talking specifically about WM. 21:36:41 And we use memcached. 21:36:51 And we're not planning on changing that. 21:36:57 <RoanKattouw> OK 21:37:01 I'll clarify it. :) 21:37:06 <RoanKattouw> Yes in that case the object cache and memcached are the same 21:37:22 <RoanKattouw> This conflation is pretty rampant in MW core as well, due to WMF-centricness 21:37:40 nod. 21:37:48 Any thoughts on the "critical paths" comment? 21:37:58 It looks good to me - anyone else? 21:38:24 <RoanKattouw> Looks fine 21:38:45 cool, moving on 21:38:45 #agreed https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Critical_paths looks good to add to "how to think about perf" 21:38:47 <AndyRussG> Important issue--we had to think about it in a bit of a different context 21:38:53 <AndyRussG> ^ re: critical paths 21:39:04 sumanah: https://wikitech.wikimedia.org/wiki/Incident_documentation/20120607-LastModifiedExtension 21:39:15 <AndyRussG> when adding some hooks for notifications for the education program extension 21:40:10 sumanah, however, MediaWiki-Vagrant uses Redis 21:40:14 I would love to ask people whether we should have something in the Performance Guidelines about logging - about what you should log, or about logging sampling, etc. I know I need to improve the "how to profile" re logging and add stuff from https://www.mediawiki.org/wiki/User:BDavis_%28WMF%29/Projects/Structured_logging#Current_logging but does any logging info/advice belong in the dev guidelines? 21:40:42 Logging does not really fit in with performance guidelines, but remembering to profile definitely does 21:41:00 Logging is more of just general coding practices 21:41:11 parent5446: maybe you don't understand me :) 21:41:14 Logging could help with tracking down performance in some cases, but like parent5446 said (used correctly) it could help with a lot of things. 21:41:24 Oh 21:41:28 parent5446: I'm asking whether there are ways that a choice to log or not log data could slow down the site 21:41:47 Oh, I didn't get that either. 21:41:54 Sorry 21:42:04 I can't think of a way that not logging would slow things down. 21:42:11 I think the logging impact is pretty small, since it uses UDP. 21:42:17 Rather than writing directly to a disk. 21:42:31 <^d> Well, logging every single page view 5 times probably isn't a great idea for performance. 21:42:41 Yeah, if you go crazy, it could cause problems. 21:42:42 sumanah: re "not planning on changing that" we already moved off memcache for some things. e.g. user sessions -> redis. 21:42:48 <^d> But generally, people should error on the side of logging more and not worrying about performance of the logging system itself. 21:42:54 <^d> *err 21:43:23 #link https://wikitech.wikimedia.org/wiki/Incident_documentation/20120607-LastModifiedExtension "LastModified extension was causing every page view to an attempted POST request to http://en.wikipedia.org/w/api.php. " 21:44:00 Good guideline - don't do anything on every page view unless you are very sure its ok 21:44:09 Yeah, I think there have been a few outages/slowdowns due to API usage. 21:44:23 Hold on - jeremyb superm401 - are you saying that we are, in some cases, choosing to implement the object cache in Redis? 21:44:33 yes we do that 21:44:42 Because my "we're using memcache and not changing that" was in response to <RoanKattouw> bawolff is right, memcached is a possible implementation of the object cache 21:44:55 or can do that. People on tool labs use redis for object cache all the time 21:45:01 and we use it for sessions 21:45:05 on wmf 21:45:09 sumanah, yes, but only for MediaWiki-Vagrant, Tool Labs, etc. 21:45:17 sumanah, memcached is the object cache in production. 21:45:21 sumanah: right, i know what it was in response to. I don't know exactly what "object cache" means. but memcache maybe isn't forever 21:45:35 But Redis is used for other things (job queue, sessions, GettingStarted) in WMF production. 21:45:37 (btw welcome Krenair to the Wikimedia Foundation since I just saw the welcome email :-) thanks for working with us!) 21:45:47 <Krenair> :) 21:45:51 Object cache is an interface, a layer; memcached and redis are implementations 21:45:59 we interrupt this programming to welcome Krenair :) 21:46:07 I think in this context "object cache" means "ephemeral storage for serialized php objects" 21:46:23 Welcome, Krenair. 21:46:27 bd808: well, doesnt it mean the includes/objectcache/ directory? 21:46:28 Welcome and congratulations! 21:46:34 bd808: because anything else would just be odd :) 21:46:47 sumanah: re logging or not logging having a perf impact see the link i hilighted you with before 21:46:52 nod. 21:47:00 bd808: Any time you need to store a Bag O' stuff, we use the bagOStuff class (object cache) :) 21:47:04 I would absolutely love for experts to give me a glossary to supersede https://www.mediawiki.org/wiki/Performance_guidelines#Caching_layers, or to edit that section to clarify it. Anyone volunteering? 21:47:12 <^d> ebernhardson: Pfft, you newbies and your subdirectories of ./includes/. IN MY DAY we put everyting in ./includes/ and WE LIKED IT 21:47:19 :P 21:47:48 * bd808 is still baffled by the lack or real or fake namespacing 21:48:03 bd808: php5.3 has a *huge* perforamnce hit for namespaced code 21:48:15 MW_Cache_BagOStuff 21:48:16 bd808: its really not a great idea until 5.4 or hhvm(e.g. 3 years ago for everyone else :P) 21:48:26 <^d> bd808: Well, that's why MWNamespace exists. 21:48:29 <^d> Used to be Namespace. 21:48:32 <^d> But then 5.3 happened. 21:48:42 #info in production, we implement the object cache (ephemeral storage for serialized PHP objects) in memcached. But in MediaWiki-Vagrant installs, on Tool Labs, and in some other contexts, we use Redis for that instead. The object cache is an interface or layer that we can implement in various ways. 21:48:47 Use long complicated class names. Its fun 21:48:51 Hence PEAR pseudo-namespaces using underscores 21:49:07 #link https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency I think this looks good and am trying to figure out where to put it 21:49:11 RevDel_* (although that was just removed) 21:49:11 sumanah, tweaked it a little re object cached. 21:49:16 sumanah: we also cache opcodes in APC. which means that we can reduce the opcode cache hit rate if too many copies of mediawiki are in use on a single box at once. that's why we limit it to 2 slots in prod at a time. (excluding testwiki. and maybe testwikidata? but test2wiki is always on one of the 2 slots) 21:49:16 Thank you superm401! 21:49:20 <^d> parent5446: Yeah, because PEAR is a beacon of good design practices ;-) 21:49:59 <^d> jeremyb: Just test, not testwikidata. But otherwise yes, you're right. We exhaust the APC cache otherwise. 21:50:25 I think latency stuff might not really be anywhere right now except in the discussion of edge caching and of general user expectation. 21:50:47 getting off on a tangent, but PEAR is also pretty much dead. not even phpunit plans to continue releasing there 21:50:53 jeremyb: The thing you just said: does it imply that developers should do or not do anything in particular? :) 21:50:59 <^d> ebernhardson: PEAR2 or bust! 21:51:08 http://fabien.potencier.org/article/72/the-rise-of-composer-and-the-fall-of-pear - I need to read this thoroughly 21:51:25 <^d> sumanah: No, most developers shouldn't be bothering with that. It's mostly a WMF deployment/operational concern. 21:51:36 <^d> Developers shouldn't be worrying about "am I going to exhaust the bytecode cache" 21:51:37 I thought so. 21:53:22 sumanah: it means that if we can't have enwiki on one mediawiki version, commons on another and dewiki on a 3rd. all 800+ wikis can only be one of 2 distinct versions. one impact for developers is that they should make liberal use of feature flags so that individual wikis can vary on those in InitialiseSettings.php even though the mediawiki version is the same 21:53:35 sumanah: and that applies to extensions too not just core 21:54:07 I feel like Brion's point on latency might be important enough to merit another top-level arch principle - "users should experience consistent speed of service despite variable network latency" or something like that 21:54:08 <^d> Feature flags should exist for about half a dozen reasons, that being one of the least imho. 21:54:53 ^d: sure, plenty of reasons :) 21:55:02 Yeah, it sounds like this does not need to go into the performance guidelines, in my opinion. General HOWTO, yes. 21:55:17 * sumanah hopes people will illuminate her thinking on the latency topic 21:55:32 s/means that if we/means that we/ 21:55:33 BTW since we have 5 min left, any questions about security guidelines or architecture guidelines? 21:56:36 RoanKattouw: Krinkle robla - would love your thoughts on my ponderings in the last ~20 lines of backscroll re https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency :) 21:58:05 sumanah, not sure what Brion means by "consistent speed of service despite variable network latency". 21:58:09 Is that in a mailing list somewhere? 21:58:20 Seems hard to make very slow connection seem equally good as a fast connection. 21:58:40 superm401: well, I'm the one coming up with that phrasing, which is why it might not make sense. :) 21:59:16 There are things you can do to make loading delay seem less annoying. 21:59:21 * sumanah has generally been doing the work of initially summarizing and synthesising developers' thoughts into one- or two-sentence principles 21:59:23 I think acceptable speed of service regardless of network latency should be the goal 21:59:32 E.g. load the center (main text) first, and the rest after. 21:59:36 * sumanah nods 21:59:39 Though done wrong, this results in annoying flashes and reflows. 21:59:42 yeah 21:59:57 cough...fundraising banners 22:00:11 bawolff: you saw https://www.mediawiki.org/wiki/Performance_guidelines#Preserving_positioning ? 22:00:14 I think a lot of the latency stuff belongs in general performance principles 22:00:51 ah, its already been said 22:00:52 (just a guess, though...would need a more careful reading to figure it out) 22:01:37 #action sumanah to consider putting https://www.mediawiki.org/wiki/Talk:Performance_guidelines#Latency into the "how to think about perf" section 22:02:02 sumanah: edited parser cache 22:02:15 Well, it's been an hour so I'll say g'night. Thanks for your feedback all. I hope the end document helps a lot of people understand how to write efficient code, and helps code reviewers have something to point at when giving thumbs-up and thumbs-down. 22:02:21 #endmeeting