Thread:Talk:Wikimedia Labs/Crap/reply (3)

If you take a look through the Technical pump archives, you'll notice a pretty normal pattern of the Toolserver being down for days or weeks at a time this year, here's a thread discussing it being down for two weeks in May. Check out most of the archives for the tech pump over the last couple years and you'll notice frequent day long outages or week long outages. Of course, this thread isn't a Toolserver bash fest. Toolserver needed hardware and it needed proper staffing.

Tool labs has had relatively few unplanned outages and this particular case was scheduled maintenance that took 11 hours longer than planned due to the amount of data that's being generated inside of the project. If you are interested in scheduled outages because it impacts your work, feel free to subscribe to labs-l for announcements of maintenance.

Neither Toolserver nor Wikimedia Labs are scoped for fully production workloads and there's an expectation in either infrastructure that tools and bots may be unavailable for extended periods of time on occasion.

Regarding your proposal on order of disabling tools: all tools were disabled at the same time because the underlying filesystem used for them was taken down for maintenance. The underlying filesystem was moved to another system to ensure better long-term stability.