Wikimedia Cloud Services team/Onboarding Arturo/Sessions


Arturo and Chase Onboarding Sessions[edit]

Dec 19, 2017[edit]

  • commands agnostic to distro? (upgrades for distro and security and wmf? -- trusty, jessie...stretch)

1001 trusty-wikimedia/thirdparty amd64 Packages

    release v=14.04,o=Wikimedia,a=trusty-wikimedia,n=trusty-wikimedia,l=Wikimedia,c=thirdparty

1001 trusty-wikimedia/universe amd64 Packages

    release v=14.04,o=Wikimedia,a=trusty-wikimedia,n=trusty-wikimedia,l=Wikimedia,c=universe

1001 trusty-wikimedia/main amd64 Packages

    release v=14.04,o=Wikimedia,a=trusty-wikimedia,n=trusty-wikimedia,l=Wikimedia,c=main

1500 trusty-tools/main amd64 Packages

    release o=. trusty-tools,n=trusty-tools,l=. trusty-tools,c=main
500 trusty-security/universe Translation-en
500 trusty-security/main Translation-en
500 trusty-security/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=universe
500 trusty-security/main amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=main
500 trusty-backports/universe Translation-en
500 trusty-backports/restricted Translation-en
500 trusty-backports/multiverse Translation-en
500 trusty-backports/main Translation-en
100 trusty-backports/multiverse amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=multiverse
100 trusty-backports/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=universe
100 trusty-backports/restricted amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=restricted
100 trusty-backports/main amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=main
500 trusty-updates/universe Translation-en
500 trusty-updates/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-updates,n=trusty,l=Ubuntu,c=universe
500 trusty/universe Translation-en
500 trusty/main Translation-en
500 trusty/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=universe
500 trusty/main amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=main

Nov 28, 2017[edit]

  • Travel!
  • going through pending tickets and patches assigned

Package upgrade workflow (

Unattended things:

   * All cloud instances get all unattended upgrades from WMF and distro by default
    - Security updates <-- add a patch (Arturo)
    - distro package upgrades 
    - wmf package upgrades
    :* Add a patch to put this behind a hiera setting (Chase)
    :* kernel updates still sleeping in toolforge (task?)
    :* packages handling configuration files correctly (which means preserving settings)
   - backports is an open question

Choosing to handle updates manually:

   * A project can choose to set a hiera key that will stop these upgrades from happening ( one key per type of upgrade candidate)
   * A script exists to run on an instance to generate a report for available package upgrades. (
   :* Broken down by wmf vs distro?
   * The script that is used to generate the report or another script can be used to do the upgrades. This is a replacment for unattended and is ...attended upgrade solution.

Nov 21, 2017[edit]

root@tools-bastion-03:~# host enwiki.web.db.svc.eqiad.wmflabs enwiki.web.db.svc.eqiad.wmflabs is an alias for s1.web.db.svc.eqiad.wmflabs. s1.web.db.svc.eqiad.wmflabs has address

   source: user_properties
   view: select up_user, up_property, up_value
   where: >
     up_property in ( 'disablemail', 'fancysig', 'gender', 'nickname' )
   limit: 2
   source: ["user_properties", "user", "meta_p.properties_anon_whitelist" ]
   view: select cast(extract(year_month from user_touched)*100+1 as date) upa_touched, up_property, up_value
   where: user_id=up_user and up_property like pw_property

Nov 2, 2017[edit]

  • tools-bastion-03

Recurrent problem.

  • arturo's onboarding page
  • Make a network diagram
  • Openstack: everything is liberty execpt horizon which is mitaka.
  • Wiki replicas <-- look at them.
  • Next week: shadow clinic duty person. Madhu?

Oct 31, 2017[edit]

  • 2017-11-01 is a public holiday for Arturo
    • We should get some/all of these for the next few months on the team calendar
  • Arturo trying to understand which servers are physical, which are virtual, and how they link together
    • Wants a map of how things fit together
  • Nick poked Arturo about setting up his User page on metawiki
  • Chase to find the newly formed ongoing topographical docs
  • Everything is physical *except* Cloud VPS tenents and a few things on Ganeti in "production" <--- KVM + DRBD (NOTE: 2017-10-31: already read the docs)

Oct 26, 2017[edit]

  • topics?
  • I've been working on this task today: nfsiostat diamond collector

To test a patch, depool a node and test in a node:

Oct 24, 2017[edit]

Puppet (how does it work)[edit]

    • LDAP is the "same sign-on" solution for all things that are not MediaWiki
    • Unix user accounts outside of Cloud VPS are not connected directly to LDAP
    • Data is managed by Puppet based on modules/admin/data/data.yaml
  • puppetmaster1001.eqiad.wmnet
    • puppet-merge

<change> y/n?

new installs[edit]

  • New installs

New server: foo.eqiad.wmnet management network: foo.mgmt.eqiad.wmnet management network: <asset tag>.eqiad.wnet == mgmt show system1/network1/Integrated_NICs

files/dhcpd/linux-host-entries.ttyS1-115200:host labcontrol1001 {

   # onboard management

Host *.mgmt.*.wmnet

       StrictHostKeyChecking ask
       UserKnownHostsFile /Users/cpettet/.ssh/wmf_mgmt_hosts

baham.eqiad.wment authdns-update

From pupetmaster1001: new-install <server>


  • Bastions (protected bastion) ()

toolforge <-- own bastion


Cloud VPS project request instructions --

Openstack vs Horizon vs Toolsadmin[edit]