SQL/XML Dumps/Puppet for dumps maintainers

This is a companion document to the slides from the Puppet for dumps maintainers presentation on December 9, 2020.

Let's have a thorough look at the file modules/dumps/manifests/generation/server/dirs.pp.

Files of puppet code are called "manifests", so we'll call them that too.

{{CodeCommentary|type=code|lang=python|start=8|icon=hand|pos=2|link=https://github.com/wikimedia/puppet/blob/production/modules/dumps/manifests/server_dirs.pp|content= ) {   class {'dumps::server_dirs':        datadir         => $datadir,        xmldumpsdir     => $xmldumpsdir,        miscdatasetsdir => $miscdatasetsdir,        user            => $user,        group           => $group,    }

# Directories where dumps of any type are generated # This list is not for one-off directories, nor for # directories with incoming rsyncs of datasets $cirrussearchdir             = "${miscdatasetsdir}/cirrussearch" $xlationdir                  = "${miscdatasetsdir}/contenttranslation" $categoriesrdfdir            = "${miscdatasetsdir}/categoriesrdf" $categoriesrdfdailydir       = "${miscdatasetsdir}/categoriesrdf/daily" $globalblocksdir             = "${miscdatasetsdir}/globalblocks" $medialistsdir               = "${miscdatasetsdir}/imageinfo" $incrsdir                    = "${miscdatasetsdir}/incr" }}

{{CodeCommentary|type=code|lang=python|start=34|content= # top level directories for various dumps/datasets, on generation hosts only file { $tempdir: ensure => 'directory', mode  => '0755', owner => $user, group => $group, }

# subdirs for various generated dumps file { [ $cirrussearchdir, $xlationdir, $categoriesrdfdir, $categoriesrdfdailydir, $globalblocksdir, $medialistsdir, $incrsdir, $mediatitlesdir, $pagetitlesdir, $shorturlsdir, $machinevisiondir ]:

ensure => 'directory', mode  => '0755', owner => $user, group => $group, }

# needed for wikidata weekly crons file { [ $otherwikibasedir, $otherwikibasewikidatadir, $otherwikidatadir ]: ensure => 'directory', mode  => '0755', owner => $user, group => $group, } } }}