Talk:Toolserver:User-store

Update
It's not clear whether one can create new directories or not here. I thought not, and that you needed to ask permission (and document it here), but I see that the page is outdated. This is the current situation, and some things seem unneeded or misplaced. Nemobis 08:31, 5 February 2012 (UTC) $ ls -l total 53980336 drwxr-sr-x 3 dschwen       users              96 Feb 28  2009 800px-thumbs-commons drwxr-xr-x 2 alebot        users              96 Sep 20 21:29 Alebot drwxr-sr-x 3 multichill    users              96 Mar  7  2009 ars_usda_gov -rw-r--r-- 1 enwikt        enwikt       38292936 Sep 23  2010 arwiktionary-20100915-pages-articles.xml drwxr-xr-x 13 aude         users            1024 Dec 12 04:48 aude drwxr-xr-x 8 seb35         users            1024 Aug 28  2010 bnf drwxr-xr-x 4 bryan         users              96 Oct 28  2010 bryan drwxr-xr-x 6 kaldari       users            1024 Jun  3  2011 bulkuploader drwxr-xr-x 8 cbm           users            9216 Jul 28  2011 cbm -rw-r--r-- 1 vyznev        users       282934189 Apr 15  2010 commonswiki-20100407-page.sql.gz drwxr-xr-x 14 contests      contests         1024 Nov 12 07:53 contests -rw-r--r-- 1 hippietrail   users            7636 Apr 19  2011 cs20100831-all-idx.raw -rw-r--r-- 1 hippietrail   users            7636 Apr 19  2011 cs20100831-all-off.raw -rw-r--r-- 1 hippietrail   users           76187 Apr 19  2011 cs20100831-all.txt -rw-r--r-- 1 hippietrail   users           22908 Apr 19  2011 cs20100831-off.raw lrwxrwxrwx 1 hippietrail   users              59 Apr 19  2011 cswikinews-20100831-pages-articles.xml -> /mnt/user-store/dump/cswikinews-20100831-pages-articles.xml -rw-r--r-- 1 mzmcbride     users             405 Nov 10  2010 curltest.bz2 drwx--x--- 5 daniel        users            1024 Jan 12  2011 daniel -rw-r--r-- 1 jsonntag      users      9559946161 Jul  4  2011 dewiki-20110621-pages-meta-history.xml.7z drwxr-xr-x 7 harddisk      users            1024 Oct 31  2010 dewiki_static -rw-r--r-- 1 balu          users         1489819 Jan 17  2011 dewiktionary-latest-all-titles-in-ns0 -rw-r--r-- 1 balu          users       214945515 Jan 14  2010 dewiktionary-latest-pages-articles.xml drwxrwxr-x 2 dispenser     users            3072 Dec 23 16:01 dispenser drwx--S--- 3 drh08         users              96 Aug 18  2009 drh08 drwxrwxrwx 3 sk            dumps           40960 Feb  5 08:13 dump drwxrwxrwx 30 mzmcbride    dumps            1024 Feb  4 21:14 dumps drwxr-xr-x 2 emijrp        users           24576 Feb  3 14:23 emijrp lrwxrwxrwx 1 hippietrail   users              34 Feb 25  2011 enlatest-all.txt -> /mnt/user-store/en20110205-all.txt lrwxrwxrwx 1 hippietrail   users              18 Oct  4  2009 enprevious-all.txt -> en20091003-all.txt -rw-r--r-- 1 emijrp        users       693939457 Oct 28  2010 enwiki-latest-categorylinks.sql.gz -rw-r--r--  1 emijrp        users       244646437 Oct 28  2010 enwiki-latest-imagelinks.sql.gz -rw-r--r--  1 enwikt        enwikt     2085194006 Oct  9 05:02 enwiktionary-20111008-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2090210148 Oct 17 05:04 enwiktionary-20111016-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2100781115 Oct 25 05:08 enwiktionary-20111024-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2105183139 Nov  3 05:02 enwiktionary-20111102-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2123503348 Nov 13 05:02 enwiktionary-20111112-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2118245759 Nov 22 05:02 enwiktionary-20111121-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2139398338 Nov 30 05:02 enwiktionary-20111129-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2153787519 Dec  9 05:02 enwiktionary-20111208-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2164077241 Dec 17 05:02 enwiktionary-20111216-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2173188530 Dec 25 05:02 enwiktionary-20111224-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2185108737 Jan  2 05:02 enwiktionary-20120101-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2196287047 Jan 10 05:02 enwiktionary-20120109-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2207838421 Jan 18 06:53 enwiktionary-20120117-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2214930203 Jan 26 05:02 enwiktionary-20120125-pages-articles.xml -rw-r--r-- 1 enwikt        enwikt     2220726357 Feb  4 05:02 enwiktionary-20120203-pages-articles.xml -rw-r--r-- 1 balu          users        25300829 Jan 17  2011 enwiktionary-latest-all-titles-in-ns0 drwxr-xr-x 4 enwp10        enwp10           1024 Feb  1  2011 enwp10 drwxr-sr-x 3 multichill    users              96 Apr 21  2009 eol-jsc-nasa -rw-r--r-- 1 emijrp        users       256896618 Jul  4  2010 eswiki-latest-pagelinks.sql.gz drwxr-xr-x  4 multichill    users              96 Aug 15  2010 geograph drwxr-xr-x 3 multichill    users              96 Nov 22 21:56 geograph_new drwxr-xr-x 2 giovanni      users            1024 Dec  8 23:46 giovanni drwxr-xr-x 2 magnus        users           49152 Feb  5 00:57 glam_page_views drwxr-xr-x 5 grphack       users            1024 Jan 16 18:23 grphack drwxr-xr-x 2 grphack       users            1024 Jan 16 18:23 halfak drwxr-xr-x 4 hoo           users            1024 Dec  5 18:17 hoo drwxr-xr-x 3 hydriz        users            1024 Feb  5 07:45 hydriz drwxrwxr-x 2 dschwen       users          966656 Feb  5 00:13 iip-cache-dschwen -rwxrwxrwx 1 harddisk      users             838 Oct 19 18:22 INDEX drwxr-xr-x 3 johang        users            1024 Jan 29 23:13 johang drwxr-xr-x 3 kolossos      users              96 Dec  4 11:47 kolossos drwxr-xr-x 8 magnus        users           14336 Feb  4 20:43 magnus drwxr-xr-x 3 master        users              96 Feb  5 04:36 master drwxr-xr-x 2 mazder        users            1024 Dec  5  2010 mazder -rw-r--r-- 1 cbm           users            9671 Dec 15  2009 md5.c drwxr-xr-x 45 saper         users            2048 Dec 27 13:50 mediawiki drwxr-xr-x 3 saper         users              96 May 10  2010 MediaWiki-import drwxrwxr-x 3 merl          merliwbot       12288 Feb  5 01:19 MerlIwBot drwxr-xr-x 2 messedrocker  users            2048 Dec 30  2010 messedrocker drwxr-xr-x 2 mzmcbride     users            1024 Jun 26  2011 mzmcbride drwxrwxr-x 3 grphack       users              96 Apr 21  2011 newbies drwxr-xr-x 16 h4ck3rm1k3   users            1024 Jan 23  2011 osmgit drwxr-xr-x 86 cmarqu       users            4096 Nov 28  2009 osm_hillshading drwxr-xr-x 3 cmarqu        users              96 Aug 18  2009 osm_overlays drwxr-xr-x 2 apmon         users              96 Dec  1 22:19 osm_planet drwxr-xr-x 9 multichill    users            9216 Jun  9  2010 OS_OpenData drwxrwxr-x 2 dschwen       users           65536 Jan 29 17:58 pano-cache-dschwen drwxr-xr-x 18 bryan        users            2048 Nov  9  2009 phase3 drwxr-xr-x 3 prolineserver users              96 Sep  5 14:30 prolineserver drwxr-xr-x 4 valhallasw    users              96 Nov 10 17:51 pywiki -rwxrwxrwx 1 mzmcbride     users             269 Aug  9  2010 README drwxr-xr-x 2 whym          users            1024 Oct 11 09:19 revision_diffs_20110405 -rw-r--r-- 1 schutz        users              89 Aug 27  2006 robots.txt drwxr-xr-x 2 russell       users              96 Sep 20 13:52 russell drwxr-xr-x 3 sk            users              96 Feb  4 09:52 sk drwxr-sr-x 41 schutz        users         1007616 Feb  5 08:05 stats drwxr-xr-x 2 harddisk      users              96 Sep 26  2010 tar_test drwxr-xr-x 5 tim1357       users            1024 Dec  9 23:49 Tim1357 -rw-r--r-- 1 overlordq     users         3345234 Mar 31  2009 torServerDirectory -rw-r--r-- 1 root          root              116 Dec 19 20:39 UcliEvt.log drwxr-xr-x 3 multichill    users              96 Nov  8  2009 wdl drwxr-xr-x 2 whym          users              96 Jul 13  2011 whym drwxr-xr-x 3 erfgoed       erfgoed          1024 Oct 30 22:21 Wiki Loves Monuments drwxr-xr-x 4 dschwen       users              96 Jan 23 23:09 wikiminiatlas drwxr-xr-x 3 darkdadaah    users              96 Oct 28 15:10 Wiktionnaire

Stats compression
A huge portion of the space is taken by visitors stats, although now they have two mirrors (WMF and IA). The oldest ones are compressed in LZMA (xz). Compressing gz or xz is useless, can only increase size. I made some tests of compression of a whole month uncompressed,  (184G): --Nemo 10:23, 22 March 2012 (UTC)
 * takes ~27h (6 cores, less than 100M memory) and gives 41G
 * takes ~56h (2 cores, 800M memory) and gives 37G
 * takes about 3 days (2 cores, 2700M memory) and gives 35G
 * tar with xz uses LZMA with standard settings and can only give worse results (I tried it but it got killed by mistake, wasn't going anywhere though)
 * individual gz are 51.2G
 * individual xz of this month are not yet available for comparison