Talk:Toolserver:User-store

From mediawiki.org
Latest comment: 10 years ago by Nemobis in topic Stats compression

Update[edit]

It's not clear whether one can create new directories or not here. I thought not, and that you needed to ask permission (and document it here), but I see that the page is outdated. This is the current situation, and some things seem unneeded or misplaced. Nemobis 08:31, 5 February 2012 (UTC) Reply

ls  
$ ls -l
total 53980336
drwxr-sr-x  3 dschwen       users              96 Feb 28  2009 800px-thumbs-commons
drwxr-xr-x  2 alebot        users              96 Sep 20 21:29 Alebot
drwxr-sr-x  3 multichill    users              96 Mar  7  2009 ars_usda_gov
-rw-r--r--  1 enwikt        enwikt       38292936 Sep 23  2010 arwiktionary-20100915-pages-articles.xml
drwxr-xr-x 13 aude          users            1024 Dec 12 04:48 aude
drwxr-xr-x  8 seb35         users            1024 Aug 28  2010 bnf
drwxr-xr-x  4 bryan         users              96 Oct 28  2010 bryan
drwxr-xr-x  6 kaldari       users            1024 Jun  3  2011 bulkuploader
drwxr-xr-x  8 cbm           users            9216 Jul 28  2011 cbm
-rw-r--r--  1 vyznev        users       282934189 Apr 15  2010 commonswiki-20100407-page.sql.gz
drwxr-xr-x 14 contests      contests         1024 Nov 12 07:53 contests
-rw-r--r--  1 hippietrail   users            7636 Apr 19  2011 cs20100831-all-idx.raw
-rw-r--r--  1 hippietrail   users            7636 Apr 19  2011 cs20100831-all-off.raw
-rw-r--r--  1 hippietrail   users           76187 Apr 19  2011 cs20100831-all.txt
-rw-r--r--  1 hippietrail   users           22908 Apr 19  2011 cs20100831-off.raw
lrwxrwxrwx  1 hippietrail   users              59 Apr 19  2011 cswikinews-20100831-pages-articles.xml -> /mnt/user-store/dump/cswikinews-20100831-pages-articles.xml
-rw-r--r--  1 mzmcbride     users             405 Nov 10  2010 curltest.bz2
drwx--x---  5 daniel        users            1024 Jan 12  2011 daniel
-rw-r--r--  1 jsonntag      users      9559946161 Jul  4  2011 dewiki-20110621-pages-meta-history.xml.7z
drwxr-xr-x  7 harddisk      users            1024 Oct 31  2010 dewiki_static
-rw-r--r--  1 balu          users         1489819 Jan 17  2011 dewiktionary-latest-all-titles-in-ns0
-rw-r--r--  1 balu          users       214945515 Jan 14  2010 dewiktionary-latest-pages-articles.xml
drwxrwxr-x  2 dispenser     users            3072 Dec 23 16:01 dispenser
drwx--S---  3 drh08         users              96 Aug 18  2009 drh08
drwxrwxrwx  3 sk            dumps           40960 Feb  5 08:13 dump
drwxrwxrwx 30 mzmcbride     dumps            1024 Feb  4 21:14 dumps
drwxr-xr-x  2 emijrp        users           24576 Feb  3 14:23 emijrp
lrwxrwxrwx  1 hippietrail   users              34 Feb 25  2011 enlatest-all.txt -> /mnt/user-store/en20110205-all.txt
lrwxrwxrwx  1 hippietrail   users              18 Oct  4  2009 enprevious-all.txt -> en20091003-all.txt
-rw-r--r--  1 emijrp        users       693939457 Oct 28  2010 enwiki-latest-categorylinks.sql.gz
-rw-r--r--  1 emijrp        users       244646437 Oct 28  2010 enwiki-latest-imagelinks.sql.gz
-rw-r--r--  1 enwikt        enwikt     2085194006 Oct  9 05:02 enwiktionary-20111008-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2090210148 Oct 17 05:04 enwiktionary-20111016-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2100781115 Oct 25 05:08 enwiktionary-20111024-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2105183139 Nov  3 05:02 enwiktionary-20111102-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2123503348 Nov 13 05:02 enwiktionary-20111112-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2118245759 Nov 22 05:02 enwiktionary-20111121-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2139398338 Nov 30 05:02 enwiktionary-20111129-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2153787519 Dec  9 05:02 enwiktionary-20111208-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2164077241 Dec 17 05:02 enwiktionary-20111216-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2173188530 Dec 25 05:02 enwiktionary-20111224-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2185108737 Jan  2 05:02 enwiktionary-20120101-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2196287047 Jan 10 05:02 enwiktionary-20120109-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2207838421 Jan 18 06:53 enwiktionary-20120117-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2214930203 Jan 26 05:02 enwiktionary-20120125-pages-articles.xml
-rw-r--r--  1 enwikt        enwikt     2220726357 Feb  4 05:02 enwiktionary-20120203-pages-articles.xml
-rw-r--r--  1 balu          users        25300829 Jan 17  2011 enwiktionary-latest-all-titles-in-ns0
drwxr-xr-x  4 enwp10        enwp10           1024 Feb  1  2011 enwp10
drwxr-sr-x  3 multichill    users              96 Apr 21  2009 eol-jsc-nasa
-rw-r--r--  1 emijrp        users       256896618 Jul  4  2010 eswiki-latest-pagelinks.sql.gz
drwxr-xr-x  4 multichill    users              96 Aug 15  2010 geograph
drwxr-xr-x  3 multichill    users              96 Nov 22 21:56 geograph_new
drwxr-xr-x  2 giovanni      users            1024 Dec  8 23:46 giovanni
drwxr-xr-x  2 magnus        users           49152 Feb  5 00:57 glam_page_views
drwxr-xr-x  5 grphack       users            1024 Jan 16 18:23 grphack
drwxr-xr-x  2 grphack       users            1024 Jan 16 18:23 halfak
drwxr-xr-x  4 hoo           users            1024 Dec  5 18:17 hoo
drwxr-xr-x  3 hydriz        users            1024 Feb  5 07:45 hydriz
drwxrwxr-x  2 dschwen       users          966656 Feb  5 00:13 iip-cache-dschwen
-rwxrwxrwx  1 harddisk      users             838 Oct 19 18:22 INDEX
drwxr-xr-x  3 johang        users            1024 Jan 29 23:13 johang
drwxr-xr-x  3 kolossos      users              96 Dec  4 11:47 kolossos
drwxr-xr-x  8 magnus        users           14336 Feb  4 20:43 magnus
drwxr-xr-x  3 master        users              96 Feb  5 04:36 master
drwxr-xr-x  2 mazder        users            1024 Dec  5  2010 mazder
-rw-r--r--  1 cbm           users            9671 Dec 15  2009 md5.c
drwxr-xr-x 45 saper         users            2048 Dec 27 13:50 mediawiki
drwxr-xr-x  3 saper         users              96 May 10  2010 MediaWiki-import
drwxrwxr-x  3 merl          merliwbot       12288 Feb  5 01:19 MerlIwBot
drwxr-xr-x  2 messedrocker  users            2048 Dec 30  2010 messedrocker
drwxr-xr-x  2 mzmcbride     users            1024 Jun 26  2011 mzmcbride
drwxrwxr-x  3 grphack       users              96 Apr 21  2011 newbies
drwxr-xr-x 16 h4ck3rm1k3    users            1024 Jan 23  2011 osmgit
drwxr-xr-x 86 cmarqu        users            4096 Nov 28  2009 osm_hillshading
drwxr-xr-x  3 cmarqu        users              96 Aug 18  2009 osm_overlays
drwxr-xr-x  2 apmon         users              96 Dec  1 22:19 osm_planet
drwxr-xr-x  9 multichill    users            9216 Jun  9  2010 OS_OpenData
drwxrwxr-x  2 dschwen       users           65536 Jan 29 17:58 pano-cache-dschwen
drwxr-xr-x 18 bryan         users            2048 Nov  9  2009 phase3
drwxr-xr-x  3 prolineserver users              96 Sep  5 14:30 prolineserver
drwxr-xr-x  4 valhallasw    users              96 Nov 10 17:51 pywiki
-rwxrwxrwx  1 mzmcbride     users             269 Aug  9  2010 README
drwxr-xr-x  2 whym          users            1024 Oct 11 09:19 revision_diffs_20110405
-rw-r--r--  1 schutz        users              89 Aug 27  2006 robots.txt
drwxr-xr-x  2 russell       users              96 Sep 20 13:52 russell
drwxr-xr-x  3 sk            users              96 Feb  4 09:52 sk
drwxr-sr-x 41 schutz        users         1007616 Feb  5 08:05 stats
drwxr-xr-x  2 harddisk      users              96 Sep 26  2010 tar_test
drwxr-xr-x  5 tim1357       users            1024 Dec  9 23:49 Tim1357
-rw-r--r--  1 overlordq     users         3345234 Mar 31  2009 torServerDirectory
-rw-r--r--  1 root          root              116 Dec 19 20:39 UcliEvt.log
drwxr-xr-x  3 multichill    users              96 Nov  8  2009 wdl
drwxr-xr-x  2 whym          users              96 Jul 13  2011 whym
drwxr-xr-x  3 erfgoed       erfgoed          1024 Oct 30 22:21 Wiki Loves Monuments
drwxr-xr-x  4 dschwen       users              96 Jan 23 23:09 wikiminiatlas
drwxr-xr-x  3 darkdadaah    users              96 Oct 28 15:10 Wiktionnaire

The biggest public dirs currently are:

du  
3.2G    aude
39G     contests
2.2G    dispenser
1.3G    dpl
2.0T    dumps
5.0G    emijrp
7.8G    enwp10
4.9G    git
8.9G    h4ck3rm1k3
2.0G    hippietrail
1.4T    iip-cache-dschwen
14G     jkroll
50G     johang
23G     kolossos
3.3G    krinkle
13G     liangent
7.8G    magnus
2.3G    marco74
1.7G    mauro742
6.7G    mediawiki
1.5G    osmgit
226G    osm_hillshading
49G     OS_OpenData
6.5G    pano-cache-dschwen
17G     phe
3.6G    prolineserver
43G     render
70G     sk
197G    stats
4.7G    svick
208K    Tim1357
2.1G    whym
21G     Wiki Loves Monuments
261G    wikiminiatlas
2.9G    Wiktionnaire
5.1G    wiwosm
4.5T    total


Stats compression[edit]

A huge portion of the space is taken by visitors stats, although now they have two mirrors (WMF and IA). The oldest ones are compressed in LZMA (xz). Compressing gz or xz is useless, can only increase size. I made some tests of compression of a whole month uncompressed, 2011-03-pagecounts (184G):

  • 7z a -t7z -m0=BZip2 -mmt=6 -mx9 takes ~27h (6 cores, less than 100M memory) and gives 41G
  • 7z a -t7z -m0=LZMA -mmt=on -mx9 -md=64m -mfb=64 takes ~56h (2 cores, 800M memory) and gives 37G
  • 7z a -t7z -m0=LZMA -mmt=on -mx9 -md=256m -mfb=64 -ms=on takes about 3 days (2 cores, 2700M memory) and gives 35G
  • tar with xz uses LZMA with standard settings and can only give worse results (I tried it but it got killed by mistake, wasn't going anywhere though)
  • individual gz are 51.2G
  • individual xz of this month are not yet available for comparison

--Nemo 10:23, 22 March 2012 (UTC)Reply

Maybe it would be worth trying xz with higher compression, but I can't locate the stats files on user-store right now. --Nemo 14:02, 20 October 2013 (UTC)Reply