SVG benchmarks

From mediawiki.org

For rasterizing SVG images we've used librsvg for some time. Another contender which we didn't choose originally is Batik. There are several respects in which librsvg is unsatisfactory and Batik might be preferable.

librsvg: In favour: fast; totally free/libre. Against: Lots of GNOME-ish dependencies, which makes tracking updates difficult; We have to maintain a security patch to disable the loading of external URL resources, since librsvg maintainers weren't interested in it; Lots of rendering bugs (but it's improved over time); Unclear fonts/text rendering situation.

Batik: In favour: more mature; higher quality rendering; built-in security mode. Against: slow!; doesn't (or at least didn't) run on free/libre Java (GCJ etc).

Benchmark[edit]

Brion did a quick test of SVG rendering (19 Dec 2006), with 50 images pulled off Commons. Test run on a MacBook, 2 GHz Intel Core 2 Duo, in Ubuntu running under Parallels (using one virtual processor). rsvg is 2.16.0 as apt-get'd via Ubuntu; batik is 1.6, running on Sun's J2SE 6 runtime.

rsvg: 18.07 seconds; 361.46ms per image
batik: 108.44 seconds; 2168.84ms per image

Batik is probably slowed down by having to start up the VM and JIT on every call. It likely can be sped up by using a daemon process and handing off render requests to that.

Batik also refused to render one file that rsvg did render.

ivar notes that to avoid JDK startup time you can use nailgun.

Additional notes[edit]

  • Batik command-line renderer appears to scale image height to be proportional with the given width. This could produce width/height inconsistencies between specified heights from MW and results.
  • I'm pretty sure Batik has a mode for increased safety (rejecting external links), which we have to maintain as a patch hack for librsvg. Not having to maintain a patch is attractive.
  • We frequently get requests about font support. It'd be nice to know how to answer such questions.
  • rsvg's resource usage can be limited with ulimit etc easily; what about a batik daemon? How to stop giiaaannttttt files from eating up RAM without taking down the server?
  • You'd have to look at jvm args... -- chris

Further testing[edit]

2008 (David Gerard,57 images)[edit]

David Gerard ran a test of ImageMagick 6.2.4, Inkscape 0.45.1, Batik 1.7.0 (on Sun Java 1.6.0_03-b05) and rsvg 2.18.2 on 57 Commons images, including some in the megabytes. Those which reliably gave an error from the renderers were left out. Test box was a Dell Latitude D610, Pentium M 1.6 GHz, 512MB memory, running Kubuntu 7.10. All test applications except Batik were Ubuntu repository versions.

All four programs were run through the 57 images 115 times and timed with /usr/bin/time. X running in Failsafe (konsole as only app), nothing else running.

median real mean real median CPU mean CPU
Batik 139.71 139.95 137.02 137.04
ImageMagick 112.43 112.67 111.7 111.73
Inkscape 51.43 51.52 51.18 51.2
rsvg 25.97 26.03 25.8 25.8

Worrying:

  • Batik's JVM crapped itself several times for no apparent reason - 39 times in the run, i.e. 39 in 4485 attempted renderings. Not on any particular SVG either.
  • Inkscape thrashed the disk. (Single physical disk in laptop.)

(Possible explanation: that rsvg is a lib whereas Inkscape is a full app - could the overhead of starting and stopping the app be significant?)

What this suggests: use rsvg if possible (anything Unix), use Inkscape if rsvg's dependencies are too troublesome (e.g. on Windows, where Inkscape is available as a standalone app) and you have a lot of memory.

See /test 20080112 for full data.

2009 (Hk kng;56 files)[edit]

Hk kng re-run the tests in 2009 using a http server solution for Batik: batikd. Full test results are at /test 20090816. WikiMedia Commons presents a gallery of the test images.

A comparison of CPU times is not as easily done, since the real image conversion for Batik is done in the separate server process. The wallclock times look like this:

Program Median time
Batik 57.76 s
ImageMagick 81.86 s
Inkscape 70.95 s
rsvg 41.48 s

2021 (JoKalliauer;337+1338+141files, including correcntess-ratio)[edit]

User:JoKalliauer tests several sperate test suites in 2021. And has detailed explanations about renderingissues, see c:User:JoKalliauer/SVG_test_suites for details.

SVG librsvg 2.50 resvg 0.14.0 Inkscape 1.0 batik 1.13; 1.14
W3C correctness 0,662 0,831 0,745 0,801
W3C time 7m11,028s 0m50,188s 16m7,465s 30m9,981s
RazrFalcon correctness 0.754 0.956 0.729 0.703
RazrFalcon time 4min 05sek 2min 30sek 46min 22sek 61min 29sek
featured correctness 0.92 1.00
featured time 5m 17,701s 4m 46,639s 15m 28,202s 11m 30,768s