Manual:$wgMaxShellMemory

Details
Under Linux, shell commands are executed with the time and memory limits given by $wgMaxShellTime and $wgMaxShellMemory. Setting either variable to 0 will disable both limits.

If generating thumbnails fails with a message like "Memory allocation failed," this value may need to be increased.

If using Batik to render SVG graphics on some systems, this value needs to be raised in order for the extension to function. (The phrase "Could not create the Java virtual machine." may appear in Apache error log if this problem is occuring.)