Manual:$wgMaxGeneratedPPNodeCount

Details
A complexity limit on template expansion: the maximum number of nodes generated by  

This allows you to limit the amount of memory used by the Preprocessor_DOM node cache: testing indicates that each element uses about 160 bytes of memory on a 64-bit processor, so this default corresponds to about 155 MB.