This week is primarily focused on stability in anticipation of a V1.009 release, and I have returned to more system memory optimizations to ensure the engine does not crash due to insufficient wiggle room. It took most of the day of painstaking investigation, but I managed to save over 411MB of system memory from squander, which is now available to the rest of the engine for more antics, including lightmapping and larger levels.
I was helped in my quest by a tool called VMMap which does a splendid job of illustrating where ALL the system memory is being spent. Even things you have no control over are displayed, and from this information I was able to make some good savings.
Once such site of squander was the inclusion of an old DBP DLL called ConvMDL which was once used to load MDL files (from the old Quake days). This innocuous DLL was eating 125MB of static memory until I excluded it from all further action in this project, along with support for 3DS, MD2 and MD3 :) Other savings (and performance boosts) were found in in the terrain visual and collision systems which together gobbled over 600MB (now more like 300MB). I dare say more savings can be found but right now it's about overall stability, and if the engine can now run on 32 bit machines without having to reset every five minutes then I will be a happy chappy.
I have noticed my 'super terrain texture' generator eating 80MB of system memory which in fact it's job is to produce a GPU video memory texture, so that's something to investigate tomorrow me thinks. Hopefully as more system memory is rescued, the engine will gain more stability and longer term will allow MUCH larger levels to be created without worrying about the dreaded red screen of death.
It's not all plane sailing though, in making my massive cut backs, my pre-baked terrain shadows have gone walkabout so Tuesday will involve a hunt for these pesky shadows and to put them back!