It took many hours of torture but I managed to wrangle the shader and real time light mapper so they cooperated. They now exist happily together and although my initial shader is not up to Reloaded standards, it's enough to demonstrate that it works.
The shader you can see creating a specular reflection off the wet floor and the light map is basically hiding around the scene. You can tell good light mapping when you cannot see it ;) Again I apologise for FPSC classic graphics used in this screen shot, please forgive me.
I also discovered the cause of the crash, and it was nothing more or less than the total lack of virtual address space. Yes friendly, the 1.8GB memory limit strikes again! Arg!
Okay, so this time rather than put an error message in there saying "make a smaller level" I have decided to tackle it head on and see what I can do. I have started the process by investigating exactly where the memory is being spent and you might be quite surprised.
The first 100MB is just the basic map editor, graphics and reserved arrays. That's fine and only represents 5% of the total spend, not bad for a foundation. It's all downhill from there though. Each small segment takes as little as 4Kb but the largest 'WW2 Cellar Segment Full' takes up 2.4Mb. Amazing huh! I did some initial digging and a function called CloneDBO is the cause for most of this, and then the 'Sync' of course which creates any new VBO objects for DirectX. Every byte is used and nothing is really wasted so it is not so much about fixing a bug but figuring a better way to do this. I had the idea of cloning all the repeated segment and entity data, and only holding the unique second UV layer for the texture lightmapping but that would be a bit of engineering to sort out. It WOULD solve the issue though so it is still on my 'possible' radar. The issue remains unresolved as of right now though.
Good Work Though
I did not spend the day making the PC crash and counting how big the memory usage was. Another crash was caused by the light mapper taking too many segments to light. If you give it a few hundred segments, and each segment with many polygons can take up to 20MB to store the light calculations, it soon eats through the 1.8GB barrier.
I solved this by slicing up the lighting task into 100 object chunks. Not scientific and I plan to measure and store the size of each segment to make this a more accurate lighting limit. For now it solves the crash due to too much lighting going on, and at least in this scenario you won't crash at the light mapping stage. This solves the classic issue with the original FPSC engine. Phew!
I just have to work out how to crunch the size of individual cloned segments and entities now. There is always a way, I just have to wait for my brain to tell me what it is.
All in all a good day, some nice looking renders now with the shader (from certain angles). I wanted to delve into a more sophisticated shader, but there will be time for that. For now, I want to reduce the memory usage to reasonable levels and then jump into player controls and physics. Cosmetics are critically important, but they can wait until the real work is done ;)