Wednesday 26 June 2013

Wednesday's Child

A Quick Wave

Today I am packing, printing out boarding passes and making sure I have everything for my vacation. I think I have delegated all my interests over the next two weeks so this will be my final blog post for a while. I had planned to blog every work day of 2013 but I did not think about holidays :)  I will make up for it on my return with more cool videos and perhaps a few betas for you lovely pledger's!

Signing Off

As a farewell for the moment, I have linked a video that made me smile and was also informative to existing FPSC users:

http://www.youtube.com/watch?v=-3ChyudNMLI

Until my return, good luck with your projects and see you soon!

Great News! AGK V2 Funded!

To our delight and surprise, the AGK V2 Kickstarter project was entirely funded in less than 24 hours!  It looks like Kickstarter users are now aiming for one or more of the stretch goals, and looks like Paul will have his work cut out :) Here is the link to the Kickstarter page:

http://www.kickstarter.com/projects/tgc/app-game-kit-v2

I will be sure to check on this from time to time when I'm in WiFi range, as I am morbidly curious as to just how far this project will be 'stretched' ;)  If you pledged as a direct result of my plea, my extra special thanks to thee!

Tuesday 25 June 2013

Tuesday's Child

Cleaning Up My Brain

I am finding my run up to the holidays a time for backing up, sorting out my desk and generally getting all those loose ends tied off.  Part of that is to assign some jobs to my team mates before I swan off to hotter climbs in a few days.  I wanted to let you in on some internal inspiration material I have passed about which should give you an idea of the quality bar we are setting ourselves:



Just to be clear, this is not another preview of Reloaded :)  It is a game currently in development and set to be released for the XBOX ONE. Having created our tracer and explosion prototypes, we confidently moved onto pastures new. Having seen the above video, we promptly decided to backtrack and see if we could not make our tracer, explosions and fire modules a little bit better.

Bearing in mind the above production took hundreds of talented dudsters and probably tens of thousands of person-hours, you might think it quite insane to even dream of getting near this level of quality. I believe that having a mentor is a good thing, and if you're going to be mentored, it may as well be the best you can find.

Reloaded Work

Apart from cleaning, packing, thinking and delegating, I will be cramming in some Reloaded work this evening when I attempt to build up the ODE replacement to the physics engine. Using the all powerful Bullet Physics, I am hopeful to add enough commands to the 'Bullet Physics Plugin for DBP' to be able to add terrain physics and character controller physics to the prototype you saw in the video yesterday.  I think it makes sense to do this as my return in 2 weeks will see me start the terrain module of the engine which will add deformable terrain geometry and variable vegetation to the level system. In order to be able to test the result of this, having a physics system in place to allow my player to walk around the terrain would be a great benefit and also tick off one of those grey area boxes that asks 'can I generate terrain on the fly, and also generate the physics element of the terrain as well'.

A Head Full Of Oculus

I finally found time last night to unbox and set-up my Oculus VR headset, mainly to tick it off my list and see what the fuss was about. Now I know what the fuss was about and it deserves to be fussed about. I have tried VR headsets before and even have a product out right now that uses what I considered the best consumer device for VR. Well the script has been permanently rewritten with the Oculus Rift.  I will not wax lyrical over it as you can find plenty of that elsewhere on the Inter-web, and I will be putting it back in it's box now to prevent myself from being too distracted from my mission to complete Reloaded.  

What I can say is that combining this hardware with Reloaded would be momentous!  As I sat there, wearing the device, now standing on a balcony overlooking a Mediterranean sea with the soft wind blowing sugar stealer's around my head, calling up an on-screen HUD which magically floated there in front of my eyes, I started to imagine a product that allowed you to create worlds with a wave of your hand, the sound of your voice and a gentle nod of the head. For the first time I saw how a real holo-deck could be built and it was good.

Alas, I must put aside these flights of fancy and get my head back in the game. We have a killer product to produce and it won't write itself (yet)!

Signing Off

As an aside to my regular blog and sign off, you will find over the next 44 days an extra section which you will find below.  As you may know, the Reloaded project owes a lot to the Kickstarter concept, which indirectly breathed life into the idea by finding us a champion investor and we are now knee deep in development goodness. My colleague who now runs the AGK Development effort needs a similar boost, and we have chosen Kickstarter to fund that acceleration. If you can help him get those early pledges, even if it's just £3, it should put us on the Kickstarter radar and get some good exposure during the campaign.

Our Kickstarter Campaign

Rather than a banner and a link, I thought it better if I actually say something about what this evolved AGK product is about.  But first, here is a banner and a link:



As you can see, thanks to a strong AGK community, we've already pushed past the half way point and we only launched it this morning :)  I think what makes AGK so special is that it has all the ease and friendliness of Dark Basic, combining some of the best cross platform technology and made accessible by it's relatively low price and awesome community. 

It ain't no toy either! The last thing I made in AGK is now charting across multiple platforms, is battling healthily in the Educational paid top ten on iOS and is raking in a truck of money every week (admittedly for the publisher, not us poor impoverished developers) :) We've already had number one positions in the free charts, and that was before V108 added over a hundred new commands.  I am looking forward to seeing what Paul does with this accelerator funding, and especially keen to see what the AGK community creates with this awesome development tool, especially on the new platforms such as Ouya!

Monday 24 June 2013

Monday's Child

The Tease Is Over

I know I have been teasing over this video for a few days now, so let me stop the silliness and show you a video of the demo I have been mucking about with over the weekend. Big thanks to Mark B. who really helped me pull this visual prototype together in such a short space of time.


As you can see, we have a few things going on here. Parallax mapping on the sand, objects, buildings and walls. Soft shadow mapping on everything, including the players weapon. The weapon has all manner of subtle and clever effects to bring it to life including cube mapping for the reflective lens. The sky uses real cloud generation and you can see some experimental grass on the outside of the compound.  Floor vegetation really brings out the terrain and we'll be finding ways to make this look great and render fast.

More Than Candy

Fear not, this was not a quick demo to show off some eye candy, but to get to grips with the format and scope required for the art assets. Before we can produce more buildings, objects and scenery, we need to know the demands of the shader and renderer to see where our bottlenecks and limits are. Already we have discovered you cannot cast real shadows on a field of grass using a forward renderer due to the insane amount of overdraw and pixel shader meltdown.

The more we study the demands on the shader, the more deferred rendering makes sense, and is clearly indicated as something that will need prototyping to see if we can get a sizeable performance gain.  Don't worry about the 30fps frame-rate, fraps was taking it down from a VSYNC 60 FPS. I have higher rates, but I found creating the DirectX device with VSYNC produces a very smooth experience where something at 70-80 fps looked ever so slightly choppy on close examination.

Next Steps

I am off on vacation for two weeks in a few days, so I am tinkering with what the best thing for me to wrap up with. I have a basic Bullet prototype working with terrain and a kinematic player controller, so I might move that into the prototype, or finish the grass shader, or add some particle experiments such as dust bowls, smoke and atmospheric effects.

My brain is a little fuzzed from the half day meeting (and what feels like all day talking) so I will make the above decisions on Tuesday when my head is clear and I've got the hunger for code again.  Right now I have the hunger for sleep and I really ought to think about packing (or at the very least looking for a suitcase).

Signing Off

Let me know if you spot anything horrid in the video. There are a few shadow artefacts at the extreme edges of the render (due to the cascade scope not accounting for the wide field of view being used here), and don't comment on the grass yet as I was in the middle of integrating it when I found every blade was an intense expense in processing power.  I've also seen some grass in a recent you tube video that really crammed it in and made it look good, so I am also going to try to ramp it up and cover the whole terrain, then see what looks like using a basic shader.

Also an open question to you guys and gals. Would you prefer us to work on one genre only, and provide a very high level of quality and design finish on those assets, or have the work spread over a few genre's during the initial release. Of course we will be producing dedicated model 'super' packs to explore genre's in detail after release, but I am curious what you would expect in the V1 default asset library?  Bear in mind that asking for three genre's means you get a third less in each one, and your levels might have to be spread a little thin from time to time.

Saturday 22 June 2013

Saturday And The Art Arrives

Fly By

Just a quick blog to say plenty stuff done today but I'm keeping it hush hush for a few days, after which I promise a video.  To keep you guessing, here is a shot of a building that is now populating my prototype:


Plenty more where that came from, and with the shadows, cloud cover shading and all laid out in a small level, it really starts to create the impression of what could be, and if all goes well what will be.

Physics Like A Bullet

I also found time last night to open up the Bullet SDK and combine the terrain and kinetic player controller, and insert it into a hijacked ODE DLL. The three commands ODE START, ODE UPDATE and ODE END now contain Bullet code which compiles and runs in the background though presently does nothing. I figured a simple INTERSECT OBJECT command to find the current terrain height was sufficient for my Monday meeting demo. Technically all physics work is scheduled for July but I could not resist taking a peek to see if I could get my fancy new controller and dynamic terrain, and I can :)

Signing Off

I think I will make a nice cup of tea now and watch Monty Python's Holy Grail on NetFlix (a special treat to myself for not BSODDING another PC). My old new PC created further havok as well today which now refuses to boot after a de-fragmentation exercise. I then tried to run Windows repair only to find Windows refused to accept the disc as the build versions we out 0.00001 (thanks again MS) and to cap it all after I reformatted the drive to install a new Windows on there, the Windows installer refused to accept a partition existed.  You can't make this stuff up!! Next job here is to open up the case, disconnect all the drives except the MAIN drive, then try installing again. With a little luck the other drives will still be there, and more importantly all the data there in. Fortunately it's all backed up here and there, so this is simply a resurrection exercise only.  It looks like I made the right call switching to the old machine and carrying on with my little project.

Friday 21 June 2013

Friday Unchanged For Millions Of Years

A Good Day

Measured by my yardstick that decrees any day which does not contain a BSOD is now a good day.  Friday was a good day.  As some of you know, I am recovering from a system crash of mysterious origin and have updated my SATA controller driver in the hope this solves it. The stress test went well, as did my creation of a new SVN repository to hold the entire project off-site and on both machines.  For now disaster has been averted.

However, I am sure you want to learn about Reloaded development so here is the latest screen shot:


I am not tempting you with a video just yet, I am holding it back until I have more final artwork in the prototype. You will understand when you see it.  Don't worry about the insane specular coming off the sand, there will be much playing around with shaders all the way up to final beta.

So What Do You See

Well you probably recognize the wall and the monolith from the previous demo, but the sandy terrain is new and so is the sky.  Mark came through with some lovely undulating terrain, parallax textures and a matching sky box.  After much shifting of dev kit linkages, I managed to recompile my code and I do not seem to have lost any of my progress so the shadows cast just fine.

What you perhaps don't see is that the clouds in the sky are not part of the sky box but are in fact real dynamically generated clouds using our DarkCLOUDS module from Dark Basic Pro. Not only is this module capable of producing clouds, but has day/night cycles, sun simulation and a whole host of settings to affect how many clouds you have and what they do.  You can even accelerate time, or even reverse time from within the command set. Very powerful, very fast and just the job. I had to mask out the old sky box sky, and in doing so realized I need some really high resolution mountains for the final product but I think the effect is really cool. You get foreground terrain, then background mountains and finally a realistic sky.

I have been REALLY tempted to add extra stuff to the shader (bloom, depth of field, sun flare, etc) but I have a product to finish and the visuals are already looking top draw so I will leave those for tweaks and polish when we have more the product shape in place.

Under Control

The final part of my deliverable for the Monday meeting is the player controller system which will allow the user to wander around the level. The old one will be thrown out and my new one will take it's place. This one has been written quickly and to my exact requirements, and allows the player to walk and run, mouse look and jump will follow soon. The best feature of the new controller is that I've added smoothing maths so everything feels slick and professional. This is now in the demo and Saturday and Sunday will be spent tweaking the prototype and adding Mark Art as it comes in.

Signing Off

It's that time of year where a small office like mine starts to heat up really bad. The sunny days, coupled with having three PCs on, two large skylights and an overworked coder all contribute to the oven I now write from. Unfortunately, I have a product to finish so I must fight through it to the next battle in this war. I've downloaded the latest Bullet Physics SDK 2.8 and have run through a few of the demos. It seems the first two things I will need is the terrain demo and the kinetic character controller class. Whether it's an easy cut and paste job, or a nightmare on earth remains to be seen.  I've also been tinkering with the idea of writing a sand-storm particle effect for the demo as well, but perhaps this is one bridge too far for the weekend.

As a mini quiz, can anyone figure out the name of the game I was thinking about when I wrote the title?

Thursday 20 June 2013

Thursday Blue Crash No 2

Darn Hardware!

A successful day, and then disaster.  After executing my plan to copy in a working proto into the engine and then slowly migrate it over, the plan worked just fine and I was able to see shadows quite quickly in the main engine. It turns out that if you use PRINT or CENTRE TEXT commands, it royally messes up the pixel shader and who knows what else.

I managed to then move all the proto code out and have my regular Instance Stamp geometry textured and lit by the new shader, complete with shadows. The only remaining artefact is the first one/two cascades are using corrupt depth buffers for some reason, but there is a pattern there which will give me a clue as to who is putting it there and why.

Stop The Press

All this wonderful developer stuff ground to a halt when I experienced my second Blue Screen Of Despair, and after three attempts at a reboot I finally used Safe Mode to inspect my most key files, and of course, sods law struck again and had completely wiped out the main shadow mapping source file, not the header this time but the main file with all the clever stuff in.


To make matters worse, the PC had time to trigger the automated backup system which promptly copied the corrupt (empty) key file to the remote network backup, erasing my reserve copy.  My local copy was way out of date (four days old) and I was pretty much up the creek.

After over two decades of being paranoid about back-ups, I had a backup plan (pun intended) which was my GIT repository which I usually refresh each evening as a matter of course (and to get used to this popular rival to SVN). Lo and behold, a 2 day old copy of the missing file was intact and sitting there waiting to bring me happiness. Granted it had also managed to take the corrupt file as well somehow (maybe a panic sync after the horse had bolted), but both GIT and SVN have the rather cool feature of recording every version of the file you commit, so I was once again saved.

Emergency Protocol

What adventures we are having, and no mistake!  I instantly made the decision to move my entire Reloaded development files and work to my old machine which I strategically kept to one side and used daily for things like wiring blogs, answering emails and accessing older projects I did not want on the new 'super' machine.  It only has 400MB left on drive C so it looks like my evening will be spent freeing some drive space, moving over some files, ensuring everything compiles again and of course set up a triple redundant backup scheme for it all.

At the same time (though probably next week now) I need to investigate why my new monster machine is unstable.  The clues are that both BSOD crashes seemed to be memory related and on the first event it had to repair sectors of Drive C.  This narrows the possible culprits to a dodgy SSD, dodgy memory or an unstable over-clock.  As the machine was running fine for over 6 months, I am inclined to suspect the SSD as the villain of the piece. Unfortunately I am in a self imposed 'no more hardware' mode so a new SSD is out the question as they are a few hundred quid for a decent one.  I will probably steal a secondary SSD and reformat the whole machine (which means at least 1/2 days of reinstalling all the darn software too).  Before any of that, I am going to do all the software only stuff first like run a full virus scan, deep disc scan and any free stress tests I can find to put the PC through. The perfect solution is that I find the exact cause and swap out the dodgy part and carry on, though just like a stuttering car, once you are left stranded on the motorway you find it very hard to get the confidence back.

Back On Track From Friday

The priority must and is the Reloaded development schedule, so I will have finished my restoration tonight and be ready to continue Reloaded development in the afternoon.  These bumps in the road are inevitable when you are a developer, and it's about how you respond to them that separates the experienced guys who do it for a living, and the guys who kick and scream at the world for being so cruel.  The kicking and screaming is absolutely essential however, as it's the only way to learn the harsher lessons development throws at you.

Art-vine

I have heard a whisper that some artwork is being massaged into life this week to compliment our new parallax & shadow shader. I have a few objectives for a demo I am producing for a Monday meeting, so if you are short on time I highly recommend checking out the Blog on Monday/Tuesday for some visual candy.

Signing Off

I'm sure certain parties will conclude that because I built the PC in the first place, the BSOD and wasted half-day is entirely my fault. Sure enough, I make a point of buying dodgy hardware from time to time to keep me on my toes and provide exciting material for my blog ;)  In fact, I'm already planning my next massive system failure and mass deleting of a few weeks work!

Wednesday 19 June 2013

Wednesday Obstacle

Two Missions

I had several little tasks today, with the two largest being a major update of my Reloaded development schedule so I can plan and track the remaining part of this project, and to move the shadow goodness into the map editor so I can see the shadows in a live-editing setting.

The Schedule

Creation of the new spreadsheet went fine, and I was able to plot out in detail all remaining tasks required including Terrain, AI, Physics, Encryption, Module work and testing.  I have planned for October as per the original deadline, but with the 'extreme research direction' we've been going that deliverable will be a stretch. I am still aiming for it to put myself under some healthy pressure but I am not too concerned if I step over the line a little. As long as I can put some great game creation software in your hands by Christmas I will be happy.

The Map Editor Shadows

Now this one was (and still is) a pig. I moved over the few bits of code which would have allowed shadows to spring to life, placed the light somewhere suitable and presto, nothing. No shadows, nada. Okay, I missed a value somewhere, back I go. The best part of five hours later and I am not only still finding the solution, but the underlying problem is the maddest one I've seen in over ten years of DirectX coding.


The spheres you can see are placed there for my debugging efforts, the problem is the orange rectangle in the top left which seems to only want to render the first 256x256 units of the object universe. Not only that, it wants to render it upside down, untransformed and use incomplete UV data.  That red to yellow stripe is actually a pixel shader plotting the UV coordinates as the RG of an RGB colour.  Clearly, it has an issue interpolating the R from zero to one and just sticks at ONE. Grr.  The bottom orange square was an early image were/where I managed to get 'something' from the spheres I was rendering, see the few pixels in the top left corner.

My best guess right now is that unlike my prototype, my main engine has a much more complicated shader hierarchy and somewhere along the way the engine is messing up the shader I am using (identical shader works fine in the prototype). It seems to take in the raw model coordinates fine, but extra things like constants and secondary vertex data seem to be missing or partially incomplete.  I've checked everything four times and the code looks just as it should. Furthermore, I am running out of clever ideas to trick this shader back to life.

Lee's Last Stand

I have one more trick up my sleeve which is to transplant the entire prototype inside the engine code (as is) to get the exact object set-up working and casting shadows. Once I have shadows, I slowly remove the prototype objects and introduce the engine objects until something breaks. When it breaks, whatever I transposed last is the culprit.  No doubt this issue will take me into the early hours but it will be worth it to see shadows being cast in real-time as you add and remove segments, rendering using a gloriously cool parallax depth mapper.

Signing Off

I've been waiting for a few deliveries today, and one of them was the game S.T.A.L.K.E.R which was one of the classic examples of excellent deferred rendering in it's day.  I have decided not to implement deferred rendering until I get back from my holiday, mainly because I want to continue working on the shader, the shadow system and the new artwork coming from Mark to create a nifty demo to leave you all drooling over.  Deferred rendering is simply another way of solving the lighting question in a game engine. If I can get the current forward renderer producing fast shadows for outdoor and indoor use, and get to keep my parallax mapper for universe rendering, we may not need to defer at all. Performance will be the key decider here, so let's carry on with the eye candy and see were/where we are when we decide to stop adding.  I was almost tempted to add depth of field and bloom to the prototype, then stopped myself in the nick of time. Plenty time for final polish once the guts of the rendering engine are running fast!

Tuesday 18 June 2013

Tuesday Tweaks

More Shadow Stuffs

After I posted my blog, I carried on cleaning up code and improving the shadow rendering for another 14 hour plus marathon, and although there are inevitable tweaks still to come I am happy with the final result.

We now have 'Percentage Closer Filtering (3x3)' for a nice soft shadow and cascade blending to remove the banding artifacts as the shadows steps into the lower quality levels in the distance (kind of like LOD for shadows).

Here is a video (minus the new art) as I was very excited to show you the latest version (and also to show off the technique below):


Notice how the shadow goes into the brickwork? By using the depth map information from the Parallax Mapping fragment shader, I was able to hijack these most previous variables and use them for my shadows.

Shadows in the Depth Map

One of the many articles I read on this subject had the idea of using the normal map (depth map) as part of the shadowing operation, so that the shadows could fall into the grooves of the normally flat surface. Although I did not read up on the correct math, my own idea seemed sensible enough, which was to shift the world position of the texel into the surface of the polygon using the tangent aligned normal. The shadow mapper would then assume that pixel was further away (deeper) and thus shade it as such, and it did. Imagine my delight when the shadow now contoured around the details in the texture depth map as well as the large blocks.

The only noticeable issues I have now is that at extreme angles the shadow calculation stops short of the width of the cascade but it's rare to see this artifact. A more serious one though is the edge bleed you can see at the base of the objects where they meet the floor. As I am using Clockwise Culling to defeat self shadowing on the side facing the light source, I only get a polygon thick wall to cast the actual shadow which bleeds non-shadow values thanks to the new PCF effect. There are numerous solutions but each one carries a side effect that is worse than the fix, so I am going to let that one brew a while. The current fix is to shrink the geometry slightly when I render the depth views, which helps a little but it's now what I am after. I am sure the solution will present itself before too long.

Signing Off

My mission now is to clean the code up some more (as there is a lot of commented out experimental code that never worked), and then see if I can apply this process to the main engine (just to see what happens). My new implementation does almost all of the shadowing in the C++ DLL side or the shader, so it should be as simple as placing a light and calling a few commands in the main engine to get my shadows in there.


Monday 17 June 2013

Monday Shadow Success

Weekend Power

It took a long 8 hour stint on Saturday and most of Monday but I finally managed to crack the cascade shadow mapping trick. Phew!  It has been a while since I coded some deep 3D or shaders, and last week it was very much a case of cut and pasting my way to victory.

Alas when things start to go wrong, or shadows render in strange and unpredictable ways, you have nowhere to turn when you base your techniques on someone else's idea.  I resolved to sit down and learn every single field, call, transform and math step and really understand what shadow mapping. This approach worked, and mid-day Saturday I began to see how the shadow coordinates where calculated to produce the desired results.

Blue Screen Of Despair

It did not help the situation when close to the end of Saturday, my new machine did a massive BSOD and completely erased one of the header files I was working on. All gone. No back-ups that I could find and despair for me. Rewriting it would have taken another half a day, and then in my darkest hour I remembered that I had set-up a network backup system to grab the contents of the drive and copy it each night at 7PM. Amazingly I had a complete version from Friday night and was able to put it back in and continue working.

Monday 6PM

As of now, what I have you can see below. The clever stuff is invisible of course, but the shadow you see is actually generated from four depth render targets and seamlessly stitched together using orthographic projections and some clever 3D math.


In order to be able to read the depth buffer for this technique, I discovered a cool hack you can make on DX9 hardware to transfer the depth buffer contents over to a regular texture that can be passed to a shader. Using the same trick, it might be possible to do some really nice effects such as depth-sensitive smoke and particle effects to avoid those sharp polygons cutting into the scene. I will leave those for another day, but now I have access to the depth buffer from a single pass with no loss of performance, it's another tool in my Reloaded toolbox for the future.

The battle is far from over as the current prototype does not use PCF (percentage closer filtering), cascade blending (to make the cascade transitions invisible) and I still need to check out the depth ranges I am using to ensure I get the highest quality depth information from each cascade.

Signing Off

Sorry if you where expecting a video, but I am going to wait for some new artwork from Mark before putting something on YouTube as I think it will show off the shadow technique better with brighter more interesting objects.  I have just made a massive backup of my files so far (twice) in a bout of healthy paranoia and I am going to switch to a few emails that need my attention. I was hoping to get the shadow stuff cracked today, and my mission was a success. The lesson learned over the last few days is that as good as cut and paste is, there is no substitute for a piece of paper, a pen and a brain that wants to learn how to do the 'working out'.  Well done brain!

Friday 14 June 2013

Friday Shader Frenzy

My Brain Has Melted

Now that was a long day! There should be a law to working so many hours, especially when there is not a pot of lovely gold at the end of the day. Alas after many burned candles, I cannot show you a glorious cascade shaded rendered scene.  This is what I can show you:


Impressive huh?!  Believe it or not I have made amazing progress, it's just the last two bits that let me show you the shadows are the two bits I need to research more, namely attaching a depth surface to a shader so I can read it in DirectX 9 and using a "sample comparison state" similar to the DX11 method used in the example I have been mining.

To explain the above scene, the bottom part that shows red, green, yellow and blue highlights are some quick cascade divisions (hacked in), but the real divisions are in the shader ready to be used and modified from the app. The upper rectangles are four cascade shadow renders at different zoomed in distances to the area to be shadowed. As I am not rendering the depth as a color in the render target you cannot see anything but a solid color, but hopefully the 'depth' data has been preserved and that is the data I want (need) to use for the main render scene. I use them right now for debugging purposes.

If anyone knows a good tool that allows me to view a DirectX 9 depth/stencil surface in real-time, please do send the link :)

Code Attack

I usually don't assault you with code, but today I wanted to give you a glimpse of the shader Code that I am working on.  It's a Mark Blosser shader with my shadow mapping code added here and there, and some hacks so I could show you that nice screen shot.

SHADER CODE (in progress):

/********************************************************************************************
Shader Model 3 Rendering System for FPS Creator
by Mark Blosser  
email: mjblosser@gmail.com
website: www.mjblosser.com
-------------------------------------------------------------------
Description: 
Shader for segments. Uses lightmapping and normal/spec.
Performs parallax occlusion mapping using the cone-step method.
INVERTED heightmap stored in normal map alpha channel.
Conemap stored in normal map blue channel.
Requires D2/I/N textures.  All textures must be in texturebank folder.

*********************************************************************************************/

/**************MATRICES & UNTWEAKABLES *****************************************************/

// shadow mapping
matrix m_mShadow;
float4 m_vCascadeOffset[8];
float4 m_vCascadeScale[8];
int m_nCascadeLevels;
int             m_iPCFBlurForLoopStart; // For loop begin value. For a 5x5 Kernal this would be -2.
int             m_iPCFBlurForLoopEnd; // For loop end value. For a 5x5 kernel this would be 3.
float           m_fMinBorderPadding;     
float           m_fMaxBorderPadding;
float           m_fShadowBiasFromGUI;  // A shadow map offset to deal with self shadow artifacts.  
float           m_fShadowPartitionSize; 
float           m_fCascadeBlendArea; // Amount to overlap when blending between cascades.
float           m_fTexelSize; 
float           m_fNativeTexelSizeInX;
//float4          m_fCascadeFrustumsEyeSpaceDepthsFloat[2];  // The values along Z that seperate the cascades.
//float4          m_fCascadeFrustumsEyeSpaceDepthsFloat4[8];  // the values along Z that separte the cascades.  
float           m_fCascadeFrustumsEyeSpaceDepths[8];
float3          m_vLightDir;


float4x4 World : World;
float4x4 WorldInverse : WorldInverse;
float4x4 WorldIT : WorldInverseTranspose;
float4x4 WorldView : WorldView;
float4x4 WorldViewProjection : WorldViewProjection;
float4x4 View : View;
float4x4 ViewInverse : ViewInverse;
float4x4 ViewIT : ViewInverseTranspose;
float4x4 ViewProjection : ViewProjection;
float4x4 Projection : Projection;

float4x4 boneMatrix[60] : BoneMatrixPalette;
float4 eyePos : CameraPosition;
float time : Time;
float sintime : SinTime;

/**************VALUES PROVIDED FROM FPSC - NON TWEAKABLE**************************************/

float4 clipPlane : ClipPlane;  //cliplane for water plane

float4 LightSource // SUN LOCATION 
<   string UIType = "Fixed Light Source";
> = {2500.0f,2500.0f,50000.0f, 1.0f};

float4 AmbiColor : Ambient
<    string UIName =  "AmbiColor";    
> = {0.2f, 0.2f, 0.3f, 0.0f};

float4 AmbiColorOverride  //need to divide by 255 in shader
<    string UIName =  "AmbiColorOverride";    
> = {255.0f, 255.0f, 255.0f, 255.0f};

float4 SurfColor : Diffuse
<    string UIName =  "SurfColor";    
> = {1.0f, 1.0f, 1.0f, 1.0f};

//Supports dynamic lights (using CalcLighting function)
float4 g_lights_pos0;
float4 g_lights_pos1;
float4 g_lights_pos2;
float4 g_lights_pos3;
float4 g_lights_atten0;
float4 g_lights_atten1;
float4 g_lights_atten2;
float4 g_lights_atten3;
float4 g_lights_diffuse0;
float4 g_lights_diffuse1;
float4 g_lights_diffuse2;
float4 g_lights_diffuse3;

//SpotFlash Values from FPSC
float4 SpotFlashPos;  //SpotFlashPos.w is carrying the spotflash fadeout value
float4 SpotFlashColor; //remember this has not been divided by 255 before being sent from FPSC (I should fix this)
float SpotFlashRange   //fixed value that FPSC uses
<   string UIName =  "SpotFlash Range";    
> = {600.00};

//WATER Fog Color
float4 FogColor : Diffuse
<   string UIName =  "Fog Color";    
> = {0.0f, 0.0f, 0.0f, 0.0000001f};

//HUD Fog Color
float4 HudFogColor : Diffuse
<   string UIName =  "Hud Fog Color";    
> = {0.0f, 0.0f, 0.0f, 0.0000001f};

//HUD Fog Distances (near,far,0,0)
float4 HudFogDist : Diffuse
<   string UIName =  "Hud Fog Dist";    
> = {1.0f, 0.0f, 0.0f, 0.0000001f};


//Shader Variables pulled from FPI scripting 
float4 ShaderVariables : ShaderVariables
<    string UIName =  "Shader Variables";    
> = {1.0f, 1.0f, 1.0f, 1.0f};


/***************TEXTURES AND SAMPLERS***************************************************/
//For lightmapped ojbects (Lightmap, D, I, N)

texture LightMap : DiffuseMap
<
    string Name = "LM.tga";
    string type = "2D";
>;

texture DiffuseMap : DiffuseMap
<
    string Name = "D.tga";
    string type = "2D";
>;

texture NormalMap : DiffuseMap
<
    string Name = "N.tga";
    string type = "2D";
>;

texture DepthMapTX1 : DiffuseMap
<
    string Name = "DEPTH1.tga";
    string type = "2D";
>;
texture DepthMapTX2 : DiffuseMap
<
    string Name = "DEPTH1.tga";
    string type = "2D";
>;
texture DepthMapTX3 : DiffuseMap
<
    string Name = "DEPTH1.tga";
    string type = "2D";
>;
texture DepthMapTX4 : DiffuseMap
<
    string Name = "DEPTH1.tga";
    string type = "2D";
>;

//Lightmap texture
sampler2D LightmapSampler = sampler_state
{
    Texture   = <LightMap>;
    MipFilter = LINEAR;
    MinFilter = ANISOTROPIC;
    MagFilter = LINEAR;
};

//Diffuse Texture _D
sampler2D DiffuseSampler = sampler_state
{
    Texture   = <DiffuseMap>;
    MipFilter = LINEAR;
    MinFilter = ANISOTROPIC;
    MagFilter = LINEAR;
};

//Effect Texture _N (could be anything here - ill,spec,normal)
sampler2D NormalSampler = sampler_state
{
    Texture   = <NormalMap>;
    MipFilter = LINEAR;
    MinFilter = ANISOTROPIC;
    MagFilter = LINEAR;
};

//DepthMapForShadow (generated in app)
sampler2D DepthMap1 = sampler_state
{
Texture = <DepthMapTX1>;
};
sampler2D DepthMap2 = sampler_state
{
Texture = <DepthMapTX2>;
};
sampler2D DepthMap3 = sampler_state
{
Texture = <DepthMapTX3>;
};
sampler2D DepthMap4 = sampler_state
{
Texture = <DepthMapTX4>;
};

/************* DATA STRUCTS **************/

// structures for shadow map depth pass
struct IN_Depth
{
float4 Pos:POSITION; 
};
struct OUT_Depth
{
float4 OPos:POSITION; 
};

// structures for final render
struct appdata
 {
    float4 Position : POSITION;
    float2 UV0 : TEXCOORD0;
    float2 UV1 : TEXCOORD1;
    float4 Normal : NORMAL;    
};

/*data passed to pixel shader*/
struct vertexOutput
{
    float4 Position     : POSITION;
    float2 TexCoord     : TEXCOORD0;
    float2 TexCoordLM   : TEXCOORD1;
    float3 LightVec    : TEXCOORD2;
    float3 WorldNormal : TEXCOORD3;
    float4 WPos         : TEXCOORD4;
    float  WaterFog     : TEXCOORD5;  
    float  clip         : TEXCOORD6;
    float4 vTexShadow : TEXCOORD7;
    float4 vInterpPos   : TEXCOORD8; 
    float  vDepth       : TEXCOORD9;
};

/****** helper functions for shadow mapping*****/

void ComputeCoordinatesTransform( in int iCascadeIndex,
                                      in float4 InterpolatedPosition ,
                                      in out float4 vShadowTexCoord , 
                                      in out float4 vShadowTexCoordViewSpace ) 
{
    // Now that we know the correct map, we can transform the world space position of the current fragment                
    vShadowTexCoord = vShadowTexCoordViewSpace * m_vCascadeScale[iCascadeIndex];
    vShadowTexCoord += m_vCascadeOffset[iCascadeIndex];
    vShadowTexCoord.x *= m_fShadowPartitionSize;  // precomputed (float)iCascadeIndex / (float)CASCADE_CNT
    vShadowTexCoord.x += (m_fShadowPartitionSize * (float)iCascadeIndex ); 

void CalculateRightAndUpTexelDepthDeltas ( in float3 vShadowTexDDX,
                                           in float3 vShadowTexDDY,
                                           out float fUpTextDepthWeight,
                                           out float fRightTextDepthWeight )
{
// This function calculates the screen space depth for shadow space texels    
    // We use the derivatives in X and Y to create a transformation matrix.  Because these derivives give us the 
    // transformation from screen space to shadow space, we need the inverse matrix to take us from shadow space 
    // to screen space.  This new matrix will allow us to map shadow map texels to screen space.  This will allow 
    // us to find the screen space depth of a corresponding depth pixel.
    // This is not a perfect solution as it assumes the underlying geometry of the scene is a plane.  A more 
    // accureate way of finding the actual depth would be to do a deferred rendering approach and actually 
    // sample the depth (ca-ching dudes!)
    // Using an offset, or using variance shadow maps is a better approach to reducing these artifacts in most cases.
    
    float2x2 matScreentoShadow = float2x2( vShadowTexDDX.xy, vShadowTexDDY.xy );
    float fDeterminant = determinant ( matScreentoShadow );
    
    float fInvDeterminant = 1.0f / fDeterminant;
    
    float2x2 matShadowToScreen = float2x2 (
        matScreentoShadow._22 * fInvDeterminant, matScreentoShadow._12 * -fInvDeterminant, 
        matScreentoShadow._21 * -fInvDeterminant, matScreentoShadow._11 * fInvDeterminant );

    float2 vRightShadowTexelLocation = float2( m_fTexelSize, 0.0f );
    float2 vUpShadowTexelLocation = float2( 0.0f, m_fTexelSize );  
    
    // Transform the right pixel by the shadow space to screen space matrix.
    float2 vRightTexelDepthRatio = mul( vRightShadowTexelLocation,  matShadowToScreen );
    float2 vUpTexelDepthRatio = mul( vUpShadowTexelLocation,  matShadowToScreen );

    // We can now caculate how much depth changes when you move up or right in the shadow map.
    // We use the ratio of change in x and y times the dervivite in X and Y of the screen space 
    // depth to calculate this change.
    fUpTextDepthWeight = 
        vUpTexelDepthRatio.x * vShadowTexDDX.z 
        + vUpTexelDepthRatio.y * vShadowTexDDY.z;
    fRightTextDepthWeight = 
        vRightTexelDepthRatio.x * vShadowTexDDX.z 
        + vRightTexelDepthRatio.y * vShadowTexDDY.z;  
}

void CalculatePCFPercentLit ( in float4 vShadowTexCoord, 
                              in float fRightTexelDepthDelta, 
                              in float fUpTexelDepthDelta, 
                              in float fBlurRowSize,
                              out float fPercentLit ) 
{
// Use PCF to sample the depth map and return a percent lit value.
    fPercentLit = 0.0f;
/*
    // This loop could be unrolled, and texture immediate offsets could be used if the kernel size were fixed.
    // This would be performance improvment.
    //for( int x = m_iPCFBlurForLoopStart; x < m_iPCFBlurForLoopEnd; ++x ) 
    {
        //for( int y = m_iPCFBlurForLoopStart; y < m_iPCFBlurForLoopEnd; ++y ) 
        {
            float depthcompare = vShadowTexCoord.z;
            // A very simple solution to the depth bias problems of PCF is to use an offset.
            // Unfortunately, too much offset can lead to Peter-panning (shadows near the base of object disappear )
            // Too little offset can lead to shadow acne ( objects that should not be in shadow are partially self shadowed ).
            depthcompare -= m_fShadowBiasFromGUI;
            // Compare the transformed pixel depth to the depth read from the map.
            //fPercentLit += g_txShadow.SampleCmpLevelZero( g_samShadow, 
            //    float2( 
            //        vShadowTexCoord.x + ( ( (float) x ) * m_fNativeTexelSizeInX ) , 
            //        vShadowTexCoord.y + ( ( (float) y ) * m_fTexelSize ) 
            //        ), 
            //    depthcompare );
// LEE, Research sampler comparison coommand for DX9 : SampleCmpLevelZero
            //fPercentLit += g_txShadow.SampleCmpLevelZero( g_samShadow, 
            //    float2( vShadowTexCoord.x, vShadowTexCoord.y ), 
            //    depthcompare );
// 
//fPercentLit += tex2D(DepthMap1,float2(vShadowTexCoord.x, vShadowTexCoord.y));// < Depth ? 0.5f:0.0f);
        //}
    //}
    //fPercentLit /= (float)fBlurRowSize;
*/
}

void CalculateBlendAmountForInterval ( in int iCurrentCascadeIndex, 
                                       in out float fPixelDepth, 
                                       in out float fCurrentPixelsBlendBandLocation,
                                       out float fBlendBetweenCascadesAmount ) 
{
// Calculate amount to blend between two cascades and the band where blending will occure.
    // We need to calculate the band of the current shadow map where it will fade into the next cascade.
    // We can then early out of the expensive PCF for loop. 
    float fBlendInterval = m_fCascadeFrustumsEyeSpaceDepths[ iCurrentCascadeIndex  ];
    //if( iNextCascadeIndex > 1 ) 
    int fBlendIntervalbelowIndex = min(0, iCurrentCascadeIndex-1);
    fPixelDepth -= m_fCascadeFrustumsEyeSpaceDepths[ fBlendIntervalbelowIndex ];
    fBlendInterval -= m_fCascadeFrustumsEyeSpaceDepths[ fBlendIntervalbelowIndex ];
    
    // The current pixel's blend band location will be used to determine when we need to blend and by how much.
    fCurrentPixelsBlendBandLocation = fPixelDepth / fBlendInterval;
    fCurrentPixelsBlendBandLocation = 1.0f - fCurrentPixelsBlendBandLocation;
    // The fBlendBetweenCascadesAmount is our location in the blend band.
    fBlendBetweenCascadesAmount = fCurrentPixelsBlendBandLocation / m_fCascadeBlendArea;
}



/*******Vertex Shader***************************/

OUT_Depth VS_Depth(IN_Depth IN)
{
   OUT_Depth OUT;
   OUT.OPos = mul(IN.Pos,WorldViewProjection); 
   return OUT;
}

vertexOutput mainVS(appdata IN)   
{
vertexOutput OUT;
    
    //float4 tempPos = float4(IN.Position, 1);
    float4 worldSpacePos = mul(IN.Position, World);
    OUT.WPos =   worldSpacePos;  
    
    OUT.WorldNormal = normalize(mul(IN.Normal, WorldIT).xyz);
    
    float3 eyeoffset = float3 (10,100,0);
    //OUT.LightVec = normalize (eyePos+eyeoffset - worldSpacePos );
    OUT.LightVec = normalize (LightSource - worldSpacePos );
   
    OUT.Position = mul(IN.Position, WorldViewProjection);
    OUT.TexCoord  = IN.UV0 ; 
    OUT.TexCoordLM  = IN.UV1; 
  
    // calculate Water FOG colour
    float4 cameraPos = mul( worldSpacePos, View );
    float fogstrength = cameraPos.z * FogColor.w;
    OUT.WaterFog = min(fogstrength,1.0);
   
    // all shaders should send the clip value to the pixel shader (for refr/refl)                                                                     
    OUT.clip = dot(worldSpacePos, clipPlane);                                                                      
    
    OUT.vTexShadow = float4(0,0,0,0);
    OUT.vInterpPos = float4(0,0,0,0);
    OUT.vDepth = 0.0f;
    // SHADOW MAPPING - transform the shadow texture coordinates for all the cascades.
    OUT.vInterpPos = IN.Position;   
    OUT.vDepth = mul( IN.Position, WorldView ).z; 
    OUT.vTexShadow = mul( IN.Position, m_mShadow );
    return OUT;
}

/****************Framgent Shader*****************/

float4 PS_Depth(OUT_Depth IN) : COLOR
{
return float4(1,1,0,1);//(IN.Depth/LightRange)+0.01f;
}

float4 CalcSpotFlash( float3 worldNormal, float3 worldPos )
{
    float4 output = (float4)0.0;
    float3 toLight = (SpotFlashPos.xyz - worldPos.xyz);
    float3 lightDir = normalize( toLight );
    float lightDist = length( toLight );
    
    float MinFalloff = 100;  //falloff start distance
    float LinearFalloff = 1;
    float ExpFalloff = .005;  // 1/200
    SpotFlashPos.w = clamp(0,1,SpotFlashPos.w -.2);
    
    //classic attenuation - but never actually reaches zero
    //float fAtten = 1.0/(MinFalloff + (LinearFalloff*lightDist)); //bit faster linear atten only
    float fAtten = 1.0/(MinFalloff + (LinearFalloff*lightDist)+(ExpFalloff*lightDist*lightDist));
    //output += max(0,dot( lightDir, worldNormal ) * (SpotFlashColor) *fAtten * (SpotFlashPos.w) );
    output += (SpotFlashColor) *fAtten * (SpotFlashPos.w); //don't use normal, faster
    
    //faster attenuation function based on min and max falloff distances
    //float fAtten = saturate((500-lightDist)/(500-50));  //50 is minimum distance, 500 is end distance
    //output=fAtten*SpotFlashPos.w*SpotFlashColor*.01;  //fastest technique - doesn't use Normal
    //output = max(0,dot( lightDir, worldNormal )) * fAtten*SpotFlashPos.w*(SpotFlashColor/100);
        
    return output;
}

float4 CalcLighting(float3 Nb, float3 worldPos, float3 Vn, float4 diffusemap,float4 specmap)
{
    float4 output = (float4)0.0;
    
    // light 0
    float3 toLight = g_lights_pos0.xyz - worldPos;
    float lightDist = length( toLight );
    //float fAtten = saturate((200-lightDist)/(200-0)); //faster distance-based atten but needs lightrange from FPSC
    float fAtten = 1.0/dot( g_lights_atten0, float4(1,lightDist,lightDist*lightDist,0) );
    float3 lightDir = normalize( toLight );
    //output += max(0,dot( lightDir, Nb ) * g_lights_diffuse0 * fAtten * 1.7); //cheaper-no spec
    float3 halfvec = normalize(Vn + lightDir);
    float4 lit0 = lit(dot(lightDir,Nb),dot(halfvec,Nb),24); 
    output+= (lit0.y *g_lights_diffuse0 * fAtten * 1.7*diffusemap) + (lit0.z * g_lights_diffuse0 * fAtten *specmap);   
    // light 1
    toLight = g_lights_pos1.xyz - worldPos;
    lightDist = length( toLight );
    fAtten = 1.0/dot( g_lights_atten1, float4(1,lightDist,lightDist*lightDist,0) );
    lightDir = normalize( toLight );
    //output += max(0,dot( lightDir, Nb ) * g_lights_diffuse1 * fAtten * 1.7); //cheaper-no spec
    halfvec = normalize(Vn + lightDir);
    float4 lit1 = lit(dot(lightDir,Nb),dot(halfvec,Nb),24); 
    output+= (lit1.y *g_lights_diffuse1 * fAtten * 1.7*diffusemap) + (lit1.z * g_lights_diffuse1 * fAtten *specmap);
    
    // light 2 -maybe optimize by not calculating spec?
    toLight = g_lights_pos2.xyz - worldPos;
    lightDist = length( toLight );
    fAtten = 1.0/dot( g_lights_atten2, float4(1,lightDist,lightDist*lightDist,0) );
    lightDir = normalize( toLight );
    //output += max(0,dot( lightDir, Nb ) * g_lights_diffuse1 * fAtten * 1.7); //cheaper-no spec
    halfvec = normalize(Vn + lightDir);
    float4 lit2 = lit(dot(lightDir,Nb),dot(halfvec,Nb),24); 
    output+= (lit2.y *g_lights_diffuse2 * fAtten * 1.7*diffusemap) + (lit2.z * g_lights_diffuse2 * fAtten *specmap);    

// (pixel shader 2.0 runs out of space here)
//    // light 2 
//    toLight = g_lights_pos2.xyz - worldPos;
//    lightDist = length( toLight );
//    fAtten = 1.0/dot( g_lights_atten2, float4(1,lightDist,lightDist*lightDist,0) );
//    lightDir = normalize( toLight );
//    output += max(0,dot( lightDir, worldNormal ) * g_lights_diffuse2 * fAtten * 1.7);
//    // light 3
//    toLight = g_lights_pos3.xyz - worldPos;
//    lightDist = length( toLight );
//    fAtten = 1.0/dot( g_lights_atten3, float4(1,lightDist,lightDist*lightDist,0) );
//    lightDir = normalize( toLight );
//    output += max(0,dot( lightDir, worldNormal ) * g_lights_diffuse3 * fAtten * 1.7);

    return output;
}


float4 mainPS(vertexOutput IN) : COLOR
{
    float4 finalcolor;
    
    clip(IN.clip); // all shaders should receive the clip value
    
    float3 V  = (eyePos - IN.WPos);  
    float3 Vn  = normalize(V); 
    
    ////////TANGENT BASIS CONSTRUCTION FOR CONEMAPPING
    // Optimisation 3:
// assume M is orthogonal
float3 p = (IN.WPos);

    // get edge vectors of the pixel triangle
    float3 dp1 = ddx(p );
    float3 dp2 = ddy(p );
    float2 duv1 = ddx( IN.TexCoord.xy );
    float2 duv2 = ddy( IN.TexCoord.xy );

    // solve the linear system
    // (not much solving is left going here)
    float3x3 M = float3x3( dp1, dp2, cross(dp1,dp2) );
    float2x3 invTM = float2x3(cross(M[1],M[2]),cross(M[2],M[0]));
    float3 T = mul( float2( duv1.x, duv2.x ), invTM );
    float3 B = mul( float2( duv1.y, duv2.y ), invTM );
    float3 Nn = normalize(IN.WorldNormal);
    // construct tangent frame basis matrix 
    float3x3 tangentbasis = float3x3( 1*normalize(T), 1*normalize(B),Nn );
    //////////////////////////////////////////////////////////////////////////////// 
    
    ///////////////CREATE NEW UV COORDS///////////////////////////////////////
    float3 newUVs = float3(IN.TexCoord, 0.0);
    
    float3 rayVec = (IN.WPos- ViewInverse[3]);
    rayVec = mul(tangentbasis,rayVec);
    rayVec = normalize(rayVec);
rayVec.z = abs(rayVec.z);
float depth = 0.03;
rayVec.xy *= depth;
float dist = length(rayVec.xy);
    
    for( int i=0;i<16; i++ )
{
float4 tex = tex2D(NormalSampler, newUVs.xy);
float height = saturate(tex.w - newUVs.z);

float cone_ratio = tex.z*tex.z;
float stepDist = height * cone_ratio / (cone_ratio + dist);
newUVs += rayVec * stepDist;
}
    
    float4 lightmap = tex2D(LightmapSampler,IN.TexCoordLM); //sample lightmap texture    
    float4 diffusemap = tex2D(DiffuseSampler,newUVs.xy);  //sample D2 texture
    float4 specmap =diffusemap.w*2;
    
    float3 Nb = tex2D(NormalSampler,newUVs.xy);
Nb.xy = Nb.xy * 2.0 - 1.0;
//Nb.y = -Nb.y;
Nb.z = sqrt(1.0 - dot(Nb.xy, Nb.xy));
Nb = mul(Nb,tangentbasis);
Nb = normalize(Nb);
    
    float3 Ln = normalize(IN.LightVec);   //light vector
    float3 Hn = normalize(Vn+Ln);         //half-vector
    
    //main lighting equation
    float4 lighting = lit((dot(Ln,Nb)),dot(Hn,Nb),24); 

    float4 dynamiclighting = CalcLighting (Nb, IN.WPos.xyz, Vn, diffusemap,specmap);
    float4 spotflashlighting = CalcSpotFlash (IN.WorldNormal, IN.WPos.xyz);
    
    AmbiColor*= (AmbiColorOverride/255); //remember to account for ambient COLOR script changes

    //diffuse and ambient contribution
    float4 diffuseContrib = diffusemap *lighting.y *lightmap;
    
    //specular light contribution
    float4 specContrib = specmap *lighting.z *lightmap  ;
    
    //Ambient, Dynamic, and Spotflash contributions + half-strength lightmap    
    //float4 ambContrib = ((0.5*lightmap) + AmbiColor + dynamiclighting + spotflashlighting) * diffusemap;
    float4 ambContrib = (((0.5*lightmap) + AmbiColor + spotflashlighting) * diffusemap) + dynamiclighting;
    //float4 ambContrib = (lightmap+AmbiColor+dynamiclighting+spotflashlighting) *diffusemap; //full lightmap overly bright
    //float4 ambContrib = (AmbiColor + dynamiclighting + spotflashlighting) * diffusemap; //no extra lightmap (too dark)
    float4 result =  ambContrib + diffuseContrib;
/*
// per pixel shadow projection (for correct split)
float4 Proj1 = mul(ViewMat,mul(IN.WPos,LightProjMatrix1));
float4 Proj2 = mul(ViewMat,mul(IN.WPos,LightProjMatrix2));
float4 Proj3 = mul(ViewMat,mul(IN.WPos,LightProjMatrix3));
float4 Proj4 = mul(ViewMat,mul(IN.WPos,LightProjMatrix4));
//float Depth=(IN.Proj.z/LightRange);
//float shadowmap=1-(tex2Dproj(DepthMap,IN.Proj+float4(-0.8,-0.8,0,0)) < Depth ? 0.05111f:0.0f); // quick PCF softening
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(0,-0.8,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(0.8,-0.8,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(-0.8,0,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(0,0,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(0.8,0,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(-0.8,0.8,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(0,0.8,0,0)) < Depth ? 0.05111f:0.0f);
//shadowmap=shadowmap-(tex2Dproj(DepthMap,IN.Proj+float4(0.8,0.8,0,0)) < Depth ? 0.05111f:0.0f);
//affect result by any shadows cast on this pixel
//float3 shadowProj1 = Proj1.xyz / Proj1.w;
//float3 shadowProj2 = Proj2.xyz / Proj2.w;
//float3 shadowProj3 = Proj3.xyz / Proj3.w;
//float3 shadowProj4 = Proj4.xyz / Proj4.w;
//bool s1 = all(abs(shadowProj1 - 0.5f) < 0.5f);
//bool s2 = all(abs(shadowProj2 - 0.5f) < 0.5f);
//bool s3 = all(abs(shadowProj3 - 0.5f) < 0.5f);
//bool s4 = all(abs(shadowProj4 - 0.5f) < 0.5f);
result = float4(0,0,0,1);
float Depth;
float shadowmap;
float fPixelDepth = IN.vDepth / 250.0f;
if ( fPixelDepth>0.75f )
{
Depth=(Proj4.z/LightRange);
shadowmap=1-(tex2Dproj(DepthMap4,Proj4+float4(0.0,0.0,0,0)) < Depth ? 0.5f:0.0f);
result = float4(0,1,0,1) * shadowmap;
}
if ( fPixelDepth>=0.5f && fPixelDepth<0.75f )
{
Depth=(Proj3.z/LightRange);
shadowmap=1-(tex2Dproj(DepthMap3,Proj3+float4(0.0,0.0,0,0)) < Depth ? 0.5f:0.0f);
result = float4(1,0,0,1) * shadowmap;
}
if ( fPixelDepth>=0.25f && fPixelDepth<0.5f )
{
Depth=(Proj2.z/LightRange);
shadowmap=1-(tex2Dproj(DepthMap2,Proj2+float4(0.0,0.0,0,0)) < Depth ? 0.5f:0.0f);
result = float4(1,1,0,1) * shadowmap;
}
if ( fPixelDepth>=0.0f && fPixelDepth<0.25f )
{
Depth=(Proj1.z/LightRange);
shadowmap=1-(tex2Dproj(DepthMap1,Proj1+float4(0.0,0.0,0,0)) < Depth ? 0.5f:0.0f);
result = float4(1,1,1,1) * shadowmap;
}
//result = float4(1,1,1,1) * shadowmap;
//result = result * shadowmap;
    */
    float4 vShadowMapTextureCoord = 0.0f;
    float4 vShadowMapTextureCoord_blend = 0.0f;
    float fPercentLit = 0.0f;
    float fPercentLit_blend = 0.0f;
    float fUpTextDepthWeight=0;
    float fRightTextDepthWeight=0;
    float fUpTextDepthWeight_blend=0;
    float fRightTextDepthWeight_blend=0;
    int iBlurRowSize = m_iPCFBlurForLoopEnd - m_iPCFBlurForLoopStart;
    iBlurRowSize *= iBlurRowSize;
    float fBlurRowSize = (float)iBlurRowSize;
    int iCascadeFound = 0;

    // The interval based selection technique compares the pixel's depth against the frustum's cascade divisions.
    float fCurrentPixelDepth;
    fCurrentPixelDepth = IN.vDepth;
    
    // This for loop is not necessary when the frustum is uniformaly divided and interval based selection is used.
    // In this case fCurrentPixelDepth could be used as an array lookup into the correct frustum. 
    int iCurrentCascadeIndex;
    float4 vShadowMapTextureCoordViewSpace = IN.vTexShadow;
iCurrentCascadeIndex = 3;
//if ( fCurrentPixelDepth < m_fCascadeFrustumsEyeSpaceDepths[2] ) iCurrentCascadeIndex = 2;
//if ( fCurrentPixelDepth < m_fCascadeFrustumsEyeSpaceDepths[1] ) iCurrentCascadeIndex = 1;
//if ( fCurrentPixelDepth < m_fCascadeFrustumsEyeSpaceDepths[0] ) iCurrentCascadeIndex = 0;
if ( fCurrentPixelDepth < 300 ) iCurrentCascadeIndex = 2;
if ( fCurrentPixelDepth < 80 ) iCurrentCascadeIndex = 1;
if ( fCurrentPixelDepth < 40 ) iCurrentCascadeIndex = 0;
   
    float4 color = 0;   

// Repeat text coord calculations for the next cascade. 
// The next cascade index is used for blurring between maps.
    int iNextCascadeIndex = 1;
iNextCascadeIndex = min ( 8 - 1, iCurrentCascadeIndex + 1 ); 

    float fBlendBetweenCascadesAmount = 1.0f;
    float fCurrentPixelsBlendBandLocation = 1.0f;
    CalculateBlendAmountForInterval ( iCurrentCascadeIndex, fCurrentPixelDepth, 
fCurrentPixelsBlendBandLocation, fBlendBetweenCascadesAmount );
    
    ComputeCoordinatesTransform( iCurrentCascadeIndex, 
                                 IN.vInterpPos, 
                                 vShadowMapTextureCoord, 
                                 vShadowMapTextureCoordViewSpace );    
    
// hack in for now 4AM!!
if ( iCurrentCascadeIndex==0 )
{
result += float4(0.5,0,0,1);//tex2D(DepthMap1,float2(vShadowTexCoord.x, vShadowTexCoord.y));
}
if ( iCurrentCascadeIndex==1 )
{
result += float4(0,0.5,0,1);//tex2D(DepthMap2,float2(vShadowTexCoord.x, vShadowTexCoord.y));
}
if ( iCurrentCascadeIndex==2 )
{
result += float4(0.5,0.5,0,1);//tex2D(DepthMap3,float2(vShadowTexCoord.x, vShadowTexCoord.y));
}
if ( iCurrentCascadeIndex==3 )
{
result += float4(0,0,0.5,1);//tex2D(DepthMap4,float2(vShadowTexCoord.x, vShadowTexCoord.y));
}
    //CalculatePCFPercentLit ( vShadowMapTextureCoord, fRightTextDepthWeight, 
    //                        fUpTextDepthWeight, fBlurRowSize, fPercentLit );
                                             
if( fCurrentPixelsBlendBandLocation < m_fCascadeBlendArea) 
{  
// the current pixel is within the blend band.
// Repeat text coord calculations for the next cascade. 
// The next cascade index is used for blurring between maps.
//if( !SELECT_CASCADE_BY_INTERVAL_FLAG ) 
//{
// vShadowMapTextureCoord_blend = vShadowMapTextureCoordViewSpace * m_vCascadeScale[iNextCascadeIndex];
// vShadowMapTextureCoord_blend += m_vCascadeOffset[iNextCascadeIndex];
//}
ComputeCoordinatesTransform( iNextCascadeIndex, IN.vInterpPos, 
vShadowMapTextureCoord_blend, 
vShadowMapTextureCoordViewSpace );  
   
// We repeat the calcuation for the next cascade layer, when blending between maps.
if( fCurrentPixelsBlendBandLocation < m_fCascadeBlendArea) 
{  
// the current pixel is within the blend band.
CalculatePCFPercentLit ( vShadowMapTextureCoord_blend, fRightTextDepthWeight_blend, 
fUpTextDepthWeight_blend, fBlurRowSize, fPercentLit_blend );
fPercentLit = lerp( fPercentLit_blend, fPercentLit, fBlendBetweenCascadesAmount ); 
// Blend the two calculated shadows by the blend amount.
}   
}

// use fPercentLit to apply the shadow :)
//result = lerp( float4(0,0,0,1), result, fPercentLit );
    //calculate hud pixel-fog
    float4 cameraPos = mul(IN.WPos, View);
    float hudfogfactor = saturate((cameraPos.z- HudFogDist.x)/(HudFogDist.y - HudFogDist.x));
    
    //Mix in HUD Fog with final color;
    float4 hudfogresult = lerp(result,HudFogColor,hudfogfactor);
    
    //And Finally add in any Water Fog   
    float4 waterfogresult = lerp(hudfogresult,FogColor,IN.WaterFog);
    finalcolor=float4(waterfogresult);    
    return finalcolor;
    
}


/****** technique ********************************/

technique DepthMap
{
  pass p0
    {
VertexShader = compile vs_2_0 VS_Depth(); 
PixelShader  = compile ps_2_0 PS_Depth();
   }
}

technique ShadowMapping
{
    pass P0
    {
        // shaders
        VertexShader = compile vs_3_0 mainVS();
        PixelShader  = compile ps_3_0 mainPS();
        CullMode = None;
        AlphaBlendEnable = FALSE;
        AlphaTestEnable = FALSE;
    }
}

Signing Off

I am keen to get the shadows working before I forget all this so will be putting some hours over the weekend to complete it.  The good news is that all the complex maths and intricate techniques for producing great shadows and filtering them are in the C++ and Shader code, I just have to get the DirectX surfaces playing nicely so we can see them.  Hopefully a few more hours will crack it.  Wish my brain luck!