Dec 13, 2007 at 6:55 PM
Edited Dec 13, 2007 at 7:18 PM
Now that XNA 2.0 is out, I was able to port everything to the Xbox and see how well it runs. I also ported the floating-point math benchmark I wrote for 1.0 to see how much the CLR has improved with floating-point code generation.

  • The Good
    • The entire codebase compiled fine on Xbox with only two notable changes:
      • The float.TryParse is not implemented on the Xbox, so I need to use an exception block to test for input errors when loading materials directly from disk. Not a good thing, but you'll probably never be loading materials from disk without going through the content pipeline on Xbox anyway. So, a hassle, but not a big deal. Ex:
        value = float.Parse(xmlNode.InnerText);
        // Output parse error

      • Multi-sampling switched from NonMaskable to TwoSamples on Xbox.
    • With MSAA disabled at 1280x720, a Release build on Xbox was getting the following frame rates:
    No Blur    Blur    Shadow Map Size
    -------    ----    ---------------
    334        156     1024x1024
    137        49      2048x2048
    • These numbers are pretty decent, especially considering there is still lots of room for improvement.
    • With the current codebase, we're spending less than 1/1000th of a percent of our time in garbage collection on Xbox. That's very good! Whether or not this can be attributed to our code, or improvements in the GC I don't know. I'll have to take a closer look at how the GC operates now. But we're only allocating about 4600 bytes a second so I'm thinking its just our code.

  • The Bad
    • I cannot get MSAA to work on Xbox. With it enabled, I constantly get OutofMemoryException errors. I'm not sure what's really going on, because it even happens if I take the shadow map size down to 64x64, and we're not using that much memory to begin with. I'll have to take a much closer look into this.
    • The floating-point math benchmarks are kind of disappointing. Overall, the average improvement on small data sets (i.e. matrix multiplications on 128x128 matrices) was 1.8%. Moving up to matrix sizes of 256x256, the average improvement fell to -0.5%. (That's not a typo, it was negative). These "improvements" are small enough that they could be attributed to statistical anomalies, but the bottom line is: floating-point performance was not improved in this release.

Overall, I have to say that I'm slightly disappointed with this release. The only new feature that I find useful is the VS Pro integration. I still have yet to take a closer look at the Xbox CLR, but from a first glance I don't think they really did much. This release seems much more focused on Live support.
Dec 13, 2007 at 7:34 PM
Edited Dec 13, 2007 at 7:38 PM
Great stats Shaw, thank you.

There were some other small improvements right? Like SpriteBatch calls not causing performance issues, and other things not performance related, like more reliable cross-platform use of render targets. Probably some other things I can't think of at the moment.

I agree it wasn't a huge change though. My guess is that the team spent most of the last year trying to get networking and Live working well, and spent less time on CLR and performance.
Dec 13, 2007 at 7:40 PM
Edited Dec 13, 2007 at 7:41 PM
Oh, and about GC, I have a feeling once I get the terrain system inplemented, you and sturm will not only need to review it thoroughly (as it will be a very big patch), but also test it for GC. I'd spent days with CLR profiler trying to reduce the amount of memory terrain used (in the template), but I'm still new to the profiler and I don't see where some of the garbage is coming from. I may need a GC lesson from you guys so that all my future code can account for the Xbox. I've never had an Xbox to run things on, so I've never had a problem with GC.
Dec 13, 2007 at 10:53 PM
Upon further examination, there are a number of little issues that are cropping up in the Xbox build. I'm going to need some time to track them all down, but they may require some significant interface changes, especially in scene manager. There seems to be an issue with loading graphics content outside of Game.LoadContent. I'm not sure what the exact problem is, I'll let you all know when I find it.

I finally got MSAA to work, but its causing some more interesting problems of its own. This is giving me a real headache! Of course all of these errors just throw DriverInternalErrorException's, and not some sort of meaningful error!
Dec 13, 2007 at 11:41 PM
Well I'm glad 2.0 come out now, it would've been a bummer to spend another few weeks on the engine and then start testing.
Dec 14, 2007 at 12:21 AM
Damn, there are some really nasty issues here. For instance, I'm having EffectParameter.SetValue() calls randomly throw DriverInternalErrorException's, and I can't consistently reproduce it for the life of me!
Dec 14, 2007 at 12:54 AM
I wish I had a 360, I could join you in debugging this one. All I can say is good luck I guess.

I thought XNA was supposed to give similar effects on both platforms.....doesn't seem like it., Not as much as I thought at least.
Dec 14, 2007 at 5:30 AM
Alright, after hours of tearing into our framework, I think I've finally found the problem: ContentManagers.

Basically, on Xbox, you need to call Unload() for all of your ContentManagers (besides the one included in Game) when Game.UnloadContent is called (for instance, when the graphics device is recreated). Then, in Game.LoadContent, reload all of the graphics content. I was already reloading everything in LoadContent, but without calling Unload when UnloadContent is called, it just gives you content with the old GraphicsDevice reference, which become invalid after some arbitrary amount of time. That's why it was so frusterating. The problem was not caused by the statements being executed, it was caused by earlier calls. I was under the assumption that this was one of the purposes of creating the "virtualized" graphics device, to avoid this issue, but I'm still waiting to hear back on the XNA boards.

Dec 14, 2007 at 5:55 AM
Now that everything seems to be working on Xbox, I have some new performance figures, with MSAA this time:

    Blur    No Blur    Shadow Map Size
    ----    -------    ---------------
    320     440        512x512 2x MSAA
    144     300        1024x1024 2x MSAA
    47      130        2048x2048 2x MSAA
    366     533        512x512 No MSAA
    150     330        1024x1024 No MSAA
    48      137        2048x2048 No MSAA

We're definitely going to have to spend some time discussing how to handle graphics device resets, since we have to dispose of all of the content when Game.UnloadContent is called and reload the content when Game.LoadContent is called. You wouldn't believe how much time I spent going through code today just to find that all of these weird bugs are caused by not calling ContentManager.Unload.
Dec 14, 2007 at 6:03 AM
Edited Dec 14, 2007 at 6:04 AM
I'd say 512x512 is a good candidate for a little bit of quality (2x MSAA), and a good amount of speed. Does a shadow map need to be a power-of-two? Not that it really matters I guess.

I agree we need to setup a good framework that each class/interface/whatever must follow so we don't end up with this bug again, especially since we're not all able to test on the 360. Unfortunately those with a 360 will need to do more of the patch reviews until other (like myself) get a 360.

Good job though Shaw, thanks again.
Dec 14, 2007 at 6:49 AM
No, there is no requirement that a shadow map be a power of 2. In fact, while diagnosing this issue, I filed a couple of Connect reports about problems with 2001x2001 render targets. Apparently, SurfaceFormat.Vector2 render targets with 2x MSAA cannot be larger than 2000x2000 without causing DriverInternalErrorExceptions and SurfaceFormat.Color render targets with no MSAA cannot be larger than 2000x2000 without completely locking up the Xbox. :)

At this point the astute reader will wonder how I was able to provide FPS figures for 2048x2048 shadow maps (in SurfaceFormat.Vector2 format). The shadow mapping targets don't use multi-sampling. The multi-sampling is only applied to the back buffer. So, really, in the figure above, the multi-sampling does not really apply to the shadow mapping algorithm. It only refers to the multi-sampling of the final back buffer using the created shadow maps.
Dec 14, 2007 at 6:51 AM
Also, for light sources like the sun, we almost need to use 2048x2048 maps. Otherwise, we can't possibly hope to shadow the entire viewable landscape with any decent quality. Look at how 512x512 looks on the Windows sample. It's decent, but not "good." And that's only a small piece of geometry.
Dec 14, 2007 at 8:48 AM
Shaw can you check in the changes you have done to the code in order to make it work on the Xbox360?
Dec 14, 2007 at 3:44 PM

shawmishrak wrote:
Also, for light sources like the sun, we almost need to use 2048x2048 maps. Otherwise, we can't possibly hope to shadow the entire viewable landscape with any decent quality. Look at how 512x512 looks on the Windows sample. It's decent, but not "good." And that's only a small piece of geometry.

Have we considered shadow volumes?

When I setup shadow mapping I had huge issues with the sun because of the distance. The only solution I found was PSSM, which have a much better look at lower shadow map sizes, but has lower performance than standard shadow mapping.
Dec 14, 2007 at 3:48 PM
Uploaded to Change Set 8576.

The significant changes for this version:
  • SceneManager now has LoadContent/UnloadContent, which are called from Game.LoadContent and Game.UnloadContent, respectively. Content loading code in SceneManager has been moved to LoadContent.
  • Xbox graphics device created with MultiSampleType.TwoSamples instead of MultiSampleType.NonMaskable when MSAA is requested.
  • Material system will not work by directly reading XML materials from disk on Xbox. This shouldn't be a problem as the content pipeline should be used anyway.

There's still a problem in SceneManager, since it keeps copies of expired content after a graphics device reset, but that code is temporary anyway and doesn't cause an issue since that content is never used.

Note that a large chunk of the work I had to do yesterday is not applicable here since a lot of it had to do with the graphics system changes that are not in the source tree yet.
Dec 14, 2007 at 7:47 PM
Just submitted bug #3 to Connect. :(

Dec 14, 2007 at 8:06 PM
Maybe 2.0 refresh will come sooner than I thought. This could be a good thing, keep finding those bugs. :oP