New Patch (450) - Engine Message Rework

Nov 5, 2007 at 11:08 PM
I've done a initial implementation of the message system rework, Shaw could you do some profiling on XBox and post the result?
Nov 6, 2007 at 2:02 AM
I'll try to get some GC numbers soon. Since its part of the engine now, I had to create Xbox project files for all of the assemblies, but I must have screwed something up because now I get a weird error while trying to deploy to the Xbox:

------ Deploy started: Project: QuickStartTest1_vce_xbox, Configuration: Release Xbox 360 ------
Deploying to Xbox...
Created container ce49de5c46034742aac0e69fcd884318 (replaces any prior container with this Title ID).
Deploying files from E:\projects\XNAEngine_update\framework\test\QuickStartTest1\..\..\..\bin\xbox\.
Deploy failed with the following error: startIndex cannot be larger than length of string.
Parameter name: startIndex
========== Build: 6 succeeded or up-to-date, 0 failed, 0 skipped ==========
========== Deploy: 0 succeeded, 1 failed, 0 skipped ==========

I can deploy other projects, but for some reason this one won't deploy. I posted it on the XNA forums, we'll see what they say.
Nov 6, 2007 at 3:11 AM
Apparently the deploy will only succeed if the project binaries are placed in a sub-directory of the directory containing the project file. So, going up to the parent folder of the project's folder is a no-no... weird...
Nov 6, 2007 at 4:43 AM
Alright, now that I got the project working on Xbox, I was able to profile the code. Since the messaging system is integrated with the whole engine now, I had to get the entire engine running on Xbox. So, good news, the engine works as-is on Xbox! Hazzah!

The engine's a bit of a mess GC-wise on Xbox currently, but I got it down to about 2.4 megs a second of allocations. Interestingly, before any code changes, it was at 82 megs a second! Apparently, you do not want to allocate SpriteBatch instances per frame. Only creating one instance in the GUI manager to be used every frame instead of creating a new one each frame brought the per second allocations from 82 megs to 2.5 megs, lol.

Anyway, on to the messaging system, the good news is the current messaging code accounts for 0% of the allocations. That's right, no allocations. However, I had to make a two small code changes to make this happen.

First, in QuickStartGame.Update():

lock (this.recievedMessages)
{
    Stack<EngineMessage> temp = this.currentMessages;
    this.currentMessages = this.recievedMessages;
    //this.recievedMessages = new Stack<EngineMessage>();
    this.recievedMessages = temp;
    this.recievedMessages.Clear();
}

Basically, instead of creating a new stack every frame, I just did a swap with the stack that's going to be deleted and cleared it. Practically no overhead (just a pointer copy) and no allocations. To get this to work the first frame, I just had to instantiate both stacks at class construction:

        private Stack<EngineMessage> currentMessages = new Stack<EngineMessage>(15);
        private Stack<EngineMessage> recievedMessages = new Stack<EngineMessage>(15);

With those two changes (and the window manager Render() method commented out), the allocations fell to 0 on Xbox.

I had a add/change a significant amount of the code in the transition to Xbox (sprite batch reuse, implementation of game pads, etc.), so give me a couple of days and I'll clean it up and commit. Then, when you finish whatever else you're doing with the messaging system, you can merge and commit.

Nov 6, 2007 at 4:46 AM
Edited Nov 6, 2007 at 4:47 AM
On another humorous note, after all of these changes to get the Xbox allocations down, my Windows framerate went from 215 FPS to 9800 FPS in the sphere/box scene, lol. That's not a typo.
Coordinator
Nov 6, 2007 at 5:14 AM
Edited Nov 6, 2007 at 5:14 AM
Was the framerate for Xbox only, or PC as well? That is awesome you got it running well on the 360. So if we are at 2.5megs per second, do that mean a GC 2.5 times per second?

As someone that understands the Xbox side of XNA, could you let me know if any of my current code will be incompatible with the 360? The only things I know that are for sure are anything having to do with mouse and keyboard, and stuff like setting Window.Title.
Nov 6, 2007 at 5:14 AM
I went ahead and applied the patch, as well as the Xbox project files and the garbage-cleanup code changes to the source tree.
Nov 6, 2007 at 5:21 AM

LordIkon wrote:
Was the framerate for Xbox only, or PC as well? That is awesome you got it running well on the 360. So if we are at 2.5megs per second, do that mean a GC 2.5 times per second?

As someone that understands the Xbox side of XNA, could you let me know if any of my current code will be incompatible with the 360? The only things I know that are for sure are anything having to do with mouse and keyboard, and stuff like setting Window.Title.


The frame rate figure was on Windows. I didn't bother to check the frame rate on Xbox. You are right, that's approximately 2 GCs a second. I did find the source though, and it's actually a known bug in XNA that's fixed in 2.0. The SpriteBatch.DrawString() method creates garbage on Xbox, but not on Windows, that's why I wasn't able to easily track it. I looked on the XNA forums, and its known and fixed. By commenting out the DrawString() call in the GUI button render method, the allocations went to 0, so we're essentially at 0 allocations right now on Xbox (once 2.0 hits).

It's hard to say what will be compatible and what won't when porting to Xbox. I could just blindly compile the project against the Xbox XNA assemblies, but all that will tell you is if you you're making function calls that aren't supported on Xbox, which I can't imagine you are. Keyboard and Mouse classes exist in the Xbox assemblies, so the code would compile fine, they just do nothing when executed on the Xbox. What you really need to look out for is the graphics hardware. Its possible that's a problem with a vertex format, shader variable format, etc. that we won't catch until the port. Unfortunately, you pretty much need to port the code to say for sure. Chances are any discrepencies between vertex format and shader input format (like we originally had, and I think I spotted again in one the recent releases) will trigger problems on the Xbox.
Coordinator
Nov 6, 2007 at 5:38 AM
Unfortunately you'll probably be the Xbox tester on this until we get me up and running with a 360 (sometime in January I am hoping). I really wish we had a dedicated graphics person on this project. I am doing what I can at the moment, but optimized shaders isn't my bag.

For instance I may get a skybox and skyplane up and running soon, and vegetation is likely going to be done soon as well. I am also hoping to be able to create a 3D cloud component (using billboards). Those things may look cool, but I can imagine what things we could do with a graphics person in here. There are also small graphics issues I have no idea on how to solve, like seeing under the terrain and entities where the waves meet them due to waveheight. And it would be really nice to get water volumes, volumetric fog, sun flares, sun billboard, HDR, depth of field, and all that fun stuff. Graphics, videos, and screenshots are going to be the main draw of any engine, at first glance at least. Then the deeper features like a GUI, Physics, A.I., animation, etc, are what keep people for the long haul.
Nov 6, 2007 at 5:40 AM
I went ahead and did an Xbox compile of your code... two issues:

  1. The Xbox hardware does not have fog render states, since it doesn't have a fixed-function pipeline. So your fog will have to become a pixel shader effect that you write yourself.
  2. The Xbox assemblies do not contain any of the Mouse* classes. I could of sworn they did, and they do include all of the Keyboard classes (dummy versions, anyway). Weird, but not really a big deal.

Since there are no game pad controls, I couldn't pan the camera to view the water, but I was able to see the terrain and weather effect in the default camera view. So, at least the majority of it works.
Nov 6, 2007 at 5:43 AM

LordIkon wrote:
Unfortunately you'll probably be the Xbox tester on this until we get me up and running with a 360 (sometime in January I am hoping). I really wish we had a dedicated graphics person on this project. I am doing what I can at the moment, but optimized shaders isn't my bag.


I definitely know my way around the graphics side of things, that was one of my original interests until I got more into high-performance systems/computing. I could devote my time to that now instead of physics, since you're right, its more important when trying to build interest.
Coordinator
Nov 6, 2007 at 5:44 AM
Edited Nov 6, 2007 at 5:47 AM
I will map the gamepad controls soon for you.

I knew about the fog states, lol, I completely forget to start a thread on it, I meant to last night. I was going to say we need someone who understands shaders rather well to go through each one and add fog to them. It would also be nice if we could different versions of shaders for people with older and newer cards to take advantage of each.

EDIT: I haven't worked with #if #endif with C# before, you may have to comment that stuff out in v0.179 after I release it. I am hoping v0.179 is the last version I release on the template framework. :)
Coordinator
Nov 6, 2007 at 6:03 AM
Wow, I forgot how gratifying it is to setup controls for a game through the gamepad, so much easier to control the entire game. I'm finishing it up right now, it will be in v0.179, which should be out in a couple days I am hoping.
Nov 6, 2007 at 8:36 AM
This is all very impressive, really good work Shaw, nice that you got the allocation down by 80Megs :) Sooo looking forward for Xna2.0 as it'll introduce so many new nice feature (and fix some annoying once).

LordIkon you can just change in your code into the source tree there is no reason to wait for the next release. Then we would have a look at how it works.
Nov 6, 2007 at 10:32 AM
I'm quite sure the 360 does fully support keyboard input, but I don't have CC so I cannot test it for myself.
Coordinator
Nov 6, 2007 at 2:57 PM
I'll post the changes I have tonight then. Vegetation is half finished, and skybox is glitchy.

I believe the 360 supports keyboard through the keyboard attachment for the 360 controller, which will be supported with XNA 2.0.
Nov 6, 2007 at 3:08 PM
XNA has always supported standard usb keyboards on the 360. Or at least since release.

http://msdn2.microsoft.com/en-us/library/bb203903.aspx
Nov 6, 2007 at 3:29 PM

Aphid wrote:
XNA has always supported standard usb keyboards on the 360. Or at least since release.

http://msdn2.microsoft.com/en-us/library/bb203903.aspx


Interesting. I could have sworn I read somewhere that keyboards on the 360 in XNA weren't supported. I tired it, and sure enough it works. I was able to control LordIkon's template code on the 360 with a keyboard.

LordIkon, in certain views, the template crashes on the Xbox with InvalidOperation exceptions on the Draw*Primitive() calls. I haven't had time to look at it, but just FYI.
Coordinator
Nov 6, 2007 at 4:09 PM
I believe the only drawprimitives calls so far are the terrain. But it is tough to test without a 360, thanks for the heads up though.