G.C. tests

Dec 17, 2007 at 7:48 AM
Edited Dec 17, 2007 at 10:27 PM
I've tested the garbage for my terrain patch + the current engine, and I've cut it down significantly. However I still get two types of data causing garbage. Vector3[] and and Single[]. I'm not instanciating Vector3[] anywhere. I attempted to test against your test patch you put up Shaw, and it won't run in my CLR profiler, very strange.

EDIT: n/m, had another profiler running bg on accident.

EDIT: Ok, I have verified that the garbage is being created in MY code, not the engine core, which matches results of your tests earliar Shaw. The patch that is up now won't reflect the GC I'm getting at the moment because I've gotten rid of some of it, especially by getting rid of foreach loops that were causing garbage. Still can't find the vector3[] and single[] garbage though.
Dec 17, 2007 at 10:08 AM
As Shaw has pointed out before, do not use IEnumerable (i.e. foreach loops). The most performant way of traversing a list is by running through it backwards, yes that's right it's faster running backward than forward.
Dec 17, 2007 at 10:37 AM
Can you tell me why looping backwards is faster? Just curious.
And on this occassion: I am still around and reading - just a little busy at the moment with my internship. But stress curve is about do decline now :)
Will do that splashscreen you'd been talking about in the other thread.
Dec 17, 2007 at 11:54 AM

Suzume wrote:
Can you tell me why looping backwards is faster? Just curious.

Not anymore, in the old days (Back in the 8086/68000 days) it was simply the structure of the bus/cpu which made dec faster than inc. All in all it simply comes down to clock cycles and how many are needed in order to do a single instruction. Since we write in a high-level language we are at the mercy of the optimizer, which has the important job of finding the most performant way of executing your code. But sometimes it simply gets the work done wrong. So why don't they fix it? Well there is a lot more to it than just that, a small tweak in a compiler might totally invalidate a lot.

Doing some research some time ago I also stumpled over this {url:http://diditwith.net/2006/10/05/PerformanceOfForeachVsListForEach.aspx} which is quite informative.

But besides fast we also have to be conserned about allocation and GC, since GC happens a lot more on the XBox than in Windows, once ever allocated meg (or something like that). Bacially every time GC happens we are just waiting, this means that loosing a few ms in order to prevent GC is acceptable.
Dec 17, 2007 at 2:13 PM
I've already eliminated foreach loop, but the fact is that a foreach doesn't always create garbage, it just depends on what type you're looping through.

Looping backwards is important for situations in which the list may be altered during the loop. If you were to remove something from the list then you'd still properly loop through everything, whereas looping forward and removing something during looping that is still in front of your iterator could result in some things being missed during the loop.
Dec 17, 2007 at 5:45 PM
How are you determining what's garbage in the profiler?

I'm seeing the majority of garbage being created by the BoundingFrustum.Intersect the first time it is called on a new instance. Since you're recreating the BoundingFrustum on Camera.UpdateFrustum, this is creating Single[], Single[][], and Vector3[][] garbage. Though, the sizes here are very small.
Dec 17, 2007 at 6:27 PM
Well I'm still new to the CLR profiler and GC, so I guess I have a few questions:

1.) How do I know the total amount of garbage I am creating? I see Gen0, Gen1, and Gen2 collection amount in bytes, is that the total?

2.) The GC seems to be running about every 8 seconds on Windows, but how do I know how often that is on the XBox? I know the Xbox collects after a megabyte of garbage, but I am not sure how to determine when a megabyte will be hit.

If the garbage amounts are very small then I won't worry about it, especially it is related to the frustum update, which is happening currently every frame, but that will be reduced by quite a bit when we're setting an update flag on the camera system.
Dec 17, 2007 at 7:47 PM

1.) How do I know the total amount of garbage I am creating? I see Gen0, Gen1, and Gen2 collection amount in bytes, is that the total?

Yes, those are totals and do not reflect "over-time" allocations/collections.

2.) The GC seems to be running about every 8 seconds on Windows, but how do I know how often that is on the XBox? I know the Xbox collects after a megabyte of garbage, but I am not sure how to determine when a megabyte will be hit.

A megabyte on Windows may be different from a megabyte on Xbox. For instance, a pointer type will be 32-bits on Windows (even on 64-bit processors, all XNA apps are limited to 32-bit), and 64-bits on Xbox. Also, the two CLRs may keep around different data for each reference type, so it's nearly impossible to use Windows memory quantity numbers. The number of allocations should be the same though.

Honestly, I wouldn't worry much about Xbox. Just concentrate on Windows development. Seeing the number of "interesting issues" on the XNA forums, and the XNA team's complete reluctance to provide information about the Xbox execution environment and code generation, I don't see us being able to support it reliably anyway. Which is a real damn shame! Let's see... I don't have the ability to examine generated code or even the Framework IL on Xbox (either would be sufficient), and I don't even have any "official" information on what code should be generated for my C#/IL. How am I supposed to effectively debug and optimize low-level code? File a Connect report and wait 6 months to a year for information about it, assuming its fixed in the next release? "Assume" perfect code generation and "assume" bugs are in my code? Usual a good probability, but far from 100%. Right now, XNA on Xbox is a play-toy, nothing more.
Dec 17, 2007 at 8:40 PM
Creating the engine solely for Windows is certainly an easy way out. And because I have no experience with the 360 I can't really argue with you on that. Compiled by the fact you have pay $100 membership just test on the 360, or to even play any games the engine could make.

If we do scrap 360 support for now I believe we should design the engine with the 360 at least in the back of our minds, so that if the XNA team can get things together to work on the 360 well then we wouldn't have huge steps to take to get it running on the 360.

I believe we should bring this up in another thread.
Dec 17, 2007 at 9:02 PM
Im against scrapping 360, as if it gets sorted wed have to start again (or close enough). I vote we work mainly with windows, and support xbox, but dont test it as extensively or worry as much about performance issues. Until they sort out the fee to even play a game, its a toy for anyone, even if they did release more compiler info. Can you imagine asking a customer to pay an additional $100 on top of the game price, the hardware and any Live fees. On that subject what are we targeting for? Indie developers, hobbyists, or what?
Dec 17, 2007 at 9:18 PM
I believe the people most likely to use our engine are either hobbyists (because its free and hopefully easy to use), and windows developers. If you follow the forums you'll notice that even if the engine is usable on the 360, the ability to sell your game to people on the 360, through XNA, is next to impossible because it would require they have a membership just to play your game.

This is the main reason I want to support sockets as well as live. Supporting sockets lets any windows developer choose our engine as a feasable solution.

I agree it may be worth it to make sure the engine at least runs on the 360 ok, but it will not be worth it to optimize the hell out of the engine to get it working on the 360 if it is going to add months to our development time and have no real benefits.
Dec 17, 2007 at 9:32 PM
We seem to be on the same page then :)

Ill definitely do sockets at some stage then if Windows is our focus, but I cant resist playing with Live. Ill probably end up doing it concurrently. What I dont understand thoguh is when you say "Supporting sockets lets any windows developer choose our engine as a feasable solution." Surely Live is? Its free, and works. I doubt any dev could reporduce the friends list or whatever other feature is a 'must have' as well as Live's. I would personally choose Live anyday, as many people have a gamertag for it. I understand what youre saying, about offering the choice, and I will do so, but I see few benefits with Sockets over live. Still its nice to have the option I guess :)
Dec 17, 2007 at 9:40 PM
I'm not saying we should scrap the Xbox builds, I'm just saying we shouldn't support it, or at least not at the same level as the Windows builds. Xbox support is one of the only reasons I use XNA for some of my game work. The Xbox builds will always be "as is," at least until the distribution issues and information availability issues are worked out. By the way, I think I'm on Shawn's bad side now. :)

With that in mind, it does make more sense to favor sockets over Live, as you're still stuck with the $150/yr issue on Windows if you use Live, and the fact that the G4WL redist is not actually redistributable, you need to require end users to install the full Game Studio, including VS Pro or Express.

As for target audience, I see it being hobbyist and student developers who want to get up and running quickly. Indies are a maybe, though again the distribution channels just aren't there unless you go with CD/DVD distribution, and quite frankly I think native development gives you much more opportunity when you're limited to PC hardware anyway. The appeal of XNA Game Studio (not to be confused with just XNA, which encompassed DirectX and all Microsoft game technologies) is two-fold:
  1. Xbox 360 game support (unlike PS3's CPU-only hobbyist environment), and
  2. Teaching the basics of game programming.

If you're limiting yourself to PC and are more concerned with features/functionality over ease-of-use, XNA Game Studio is most likely not the product for you. There's quite a few required redists now just to get a basic XNA game up and running on end user machines.
Dec 17, 2007 at 9:43 PM

mikelid109 wrote:

What I dont understand thoguh is when you say "Supporting sockets lets any windows developer choose our engine as a feasable solution." Surely Live is? Its free, and works. I doubt any dev could reporduce the friends list or whatever other feature is a 'must have' as well as Live's. I would personally choose Live anyday, as many people have a gamertag for it. I understand what youre saying, about offering the choice, and I will do so, but I see few benefits with Sockets over live. Still its nice to have the option I guess :)

Live is NOT free. That's the point. It requires Creator's Club membership plus Gold membership to matchmake on Windows, and you CANNOT distribute the G4WL redist with your game. You have to require end-users to install the full Game Studio product, including Visual Studio pro or Visual C# Express.

The more I think about this, the more it starts to really bother me!
Dec 17, 2007 at 9:53 PM
Silver is free, but I take your point on CC membership. Why cant you redistribute the G4WL exe, like the DirectX one? Is it a licensing thing, or is it not available fro us?
Dec 17, 2007 at 9:57 PM
Probably because of its use in commercial games.
Dec 17, 2007 at 9:57 PM
You need Gold to matchmake on Windows with XNA, unless you use System Link. You cannot include the G4WL redist with your game due to the EULA. The redist file is there in the XNA install directory, we're just not allowed to use it.
Dec 17, 2007 at 10:26 PM
Basically what it comes down to is that it would commendable if we had the engine working great on both Windows and the 360, but that it will take quite a bit of work to do so, and few people would notice. If we're focused more on Windows users we can squeeze a lot more from the engine I believe. I still would like it to run on the 360, even if it doesn't run great.
Dec 17, 2007 at 11:28 PM

Basically, Xbox CLR improvements are "planned," but nothing for the immediate future. I understand the technical hurdles, but why it's such an undertaking to get any information out of anyone regarding the Xbox CLR is beyond me.

Alright, I'll stop ranting now...
Dec 17, 2007 at 11:46 PM
You need an insider to get you face-to-face, or at least chatbox-to-chatbox with one of the CLR guys it sounds like.
Dec 18, 2007 at 1:00 AM

LordIkon wrote:
You need an insider to get you face-to-face, or at least chatbox-to-chatbox with one of the CLR guys it sounds like.

I've already been told most of the "technical information" involves trade secrets that cannot be shared. If I were to get any information, it would probably come with an NDA and I wouldn't be able to use it on any open-source projects. That's part of the problem: someone would need to actually document the Xbox CLR, then have the lawyers cut out 90% of it.

Guess we're not getting VMX support anytime soon. ;) Not that the Desktop CLR can even generate good SSE code.
Dec 18, 2007 at 4:00 AM
Edited Dec 18, 2007 at 4:07 AM
<evil, sadistic mode>

Anyone dare me to post the following table to the "How fast is it?" (http://forums.xna.com/thread/37309.aspx) thread on the XNA boards?

4x4 Matrix Multiply (10,000,000)  [EDIT: Core 2 Duo E6600 @ 3.0 GHz, for reference]
Managed XNA Matrix.Multiply(ref,ref,out):  557 ms
Managed w/ P/Invoke to SSE routine:        341 ms
Native C++ - Compiler Optimized:           300 ms
Native C++ - Hand-optimized SSE:           138 ms

A 5-page flame fest could be fun! :)

</evil, sadistic mode>
Dec 18, 2007 at 4:12 AM
How long does a manual matrix multiply take?

I've no problem with you stirring up the forums a little bit. If everyone is bitching about it then they might take a hint.
Dec 18, 2007 at 4:20 AM
What do you mean by manual matrix multiply?

I'm more joking than anything else. But those are real figures that I just measured using .NET 2.0 and the Visual Studio 2008 C++ compiler. For some reason, I'm in a sadistic mood, and the thought of a 5-page flame war with all of the .NET lovers is humorous. :)
Dec 18, 2007 at 5:07 AM
And just because I'm feeling spiteful and sadistic: :)

Managed XNA Matrix.Multiply(ref,ref,out) - Xbox:  4120 ms
Dec 18, 2007 at 5:28 AM
You're using a multiply function, what if you multiplied the matrix out on your own? In the presentations I saw that inlining seemed to speed things up a bit, I'm not sure if this applies to matrices though. Honestly I know how to multiply matrices, I'm just not sadistic enough to try it "longhand".
Dec 18, 2007 at 5:31 AM
Have you posted it? I love a good argument. Not that I'll have much to contribute.
Dec 18, 2007 at 6:07 AM
Good point, the .NET runtime won't inline that while the C++ compiler is more than happy to.

Managed XNA Inlined Matrix.Multiply - Windows:  466 ms
Managed XNA Inlined Matrix.Multiply - Xbox:     4923 ms

Apparently the Xbox version of Matrix.Multiply is different from the Windows version, since I just copied the Windows version from Reflector for the above figure. Or the Xbox chokes on 64+ bytes of stack allocations. I'll have to take a closer look at this.
Dec 18, 2007 at 6:50 AM
Edited Dec 18, 2007 at 6:50 AM
Well I really doupt that you will find anyone stating that Managed is faster than Native. I also think that this is an unrelated test. If you want the fastest possible engine, go write it in assembler, or machine-code if your are that kind of guy. I think that the comparison should be how difficult it is to write and get an acceptable result.

There are things you will end up doing in c/c++/asm in order to get the performance you need, but most likely the initial code was written in C# in order to see if it was possible. Also imagine if we were to implement this in c++ and have it run on both Windows and Xbox, how far do you think this project would be. How many would actually understand the code that would have been written?

So yes the managed code will always be slower than the same code written Native.
Dec 18, 2007 at 6:56 AM
Edited Dec 18, 2007 at 6:59 AM
1/10th speed for XNA to matrix multiply managed code. Ouch! You'd think one of the most common methods to 3D games could've been written better. Of course, I'm assuming you're doing this right.

Would using inlined code be a good idea for sections of the engine that need better performance? I know it can look ugly, but if it is mostly code that nobody else will be touching then I wouldn't mind.
Dec 18, 2007 at 7:10 AM
Hobbyist C++ on Xbox is non-existance nor will it probably ever be possible. Unless you hack your Xbox 1 or get an Xbox 360 dev kit off eBay. Not that either of those are strictly legal.

My problem is not directly with the performance and prototyping abilities of C#/.NET. To the contrary, I find .NET easy to use and it definitely has its uses. My problem is with people who try to push it as a replacement to C++ and claim that C# is as fast if not faster in all cases. This is just simply not true.
Dec 18, 2007 at 7:11 AM
1/10th? Where are you getting that figure?
Dec 18, 2007 at 7:15 AM
Inevitably, .NET people love to bring up the "competition" between an "expect C# programmer" and an "expert C++ programmer" where the C# programmer beat out the C++ programmer in terms of program performance up until the end when the C++ programmer pulled out some really nasty, ugly tricks to get the final edge. This is seen as a great accomplishment for C# because it was so much easier for the C# programmer to get just as good as performance with clean code. But... the program was text processing? Who has ever claiming that C++ was the master of text processing? Guess what... that comparison has absolutely no bearing on tight math code!
Dec 18, 2007 at 7:28 AM

shawmishrak wrote:
1/10th? Where are you getting that figure?

I meant 1/10th the speed for the Xbox as compared to Windows. I was referring to the ~4900 and ~470ms figures.
Dec 18, 2007 at 4:14 PM
This is so very depressing. The xbox, with all its next gen hardware, is a lot slower. I am likeing the xbox xlr less and less the more I read.
Dec 18, 2007 at 5:13 PM
It is unfortunate. The hardware itself is actually fairly nice, I think it is just the CLR that we have to work with for XNA that shafts us.
Dec 18, 2007 at 5:14 PM
I've had issues with the Compact Framework on Xbox since 1.0 first came out.

Without looking at the PPC assembly dumps from the Xbox CLR (which is impossible), I can only speculate here (again, some documentation would be nice!), but my guess is that it just uses a fairly unoptimized direct code generator. My tests have shown that it does not unroll even simple loops, and I have a strong feeling that it doesn't do any instruction scheduling optimizations, which is important on hardware like the Xenon processor. Also, floating-point performance will never reach optimal levels until VMX (equivalent of SSE) is exploited, but you cannot do that manually in pure managed code, and I don't expect the Xbox CLR to ever support VMX code generation.
Dec 18, 2007 at 5:33 PM
Shame. I take it youve got CC subscription Shaw? When I get some Live network code up could you test I please as havent got mine yet. Wiating till at least the new year.
Dec 18, 2007 at 6:05 PM
Yeah, I can test it out for you.
Dec 20, 2007 at 7:21 PM
If anyone's interested, I finally posted that table along with an article-length summary on the XNA boards. http://forums.xna.com/2/38297/ShowThread.aspx#38297
Dec 20, 2007 at 7:45 PM
Very nice, its unfortunate you didn't post the Xbox results, I wanted to see that spark a little 'discussion'. And by discussion I mean feud.
Dec 20, 2007 at 7:53 PM
I would have, but ZMan already mentioned it in the thread:

ZMan said:

Note that we are all well aware of the floating point issues on the 360.

Dec 20, 2007 at 7:54 PM

LordIkon wrote:

And by discussion I mean feud.

And by feud, you mean "ShawMishrak gets banned from the XNA boards for provoking yet another Xbox performance-related flame-war." ;)
Dec 20, 2007 at 7:56 PM
Yes, but by 'we' he meant those who are already familiar with Xbox CLR issues. Posting the numbers for the public to see might make the issue a little less transparent. But as Shawn said, the CLR guys are very busy as it is, so it just depends on how much of a priority they place on it.
Dec 20, 2007 at 8:02 PM
I definitely understand that argument, and I'm sure the Compact Framework guys have their hands full. What I don't like about the Xbox environment is the complete lack of technical information. If you look hard enough on the internet, or are familiar with x86 assembly, you can find out just about anything you need to know about the Desktop CLR. That's not the case on Xbox. There's no technical documentation on the Xbox CLR, and so far the Xbox environment has resisted all of my attempts at using Reflection and unsafe code to get at the IL or PPC disassembly on Xbox.

It's funny how the disassembly code I use (using Reflection) works just fine on Windows, but crashes on Xbox. All the methods are there, but Game Studio Connect mysteriously dies when the code is executed. :)
Dec 20, 2007 at 8:06 PM
I wonder if it could be a security precaution.
Dec 20, 2007 at 8:10 PM
I'm sure it is! I bet they had to lock that down tighter than Area 51 to get it to pass certs, especially after people were able to hack the Xbox and run unsigned code by modifying a shader on the King Kong demo disk.

My favorite so far is the UnauthorizedAccessException when trying to get a directory listing of the XNA system files. It acts like I'm trying to hack into a government mainframe!
Dec 20, 2007 at 8:12 PM
Originally I was trying to copy over the Microsoft.Xna.Framework.dll and Microsoft.Xna.Framework.Game.dll assemblies from the Xbox so I could get a look at them in Reflector to see what's different between them and the Windows versions, but that's locked down very tight. Can't even get a directory listing, lol.
Dec 20, 2007 at 8:23 PM
Sometimes I understand hackers. But I also understand Microsoft's security concerns......I just don't care. :oP
Dec 20, 2007 at 10:37 PM
I guess you could consider some of the stuff I do on the Xbox as "hacking," though it's never with a malicious intent. I just want to know how it all works, especially the XNA components, with the end goal of writing the best, fastest possible software given the environment.
Dec 21, 2007 at 2:24 PM
So youre a white-hat hacker? :)
Dec 21, 2007 at 4:12 PM
I had to look that one up on Wikipedia. :)

But yeah, I suppose that's mostly accurate.
Dec 31, 2007 at 10:30 PM
It's a damn good thing we don't need to support Mono. I just tried out the C# matrix math test code I had written for this thread on Mono in Fedora Core 8, and the execution time was over a second. Ouch!
Jan 16, 2008 at 9:17 PM
I am seriously considering abandoning the XNA forums. There are way too many egotistical children on there who take arguments personally and refuse to look at facts!
Jan 16, 2008 at 10:58 PM
Did I miss an important thread/post? Please send link :)
Jan 17, 2008 at 12:14 AM
I think he's refering to this nice thread about (or at least became) C++ vs C#, Green vs Blue, Taste vs Smell etc.
Jan 17, 2008 at 6:36 PM
That's one of them, yes.

If people would just say, "I disagree with you. Here's why..." I would be cool with it. I'd probably learn something. If the arguments were valid, I'd reconsider my own thoughts. But all I keep seeing is no, you're wrong. No explanation, no supporting facts.

I'm just so sick of people preaching opinion as fact. Arguments like: "We have C# on Xbox now; C# is more productive than C++ all the time; therefore, C# will replace C++ in the entire games industry." Yes, that's paraphrased and somewhat exaggerated, but a lot of posts have that sort of logic to them. There are so many things wrong with arguments like that, I cannot even comment on them all.

I'm not claiming C++ is a better overall langauge. Neither am I claiming C# is. They both have their purposes, use whichever fits your project best. Making blanket statements comparing them in terms of absolutes is just stupid. Too many people expose themselves to just C#, and then make arguments against C++ without knowing all of the facts. So what do I do to make my point? I make a post showing a case where C++ is better suited to a particular task. I say I'm not claiming C++ is superior, just faster for a particular task; as an example. I explicitly state this in the post! What are the comments? The test isn't fair. You used non-standard language features. You used platform-specific code. That code won't work as-is on Xbox. Does this have anything to do with the point I was trying to make, which was apparently completely missed.

People take this shit waaaay too personally. So what if I post some experimental results about one particular operation? Is it really necessary to bitch about how unfair the test was just because you don't like the results? If the unfair argument held any truth, I would redo the test in a heartbeat to make it as fair as possible. But no, I just get bitched at for spreading misconceptions. How many others have posted actual empirical performance evidence?

Is this game programming, or the Spanish Inquisition. I'm not sure anymore. I'm just so sick of this shit.
Jan 17, 2008 at 6:58 PM
I think that Shawn Hargreaves nailed it better than any one statement I've ever seen:

Shawn Hargreaves wrote:
I find it amusing how if you talk to many of the hardcode "I could never use C#, it's just too slow" crowd, you eventually find out that although their graphics engine is pure C++, they turn out to be using Lua (a garbage collected, bytecode interpreted runtime) for all their game logic and AI scripts :-)

Just for reference, I'm not going to get dragged into any discussion on C++ vs C#. I totally agree with you shaw, they each have their valid use.
Jan 17, 2008 at 7:13 PM
Edited Jan 17, 2008 at 7:13 PM
I agree as well, that is why we've distanced this project from the 360 as much as we have. Windows XNA is quite a bit more managable.

What is comes down to, I believe, is that 95% or more of the members on the XNA forums don't understand the low-level details of why XNA cannot perform "great" on the 360 in its current state.

To me, the discussion of differences between C# and C++ is a waste. Yes, managed has some overhead, but not so much you can't make a great game out of it. The discussion needs to be over what XNA lacks, and how to fix it, or if it will ever be fixed, and what implications that has on game designers and/or hobbyists.
Jan 17, 2008 at 7:13 PM
Edited Jan 17, 2008 at 7:15 PM
I liked the power of c++ when I used it, but found it difficult to self teach, so moved on to .net and c#. It was familiar (ish) as I began with Java :)

Before anyone asks, it was really pointers that scared me off. :) I agree each language has its uses, but I must admit .net is great for getting started on. I really should get back to c++ one day though...

EDIT: Ikon posted while I typed. I think hes got the best take on this ive yet to read compared to the official forums.
Jan 17, 2008 at 7:15 PM

mikelid109 wrote:

Before anyone asks, it was really pointers that scared me off. :)

Pointers are one of the more difficult things to grasp, and they can be dangerous to use, but they're incredibly powerful. I would advise taking the time to learn them in detail, if you ever plan on creating high-performance code for C++ you will need it.