Terrain Porting

Nov 2, 2007 at 1:17 PM
I would like to start porting over the terrain code from the old quickstart to the new framework. if noone else has already started.
Is there anything i need to be aware of that might break me ? my aim is to get what is there now but with the new coding guidelines applied and with it in the new structure. We can then take it from there :)
Nov 2, 2007 at 1:27 PM
We need to at least discuss how the port should interact with the new framework. I'm sure there will be changes to the design of the terrain system. For instance,

  1. What's the relationship between the terrain, water, and quad-tree components?
  2. Should the quad-tree implementation become a scene manager, or just a scene node?
  3. Should terrain and water be separate or combined?

These are just a hand-full of questions I can come up with off the top of my head, and we should discuss them before the porting starts.

I'd like to head LordIkon's thoughts on the port.
Nov 2, 2007 at 1:35 PM
During porting of any system you should also consider which components should be exposed to the console, that is which properties should be changeable runtime. There will be a general way of registering this implemented at a later time, but thinking about this now and implementing a mockup (maybe just a simple method call) would be good. Also if there are special requirements to this please let me know
Nov 2, 2007 at 2:55 PM
Yes it does need to be discussed. hence the post.

The quadtree is a good question, i think this should be the scene manager as it can then be used to increase performance of rendering world objects.
My personal preference to the water and terrain would be to keep them together. unless you can think of a situation where you would want water and not interested in terrain.

Im currently just looking at ways this would fit in, i dont expect to do any code porting for a couple of days at least until i have a good idea of what the outcome will be.

if anyone has already put some thought into this and has ideas please post.
Nov 2, 2007 at 3:11 PM
I could imagine having liquids in containers which are not terrain.
Nov 2, 2007 at 3:13 PM
uhm.. that is a fair scenario, just trying to think if it would still need parts from the terrain. if i cant think of a reason then thats a good enough reason for it to be seperate if everyone else agrees
Nov 2, 2007 at 3:22 PM
Ok one of the first things that should be decided IMHO is
Which namespace is it going to sit in ?

I think common is a sensible place for it. QuickStart.Common.Terrain & QuickStart.Common.Water
Another option would be to have it as another seperate project, but id personally like to keep dll count down, and i see it as being one of the most commonly used things after the camera.
Nov 2, 2007 at 3:27 PM
It seems like it would be reasonable to define water as a volume instead of a plane. Then, you can insert water volumes into the scene, and have greater control over where the water is. The "water" component shouldn't depend on the terrain component. Even for reflection/refraction, the renderer should take care of generating the appropriate reflection/refraction maps given arbitrary world geometry. I can think of any number of situations where I'd want water below say z=0 for a part of the world, but then not have water for some underground cavern below z=0.
Nov 2, 2007 at 3:35 PM
Yeah i totally agree with that scenario also... this would throw up some interesting questions with the physics implementation as it would need to work against both water and terrain combined and seperate, this could quite easily introduce a massive bottle neck into the systems performance, what do you think? there is essentially going to be almost two terrains with every scene that has water and terrain. the water would also need to be part of the quadtree
Nov 2, 2007 at 3:38 PM

conkerjoe wrote:
Ok one of the first things that should be decided IMHO is
Which namespace is it going to sit in ?

I think common is a sensible place for it. QuickStart.Common.Terrain & QuickStart.Common.Water
Another option would be to have it as another seperate project, but id personally like to keep dll count down, and i see it as being one of the most commonly used things after the camera.


That depends on the design. We still haven't decided on the level of separation between Terrain and QuadTree. It only makes sense for the QuadTree implementation to become a scene manager, though it will need to be adapted to do a lot more than it currently does in order to become a full scene manager.

As such, QuadTree should be pulled out into a separate assembly. That way, clients can decide whether or not to reference it and create instances in their games. Ultimately, there will be several different kinds of scene managers.

The water volumes are certainly common enough to be put in the common assembly. It actually seems like more of a scene node, to be placed in the scene manager.

I'm honestly not sure about the Terrain component. I think it makes sense to make it a scene node, probably as a root node. It done right, I think it can be implemented in such a way that it is decoupled from the quad tree implementation and will work with any scene manager. That way you can have a small terrain patch placed into say a BSP scene.
Nov 2, 2007 at 3:41 PM

conkerjoe wrote:
Yeah i totally agree with that scenario also... this would throw up some interesting questions with the physics implementation as it would need to work against both water and terrain combined and seperate, this could quite easily introduce a massive bottle neck into the systems performance, what do you think? there is essentially going to be almost two terrains with every scene that has water and terrain. the water would also need to be part of the quadtree


I'm not too worried about physics. The terrain collision detection and response will be the primary concern. With the water volumes, all you need to do is detect if the object is intersecting with the water volume and apply appropriate impulses. It's not a constraint that needs to be mathematically solved like penetration constraints.
Nov 2, 2007 at 3:49 PM
im not sure how a qaudtree terrain would work without direct connection to the quadtree. as you sugest work with any scene manager.

Im not overly thrilled at the amount of assemblies there are at moment and adding more will just cripple the system in the long run. There is a point where too much seperation applies, especially with the overhead of getting a seperate assembly into memory, and cross assembly calling. I agree it shouldnt be in 1 big assembly but i also think 5 is too many and any more would make it very dificult to work with as a developer on it and as a user.
is there a paticular reason it needs to be so seperate in terms of assembly count? seperation can still be achieved with namespaces

I would hate to see this change from a quickstart engine to a intense 3d FPS gruelling engine, where you need to be a rocket scientist to render a square on the screen.
Nov 2, 2007 at 4:10 PM

conkerjoe wrote:
There is a point where too much seperation applies, especially with the overhead of getting a seperate assembly into memory, and cross assembly calling.


If we were working in native code, I would agree with you. Calling across DLL boundaries natively incurs a small performance hit. However, this is not true in the managed world. To the CLR, there is no difference if a class is in the executable or a references assembly, it all gets compiled the same.

In the native world, compilers can do a better job of optimizing and sometimes even eliminating function calls when everything is linked into one monolithic executable. In the managed world (with the exception of C++/CLI), there is practically no optimization done at compile-time, even in release builds with optimize turned on in the C# compiler. The CLR at run-time will take all of the assemblies referenced by an executable and compile it all down into one executable image and do whatever optimization it can. At the moment it's not entirely efficient, but that's not something we have control over with XNA or .NET in general, unfortunately.


conkerjoe wrote:
is there a paticular reason it needs to be so seperate in terms of assembly count? seperation can still be achieved with namespaces


It was a conscious design decision that was aimed at allowing developers to pick and choose what they want to use. The only times it ever creates more work on clients is when a project is first created and the assemblies must be referenced, and when the installer is created. In both cases, the amount of extra work is negligible.
Nov 2, 2007 at 4:12 PM

conkerjoe wrote:
im not sure how a qaudtree terrain would work without direct connection to the quadtree. as you sugest work with any scene manager.


The terrain itself wouldn't be quad-tree aware. The terrain would just define "cells" of terrain, that the quad tree scene manager could use, as well as any scene manager. The quad tree scene manager would do an efficient job of managing the terrain cells to create vast terrains, while a BSP scene manager may only load up one small terrain cell is a portion of the map.
Nov 2, 2007 at 4:34 PM
"The CLR at run-time will take all of the assemblies referenced by an executable and compile it all down into one executable image and do whatever optimization it can"

Surely this isnt true, surely code is compiled on first run (JIT) Just in Time., therefore if you never called into the physics dll it wouldnt be JIT'd. if im wrong then my last 5 years of .net work i may as well throw out the window because .Net is something completely different to what i thought it was. Cross assembly calling is one of the biggest overheads when i run profiles against our code in my day job.
We actually completely re engineered the assembly structure from 15 assemblies to 5 in a large project, with no code change other than copying and pasting into the new files inside an existing project, we increased performance by 90%

now this project did have a lot of intesive stuff going on in the cross calling so a 90% saving wouldnt be standard across the world but there was definatly an increase when managing my dll count in every project iv worked on.


"it was a conscious design decision that was aimed at allowing developers to pick and choose what they want to use. The only times it ever creates more work on clients is when a project is first created and the assemblies must be referenced, and when the installer is created. In both cases, the amount of extra work is negligible." this my point exacly. if all the hard work is having to be done before the end user can actually see any results, then its not exacly starting quickly, and users would come down like a ton of bricks on the QuickStart name of the project.

I totally agree that users need to be able to choose which bits they want in their project and which they dont, but im sure there must be a better way to implement this than make a mockery of the quickstart name ?? if im wrong and out of line then i'll wave a white flag, but i do believe this makes a mockery of the QuickStart name.
Nov 2, 2007 at 4:42 PM
I am now really confused about CIL and JIT. :S
Nov 2, 2007 at 4:48 PM

conkerjoe wrote:
Surely this isnt true, surely code is compiled on first run (JIT) Just in Time., therefore if you never called into the physics dll it wouldnt be JIT'd.


That's really up to the CLR, but it tries to do as much as it can up-front. Of course you can always dynamically load code that will be JIT'ed after you load the assembly. So if there are any references to a class in the physics dll, it will probably JIT it at start-up. Now, if the assembly is referenced but no code ever make reference to anything in the assembly, then it may decide to ignore it.

When you say cross-assembly calling, how was it implemented? Were assemblies loaded dynamically at runtime with a call to Assembly.Load()? If so, then of course you're going to take a hit at that point while the CLR JITs the code. Were these assemblies loaded in separate AppDomains? If so, that's where your performance hit was. I think there's a lot more to your story than just cross-assembly calling.

Even in the native world, a virtual call is going to be worse than a cross-DLL call. A standard function call across DLL boundaries has practically no overhead.



conkerjoe wrote:
this my point exacly. if all the hard work is having to be done before the end user can actually see any results, then its not exacly starting quickly, and users would come down like a ton of bricks on the QuickStart name of the project.


Hard work? It's just a matter of pressing Ctrl and hitting the mouse button one extra time to select several assemblies instead of one. It's not like we're forcing users to call Assembly.Load(), then use Reflection to get to our classes.
Nov 2, 2007 at 5:03 PM
firstly how do i quote in codeplex?? lol

in the case i put above there was no using reflection to load assemblies. it was direct references.


"
Hard work? It's just a matter of pressing Ctrl and hitting the mouse button one extra time to select several assemblies instead of one. It's not like we're forcing users to call Assembly.Load(), then use Reflection to get to our classes.
"
Its not "hard work" as such, selecting a couple of dlls, but without FULLY understanding the engine how would you know what to choose ? i get the picture of someone doing trial and error until they have what they want, again this isnt exacly QuickStart.
If the mega assembly count stays then maybe we could make the choice easier somehow, maybe with a gui interface to make it more obvious which dlls contained which bits. this would also give the user a basic understanding of the engine structure without them having to traul through documentation to understand.
I know we're a long way off it being a complete engine where this kind of stuff would be very important but something like assembly seperation should be decided on from the outset. If we stick with the dll count and more then at least its been discussed and curve balls have been batted away making us more confident its the right choice.
Nov 2, 2007 at 5:19 PM

conkerjoe wrote:
firstly how do i quote in codeplex?? lol


That above would be:

{quote:}
conkerjoe wrote:
firstly how do i quote in codeplex?? lol
{quote}

Hitting reply to the right of the message instead of just "Post Message" or whatever the button is will auto-quote the whole message.


conkerjoe wrote:
Its not "hard work" as such, selecting a couple of dlls, but without FULLY understanding the engine how would you know what to choose ? i get the picture of someone doing trial and error until they have what they want, again this isnt exacly QuickStart.
If the mega assembly count stays then maybe we could make the choice easier somehow, maybe with a gui interface to make it more obvious which dlls contained which bits. this would also give the user a basic understanding of the engine structure without them having to traul through documentation to understand.
I know we're a long way off it being a complete engine where this kind of stuff would be very important but something like assembly seperation should be decided on from the outset. If we stick with the dll count and more then at least its been discussed and curve balls have been batted away making us more confident its the right choice.


Well, for one, there's a direct 1-to-1 relationship between assembly name and namespace. The QuickStart.Physics assembly will not contain any code in the QuickStart.Common namespace. Second, its a matter of: "if you use namespace QuickStart.x, then make sure you include the assembly QuickStart.x". Third, if someone is trying to use the engine without reading any of the documentation, tutorials, online help, etc., which will explain how to do this, then they deserve for me to beat them with the manual! They're going to be following a tutorial, or read the documentation, or something.
Nov 2, 2007 at 5:35 PM
cool thanks for the tip on quotes. hope this works

The one to one relationship on namespace would make things much easier to guess at... but how does someone guess whats inside common ?


if someone is trying to use the engine without reading any of the documentation, tutorials, online help, etc., which will explain how to do this, then they deserve for me to beat them with the manual! They're going to be following a tutorial, or read the documentation, or something.


If this is the actual response to the problem then i dont really hold much hope for anyone other than a fully fledged games developer being able to use this.

for starters i dont believe for one second the help and documentation would be up to standard any time soon, or anytime around release of code for that matter. getting documentation right takes an incredible amount of resource and time, we may have the odd document but i dont think we will have intros, tutorials documentation up to the standard that a newbie user would require to get going at any time around release.
maybe im wierd or something but i wont touch documentation until i need to.. if something starts up saying intro or whatever itgets closed.. i go in feet first try to find my way then if i cant i would go to the help.. for something like the .net framework you're not going to get very far without help, but for something like this, id expect to be able to stumble through it without reading a single line of help. if i compare it to another commercial product which is a 2d engine which also said it was quickstart, had i not being able to get anything working in the first 5 minutes without help it would of been uninstalled, bonus points to them for making it so easy a total newbie could use it :) and their documentation was lacking so its a good job that was the case lol..
Nov 2, 2007 at 6:52 PM
So taking documentation out of the picture, how does having 1 assembly help? They're not going to be any farther along in their 'education.' It's no different than if they were to just include all of the assemblies.

I'm sorry, but I really don't buy this argument. I don't see how having multiple assemblies makes it any harder. If anything, it makes it easier. If I see one QuickStart.dll file, I'll have no idea what is inside of it (assuming for the sake of argument that I, the user, do not know how to use Reflector, or read documentation). Now, if I see QuickStart.Common.dll and QuickStart.Physics.dll, I'll know right away that the Physics file probably has classes to simulate physics.
Nov 2, 2007 at 7:39 PM


shawmishrak wrote:
So taking documentation out of the picture, how does having 1 assembly help? They're not going to be any farther along in their 'education.' It's no different than if they were to just include all of the assemblies.

I'm sorry, but I really don't buy this argument. I don't see how having multiple assemblies makes it any harder. If anything, it makes it easier. If I see one QuickStart.dll file, I'll have no idea what is inside of it (assuming for the sake of argument that I, the user, do not know how to use Reflector, or read documentation). Now, if I see QuickStart.Common.dll and QuickStart.Physics.dll, I'll know right away that the Physics file probably has classes to simulate physics.




I wasnt suggesting 1 dll even though i do believe this is better than 5,6,7,-10 how many will it get to ?

Right ok, fair enough. its obviously not going anywhere, at least it was brought up for future reference one way or another ... not neceserraly a bad discussion
Nov 2, 2007 at 7:44 PM
Edited Nov 2, 2007 at 7:46 PM

conkerjoe wrote:
Cross assembly calling is one of the biggest overheads when i run profiles against our code in my day job.


Are you sure you're not talking about virtual-call overhead? If you refactored the code to not only eliminate assemblies but eliminate virtual calls, then a performance increase would be understandable. Not 90% though.

Just to verify my own knowledge, I ran a quick benchmark on windows and xbox. I created a simple method that just increments and returns a number. One copy of the method was in a class in the same assembly as the executable, and another copy was in another assembly. Each method was executed 1,000,000,000 times in a loop and timed. The difference in loop execution time was less than 1 ms over 2.5 seconds. Most of the time the timings measurements were exactly equal, but on the rare chance that they weren't, the difference fluctuated both ways, so again its just a timer resolution problem, not a significant statistic.

On a slightly more distressing note, while it took approximately 2.5 seconds to perform the 1,000,000,000 iterations for one of the loops on Windows (Core 2 Duo), the Xbox took 46.5 sec. I really wish I could get the PPC assembly dumps from the Xbox CLR, because it is really generating some shitty code. I understand the architectural differences and can understand that the CLR on Xbox is new and immature, but damn!
Nov 2, 2007 at 9:17 PM
WIth regards to the assemblies and namespaces, remember that this is still a prototype code will most likely move and migrate. I would very much imagine that the GUI code will merge into common, as it's difficult to imagine any game without a GUI.

Code shouldn't move to another assembly/namespace just because it's different. The real question is if it makes sense, can the code X live without code Y. Just have a look inside Windows.Forms or WebControl. Thought they are "wast" they make sense, you just know where to get controls or related code. I think that the current GUI assembly should merge with Common just as mentioned above. It still belongs inside a different namespace, as it's not a common thing.

So if you feel like creating a new assembly do that, then later we can have a look at what belongs to where.
Nov 2, 2007 at 9:34 PM

Sturm wrote:

Code shouldn't move to another assembly/namespace just because it's different.


Okay, so then when should it move to another assembly/namespace? Two pieces of code with two different purposes sure seems like a candidate for separation.
Nov 2, 2007 at 9:56 PM
Consider this:
  • Would you have a game without a GUI?
Mostlikely not, so then there is no reason to have seperate assemblies as you always have to include both.

  • Would you create a .net application without a windows interface
Most likely yes, then there should be two assemblies.
Nov 2, 2007 at 10:27 PM
The point of the modularity was to let clients pick and choose what they want to use. There's much more to it than just what parts are going to be used most often. Clients may decide to write their own GUI system, but use our physics library. With your logic, everything we have so far should be one assembly. It's not so much that I'm against a monolithic assembly, I'm just that I'm not sold on your reasoning.
Nov 2, 2007 at 10:55 PM
I'm not arguing for at monolithic assembly either. I do think that Physics isn't something all games are going to be in all games, there are many games which do not use physics. But it will also depends on the implementation of scenegraph as terrain will be part of the graph somewhere. I guess that the graph would be part of common, but if it is how would you add physics without adding the assembly as a reference.
Nov 2, 2007 at 11:02 PM
That's one of the reasons all of the interfaces were put into common. The components would communicate through the interfaces, so only Common would need to be referenced in all of the other assemblies.
Nov 2, 2007 at 11:22 PM
I can see the pattern in that, my only consern is that Common will become a interface hell. Though right now I can't see any other solution, and another might not be needed. This is also why I stated that conkerjoe should just create another assembly and add the code there. We can always restructor the source layout at a later time.
Nov 3, 2007 at 12:10 AM
I don't think every file in each assembly will need an interface in common, only the main subsystems, as long as none of the other assemblies reference that class.
Nov 3, 2007 at 3:10 AM
Correct, the only classes that need interfaces are the classes in assemblies other than Common that need to be accessible in other assemblies.
Nov 8, 2007 at 7:58 PM
As we get deeper and deeper into the new framework, I keep thinking about this discussion more and more. The one question I can't seem to answer is: do we really want to virtual-ize the entire engine? For certain components that are very likely to change from one game to the next, this is a good idea. But for the core systems, like Input, Graphics, Audio, Network, etc., I'm starting to wonder if it wouldn't be better (from an ease-of-understanding level as well as from a performance level) to go with the XNA approach of making these directly-accessing objects.
Nov 8, 2007 at 10:14 PM
I can see where you are going here, though by doing this you are advokating towards less assemblies, as there isn't a reason to have something in a different assembly which most users will use anyway. So if you are saying that most users will not create a custom input control, you are also saying that most users will have to reference both Common and Input in their projects, and then where is the real value?
Nov 8, 2007 at 11:21 PM
I'm not sure I understand your question. The real value of what?

At first, I thought strict modularity was the way to go with the new framework, including separate assemblies with clear demarcations. That's just the way I'm used to doing things, and its a great approach in the native, cross-platform world. Now, I'm just questioning if its worth all of the extra effort. I can spend time writing interfaces for all of the graphics components, but what's the point? Obviously, if we were working with a native C++ engine, the interfaces would be great for cross-platform portability (audio, input, etc.), or even supporting D3D9/D3D10/OGL simultaneously. But with C# and XNA, it's not like we're going to have different renderers, or different audio API bindings, or different input handling. By virtual-izing the core engine components with this engine, it seems like the only thing we really get out of it is excessive virtual-call overhead to components that will never change at run-time.
Nov 9, 2007 at 8:43 AM
Well if you always have to include Common and Input for all projects, then there isn't any real reason to have these as seperated assemblies. There are some elements in the base code which are virtual which wouldn't actually have to be. Using the interfaces as the main entrypoint for extending the engine should mean that there should be any virtualization for the corrosponding engine types.
Nov 9, 2007 at 1:41 PM
So what kind of assembly structure seems reasonable to you?
Nov 9, 2007 at 3:49 PM
I don't know, since you've already have been thinking about it, I would suggest that you made a list with the things that 90% of all using the platform would use without extending just use strait out of the box, this could be Input (mouse/keyboard/gamepad/joystick) and also those who would most likely be extended i.e. AI, and Physics. We would of cause create interfaces for each system in order to provide a extensibility story for all systems, but the main systems would be inside the same assembly.

Thoughts?
Nov 9, 2007 at 4:18 PM
Even with physics and AI, clients will most likely extend instead of replace. I'm not too well-versed on the AI side of things, but with physics you're much more likely to implement new joint types than new constraint solvers.

I guess what I'm really starting to question is the need for interfaces to everything. What are we getting by using interfaces for input, graphics, etc.? It seems almost like writing the components to not really use interfaces but instead be virtual would be a somewhat better idea. Then, instead of replacing components, clients could extend them to add functionality.
Nov 9, 2007 at 6:22 PM
If dev are't going to replace but extend then we need at virtualized engine. If we drop the interfaces we are going to have issues as the Common needs to reference the Input and Input needs to reference Common. The only way to go around this issues is:
  • Create interfaces and define them in common, this is the current approach
  • Merge assemblies, though this moves toward a monolithic assembly (Which under the assumption of extending is more common is ok)
Nov 9, 2007 at 6:57 PM
Yeah, if interfaces aren't used, the classes would definitely be in the same assembly. I'm just trying to enumerate cases here, and I can't really come up with any realistic situations where clients would implement their own custom renderer, input manager, audio manager, etc. Chances are, the existing components will be modified/extended. The question I keep analyzing is: if I'm writing a game using the engine, what am I likely to change/write myself.

Basically, for graphics, the inheritance would go something like:

ISubSystem -> SubSystem -> GraphicsSystem

This simplifies the content pipeline code significantly, and eliminates the need to create interfaces for everything in the graphics assembly. Basically, we lose some abstraction and gain a slight performance increase by reducing virtual overhead (not a lot, just a bit), but what did the original abstraction gain us?
Nov 9, 2007 at 7:40 PM
I can't see the value of ISubSystem here as you would wish to inherit SubSystem directly. I would suggest a name change for SubSystem:
  • either just System
  • Or maybe QuickStartSystem
As I don't think sub is really hitting the point, though GraphicsSystem is a system contained within Game, It's at least as important as Game. So for me it's not a SubSystem.
Nov 9, 2007 at 7:49 PM
Well, whatever the current inheritance structure is... the point was that there probably doesn't need to be an IGraphicsSubSystem in there.

I'd stick with QuickStartSystem, and just System is begging for conflicts with the .NET library.
Nov 9, 2007 at 7:58 PM
I can rework it later tonight, and then submit a patch for it.
Nov 9, 2007 at 8:22 PM
If you agree with this direction, that's fine, but I've just been brain storming here. I have a lot of graphics stuff that I need to merge in too.
Nov 9, 2007 at 8:41 PM
I think merging your changes first is a better solution.
Nov 9, 2007 at 10:49 PM
Give me some time to finish up what I'm working on. I hope to have the initial renderer done this weekend.

How much work have you been doing on the interfaces? I'm just trying to judge how much overall/conflict there will be in our work.
Nov 10, 2007 at 12:54 AM
Most of it is already in. I've done some work on the input system, but that shouldn't affect you. I'll do the conflict resolve once you have checked in your code :)
Nov 10, 2007 at 1:54 AM
Alright, I'll let you know when that's up. It might be awhile because I still have some content pipeline code to write, and I need to clean up some stuff.