Prototype design

Nov 3, 2007 at 7:04 AM
I was thinking about how torn on I am about a long design process. I though I'd voice my thoughts for discussion.

For instance, there are benefits to a long design process:
1.) Less design time needed during implementation.
2.) Less change for unforseen bugs/issues during implementation.
3.) Design is a good time to flesh out ideas that would require huge refactoring time if an engine was already a good portion of the way finished.

There are downsides as well:
1.) People get impatient, and want to see results.
2.) You cannot always foresee certain issues during design, and sometimes you need to have a running demo to test against.
3.) Having a working engine to play with sooner is so much more fun....

My personal opinion on the matter is that design is good, to a point. I believe we should wrap-up design in the next 7-10 days if possible. All major systems should be thought out as much as possible, and all minor issues, like small optimizations should wait until later. Small features should wait as well. For instance, artwork variations would not be important, we should only have enough art to test with. If an artist would like to create assets during this time that is fine however. Basically if we get all major systems laid out in a way where we can understand their main communication between each other, and believe it will work and be efficient enough to use, lets do it. I personally have been so busy and out of the loop on much of the design conversation, however I learn well from experience, and once the basic prototype is finished I will be more willing to sit down and experiment with it and getting components placed into it. We must remember to convert to coding standards as we first go through as well.

If you guys absolutely believe 10 days is not enough then let me know, just bear in mind we can design the small stuff later, as long as we have a good framework that we can expand on then the fine details should be able to come easily later on.
Nov 3, 2007 at 9:47 AM
Edited Nov 3, 2007 at 12:56 PM
In my experience, designing anything in 10 days, unless it's a hello world application, is not do able. But that being said, that's not needed for a project like this.

We do not really need a throughly designed and documented interface. This is a project which will envolve over time, there will be not evolutional progress but a lot of revolutional progress. In larger commercial a revolutional progress is often bad for customers, and therefore everything has to be well designed, that's not an issue for us.

So I do think we can do it in less than 10 days, I can do common, in order to lighten the load off Shaw.
Nov 3, 2007 at 3:42 PM
Then we should probably draw up a list of the critical items that we expect to be "ready" in 10 days.

Sturm, are you still planning on revamping the messaging system along with your work on the common assembly?

Clearly, we need a fully-functional messaging system and underlying backbone. On top of that, to make it worthwhile as a demo, we need a functional renderer, scene manager, and input manager. The input manager is pretty much in a working state, it will just need to be refactored for whatever the new messaging system ends up being.

With the renderer and scene manager, however, we're pretty much at square one. We need to flesh out the scene manager interface and transfer the template's quad-tree code as a quad-tree scene manager. That'll be an interesting task. It would be nice to have the terrain code as well, to actually have something to render, but that'll be quite a task, especially when you factor in all the time that will need to be spent to make the quad-tree a full scene manager and to fit the terrain component into the scene manager. I'm thinking the terrain should be modular and not rely on any particular scene manager. I know conkerjoe had expressed interest in porting over the quad-tree, terrain, and water components, but I think LordIkon should be involved in that at some point. It's his code and he knows it better than anyone else here.

I can start working on the renderer.
Nov 3, 2007 at 5:45 PM
Some things you guys should know about the current Terrain/Quad-Tree relationship. The terrain creates a Quad-Tree root node. That root node uses the information from the terrain to create the entire tree. I'm not sure how we would create a quad-tree first. For instance, the quad-tree bounding boxes are determined by the min and max vertices in each quad-tree section, and off of the highest and lowest heights of the vertices in each section. I believe we could create a quad-tree from the terrain and create a scene manager out of that.

For example we said we will have different scene managers, so if we chose an exterior scene for the manager then the manager could load up the single terrain chunk, split it us using the quad-tree system, and the root node of the quad-tree could be given to the scene manager. At that point the scene manager could be in control of the quad-tree component and from there we could decide what happens.
Nov 3, 2007 at 6:29 PM
Edited Nov 3, 2007 at 6:30 PM
Well, if it's going to be too much work, then I guess we can just make the height-map terrain functionality a part of the quad-tree scene manager for now.

In the future, though, I think it would be a very good idea to separate them and let terrain nodes be a part of any scene manager. Not necessarily large terrain, just little height-map cells inside of a larger world.
Nov 3, 2007 at 10:36 PM
Oh, and I'm not saying 10 days design this thing, I was saying 10 more days, you guys have been working on it and going back and forth for over a week now, I just want to get this ball rolling.
Nov 3, 2007 at 11:36 PM
I can redo the messaging system on common, though if you have already a working version, check it in and I'll take a go from it there.
Nov 4, 2007 at 2:43 AM
I haven't touched the messaging system in awhile.
Nov 4, 2007 at 3:44 AM
If you're redoing common, can you give us an idea of what the new messaging system will look like? Is it going to be essentially the same as the one we discussed in the other thread? Or a no-garbage version? I just want to get an idea of what to expect.
Nov 4, 2007 at 8:54 AM
It's going to look similat, I'm going to create a ObjectPoolManager where bacially everyone can aquire object which are not deleted when released. I'm thinking something like:
KeyboardEvent keyboardEvent = ObjetPoolManager.Aquire<KeyboardEvent>();
If the type on release will be needed or not, I'm not too sure about yet.
Nov 4, 2007 at 6:58 PM
Cool, I can't wait to see it so I can start integrating with it?

Any ETA?
Nov 4, 2007 at 7:02 PM
I would say within next monday. But I expect that I can do a design within the next few days.
Nov 5, 2007 at 6:18 AM
I believe having the Quad-tree component as part of the scene manager may actually be feasible. Here is how I see the relationship of the scene manager, and quad-tree

  • Scene manager has a root quad-tree node.
  • Root quad-tree node has 4 nodes, and of course, every one of those nodes that aren't a leaf have 4 nodes, and on and on, etc.
  • Scene manager loads up the initial terrain as a single chunk, and terrain performs any scaling and smoothing on itself.
    • Terrain is analyzed, and its normals, positions, texture, and vegetation types are all stored in the scene manager.
    • Quad-tree creates itself using the information scored in the scene manager.
    • All terrain info stored in the scene manager is set to null for GC, except the normals and heightdata (which are used for physics).
    • Quad-tree now has all the information needed for the terrain in the scene, and should never need the terrain again.

Basically this is exactly what is going on now, except without a scene manager. The current terrain component is holding all the info a scene manager would hold. The terrain component in v0.179 only holds the vertexbuffer that all quad-tree nodes will need. The quad-tree itself holds the different LOD patches, which hold their own index buffers.

I believe it would be reletively easy to have a scene manager hold the quad-tree rather than the terrain, however I don't believe this will be necessary unless we plan on using the quad-tree for things other than view culling of terrain and vegetation.
Nov 5, 2007 at 6:36 AM
I'm not getting it. If the SceneManager (or SceneGraph??) is a quad tree, how does Models (Dynamic and Static), Lights, Cameras, Sound and Sky fit into the image?
Nov 5, 2007 at 6:51 AM
I am not sure it is a quad-tree, I am guessing that it manages one. The scene manager would have all the things you just mentioned, the things that GameMain in the template currently have.
Nov 5, 2007 at 7:53 AM
Edited Nov 5, 2007 at 7:53 AM
So it would be something like this:
Root - SceneGraph
    |    Terrain - QuadTree/BSP/Whatever
    |    Models - Some kind of list
    |    Statics - Some kind of list
    |    Camera - List of cameras
    |    Lights - List of cameras
    |    Etc ...
Nov 5, 2007 at 7:59 AM
My current setup is to have all entities inside an entity manager. That would have to be inside of the scene manager. Of course that is if we keep an entity manager. Currently my entity manager takes care of deleting any marked entities in between frames (or at the very end of a frame), and takes care of which entities to check for collision, and their order of collisions, etc. Much of this would not be needed after a physics component, as that will be handling all collisions. But the entity manager does keep track of all the entities for rendering, and could work with a culling method for which entities to render.

The scene would then include, as you said, lights, camera, and possibly other info, like wind direction and magnitude, gravity direction and magnitude, weather type and intensity, water volumes (or planes if we must), etc...
Nov 5, 2007 at 8:04 AM
Are you planning on having different nodes for static and dynamic entities are are they treated equally?
Nov 5, 2007 at 8:35 AM
As far as being treated equally by nodes, what do you mean? As far as processing time, static entities update calls would be empty or darn close to empty, while dynamic entities will have quite a bit more, including things derived from dynamic entities, like npcs/bots which will have A.I. to process.
Nov 5, 2007 at 11:09 AM
I could imagine having a SceneGraph class which looked something like this:
Root - SceneGraph
    |    Terrain - ITerrain : IDrawableEntity
    |    Entities - SortedList<IDrawableEntity>
    |    Statics - SortedList<IDrawableEntity>
    |    Cameras - List<IEntiry>
    |    Lights - SortedList<IDrawableEntity>
    |    Interface - List<IDrawableEntity>
    |    Sounds - List<IEntity>

There are more elements to the list than this. but just an initial thought.
Nov 5, 2007 at 1:32 PM
So then each leaf would contain it's own lists of indices into these root lists?

Nov 5, 2007 at 1:58 PM
I am not sure that the quad-tree is a good candidate for culling of these items, unless it is for scenery (trees, buildings) or static/earthbound objects. The quad-tree for instance couldn't contain info about an airplane in the sky, as it would be outside the bounding box. And expanding the bounding boxes very far into the sky would cause terrain to be culled out of view less often, which would lower framerate. Even if each leaf contained info about entities in its area, things are bound to move during runtime, we would have to check collision between all entities and all boxes every frame.
Nov 5, 2007 at 2:00 PM
So what's going to be used for rendering the entities?
Nov 5, 2007 at 2:59 PM

shawmishrak Wrote:
So then each leaf would contain it's own lists of indices into these root lists?

I'm not sure I got that, but I'll try to explain.
  • The terrain node contains the terrain which inherits from ITerrain and comtains Update/Draw, what should be done in update and draw is determined by the actual ITerrain implementation, which could be a QuadTree implementation
  • Entities move around a lot and need an efficeint cull method, which handle fast updating items
  • Statics and prob be sorted in some way which would make it very fast to cull, as these never move
  • Camera/Sounds do not need to render but might need some kind of culling anyway

shawmishrak Wrote:
So what's going to be used for rendering the entities?

Either each IDrawableEntity can render itself or a RenderManager should be used. Prob a manager is a better idea as rendering can be optimized.
Nov 5, 2007 at 3:28 PM
What I'm getting at is that we need to cull the renderable entities in the scene manager, given the current view's camera. The renderer will take care of proper sorting of the draw order, but it won't do any CPU-based culling. That's the job of the scene manager.
Nov 5, 2007 at 3:44 PM
I could imagine that we would be able to do something like this:
In Game.Update()
And then in render:
In Game.Render
How to cull is determined in the lists containing the entities/terrain, So we would create a QuadTreeList or a BPSList.
Nov 5, 2007 at 4:11 PM
That looks fine, we'll just have to be very careful of the order of things. The scene graph Render() method must be called before the control is passed along to the renderer, as the scene graph Render() method must build up the render queue for processing by the renderer. Now, from the renderer point of view, I have one more major question: how can we cache this? If the view isn't changing and there are no new entities coming into the view range, it's unnecessary to have to rebuild the render queue each frame. Even for moving objects, it shouldn't be necessary to have to re-traverse the whole scene graph. I don't think its too much of a stretch to give the renderer direct access to the entity instances to extract position, orientation, and potentially animation data. This could save a lot of traversal time, and the scene graph would only be responsible for adding and removing from the render queue for objects coming into and out of the render view range, not updating each frame. This would require more logic in the camera <-> scene manager interaction, as instead of building a list of visible objects, it would build a list of newly visible objects and a list of no longer visible objects and adjust the render queue accordingly. The render could even maintain the sorted queue across frames, so the only per-frame sorting penalty will come from inserting and removing entities from the queues.
Nov 5, 2007 at 6:16 PM
This sounds much like a job for events, a OnPropertyChanged would help a lot as the Graph wouldn't need to traverse all children to find all changed properties.
Nov 5, 2007 at 7:36 PM
Yeah, I would limit the scope of it though. Have an event that's triggered for any state change that could affect the visibility of the object, not just any change. Otherwise, you'll end up with a hell of a lot of event calls (read: slow) for things that can't change visibility, like animation frame advancement, world triggers, player health changes, etc.
Nov 5, 2007 at 9:21 PM
Or instead of an event a delegate could be registered which would be invoked if the entity moves in or out of view. Something like this:
    public delegate void StateChanged(IEntity entity, int state);
    public interface IEntity
    public class Entity : IEntity
        public StateChanged ViewStateChanged;
        public void Update(GameTime gameTime)
            if (visible changed or moved in/out of camera frustrum)
                if (this.ViewStateChanged != null)
                    this.ViewStateChanged(this, EntityState.Visible);
The main graph would then have
    public static class EntityState
        public const int Visible = 0;
        public const int InCameraFrustum = 1;
        public const int Position = 2;
    public class SceneGraph // Or might be SortedList/QuadTreeList
        public void Add(Entity entity)
            entity.ViewStateChanged = this.UpdateEntityList;
        public void UpdateEntityList(IEntity entity, int state)
            switch (state)
                case EntityState.InCameraFrustum:
                case EntityState.Visible:
            ... // Rendering queue update code here
Or something simiar.
Nov 16, 2007 at 11:16 PM
Ok, seriously, I think we need to get something out there soon that will allow us to integrate in current system. Having a version that allows us to integrate new stuff, even if it isn't finalized, will allow everyone to see how the prototype engine works with the old template, and from there I think design could go much quicker.

There is only so long a small team can really design something over the web, and I think some people may be getting burned out.
Nov 17, 2007 at 12:33 AM
Yeah, I know what you mean. I've been working on the rendering base code, but I honestly just don't have the time that I would like to put into this. I haven't touched any of the code since last weekend. Hopefully I can get the new code in a state ready for upload this weekend.

I think Sturm's pretty much taken over the architecture of the engine. I just wish I had more time to actually get some work done instead of just having a few minutes here and there to troll this board.

I honestly think our biggest problem is that we're all approaching this from different viewpoints. I'm a performance guy. I have no qualms about sacrificing a bit of code readability in favor of pedal-to-the-medal performance. In contrast, I feel that Sturm is approaching all this from a "all I really care about is 100% conformance to whatever standards .NET suggests," and I get the feeling you're approaching it from a "whatever gets the job done, I want results" angle. I'm not saying any of these approaches are in any way inferior to another, and I could be completely wrong about my assessment here. I just firmly think that we're not all on the same page here.
Nov 17, 2007 at 2:04 AM
Edited Nov 17, 2007 at 2:12 AM
That sounds about right to me. I, the same as you, don't have a whole lot of time, so I certainly don't have time to check the prototype and make sure its design is great. I've been spending my little bit of time trying to get performance of the template good so that when I integrate it into the prototype I don't ruin it :). Three weeks ago on a 1024x1024 map with all components running I was getting 10fps on my 7900gs nvidia, now I'm getting around 150-180fps. I'm getting around 50-80 on my Radeon 9800se. So I'm pretty happy with the state of performance. I'm sure once it is integrated you'll be on top of performance changes as well.

If Sturm truly is all about design of the engine that is fine, but as far as standards go, we need something that XNA users are used to, comfortable with, and will allow for high performance. And seriously, we really need to get this thing going, I'm waiting to start a lot of things until the prototype is integrate, like animation, A.I., custom models (not just spheres), etc....
Nov 17, 2007 at 3:57 PM
Im also waiting for the prototype to be finalized before I start serious work on the engine.

I would usualy agree on shaw about performance/readability, but if this engine is designed for users to learn from, the code needs to be as readable as possible.
Nov 17, 2007 at 4:16 PM
I also don't have a lot of spare time, but I still feel strongly about having a good model. As Aphid said if we are going to release this to others, then having a consistent and easily understandable model is the key for success.

I'm not all standard or nothing, and I know that performance comes at a cost, it's just my experience that when it start to cost a lot, there's a design flaw. Shaw is prob refering to PIMPL and the fact that I advocate having all fields as private. The main issue here is that that costs when physics has to be applied, because it will happen often and if we use properties we incur the penalty of copying to mem a lot. But there are possible solutions for this which have very limited overhead and don't break PIMPL i.e.
public struct PhysicsData
    public Vector3 Position;
    public Matrix Rotation;
public class Entity
    public virtual void Update(ref PhysicsData data)
        this.position.X = data.Position.X;
I can find other patterns we can use in order to solve this if needed.
Nov 17, 2007 at 9:29 PM
I think another, perhaps larger problem here is that we need to have a definite goal. Its just not feasible to think that we can write something that's good at every type of game. If you generalize too much, you end up with an interface roughly on par with XNA, or just a library of components. That's what I see when I look at the current Template code. Its not really an engine so much as a library of pseudo-independent components. That's nothing wrong with that, I just wouldn't consider it an "engine."

I honestly feel that we need to take a step back here and actually decide what we're doing. We've all been operating on our own definitions of "engine" and what we want to accomplish in this project, and I think we're starting to hurt now because none of us really know what's going on. Also, like I've mentioned a few times before, if the end goal is a "game engine," then we need a game concept first and foremost. At this stage, it can be relatively simple and only roughly designed, but we need an idea of the direction we're going with this. Just blindly creating a "game engine" without a type of game in mind is an exercise in futility. You're bound to miss key components or perhaps miss the boat entirely if just implement components without designing for specific goals, like for the scene renderer: "I need to render an N x N meter landscape, with pseudo-infinite water, lush vegetation, day/night cycle". Otherwise, you'll end up being either too general and not really accomplish anything, or too specific and not be able to actually write any kind of game without heavy modification.

To summarize, some of the important unresolved issues I see are:
  • We need to decide if we're creating a game engine or a library of components. This has a huge impact on interface design. Specifically, what I mean here is: do we (a) create independent components that are usable in any XNA game through the GameComponent framework, or (b) create an engine that handles a lot of things for the programmer at the expense of not having the individual components usable outside of our framework.
  • We need to decide on a reasonable end goal, i.e. a game concept.
  • We need to do a better job of explicit task management. Right now, everyone's doing everything and making a big mess of things. No offense intended.
  • There's been a lot of discussion on class design methodology, and we need to come to a final decision.
  • We need to come to a final decision on assembly distribution. The prototype was designed to be split among assemblies, but there has been considerable resistance to that idea.

That's all I can think of off the top of my head, but looking at the list I strongly feel we need to resolve all of these issues before another line of code is written. LordIkon, I know you're going to hate me for saying this, but I think we need to take a few steps backwards before we can take a step forward.
Nov 17, 2007 at 10:04 PM
I feel that you Shaw and LordIkon should sit down and do the road map for the project, since you two are the coordinators of the project. Once you've completed that post it on the main page, and then the devs can decide if and how we can contribute to the project.
Nov 17, 2007 at 11:43 PM
I don't see why everyone can't participate in this.

Though, as LordIkon is really the leader of the project, I really want to get his input on this.
Nov 18, 2007 at 4:47 AM
I agree everyone should have a say in it, but someone really needs to get something laid out for all to see, then we could iterate on it. Otherwise we'll all both throwing in all of our ideas at the same time and getting nowhere fast.

I believe this should be a game engine, complete with gui editor, and as much functionality as we can get to allow for many games. Now we aren't going to be able to make an engine for EVERY type of game, just a reasonable amount. I believe most 3d game types should be supported. We should start at a very high level design, which means we need to make a list of things we all want, and decide which are reasonable. Once we have a finalized list of feature that we can agree on, then we can design something that will accomodate all of those features, and still run on a 360, and still run well on as many pc's as possible.
Nov 18, 2007 at 5:41 PM
Yes, we should keep it general to support many types of games, but right now we need to design towards a specific game concept. This game concept can be small and simple, but it needs to include all the functionality we wish to put in the engine, and it'll act like a "tech demo" of sorts. That's really the only way to make sure we're considering everything necessary to create a full-fledged game engine. Otherwise, we could end up with a "theoretical" engine that looks good on paper and in simple demos, but doesn't actually work in a full game. We don't necessarily need a brilliant back story, we just need to parallel-develop a game like we envision being created with the engine.

On a high level, I see it breaking down like this:

  • Environment:
    • Each level is a large terrain, i.e. Far Cry. We could also attempt split one, vast terrain into chunks and stream at play-time like in Oblivion or many of the MMORPGs.
    • Real-time day/night cycle.
    • World bounded by infinite water or non-passable mountains. This is a gameplay hack, I know, but it serves our purpose.
    • Mesh-based terrain with adaptive LOD. In the editor, we can import heightmaps to create the world chunks, but at run-time everything will be geometry.
    • Fully-interactable. As much of the world is dynamic as possible. This will of course start off simple and get more advanced over time.

  • Entity System:
    • All objects in the world (characters, weapons, triggers, etc.) are part of an entity-system.
    • Entities can be serialized to/from disk for saved games and data-driven creation of entities.
    • C# scripting back-end for entity logic (AI characters, trigger events, weapon effects, etc.)
      • Engine can provide "AI Library" that can be used by script writers for creating character behaviors.

  • Player:
    • Controlled by user-defined input events (or network events for remote players).
    • AI characters are aware of player's presence.
    • Possible client/server multiplayer.

  • Audio:
    • Dynamic and ambient sound tied into environment, i.e. walking through grass or gravel should produce different sounds. Night time ambient should be different from day time ambient.
    • Controlled by script back-end.

  • Graphics:
    • Capable of rendering the environment at full-detail on Xbox 360 and high-end consumer cards, and at reduced detail on low-end consumer cards.
    • Dynamically adjusts to hardware.
    • Optimized for outdoor rendering.
    • Minimal: basic per-pixel lighting, shadow mapping from "sun."
    • Advanced: High dynamic-range rendering with post-process tone-mapping, water effects.

  • Editor:
    • Several Components:
      • World editor: allows users to generate/tune terrain chunks either manually or by "painting" terrain with height maps. Place entities in world.
      • Entity editor: allows users to generate/tune entities based on parameters. Bind C# entity class to entity definition. C# script can define associated parameters adjustable in editor (i.e. character health, weapon damage, trigger activation probability, etc.) similar to HLSL SAS parameters.
      • Material editor: allows users to generate/tune renderer materials that are bound to entity models/world geometry.

  • Other issues:
    • Possibility of indoor rendering. Requires separate module in editor, and a different environment/renderer codepath, along with a different way of handling the geometry.

Feel free to contribute/elaborate on this. At this stage, let's not concern ourselves with the low-level class design. Let's get the concepts finalized first.
Nov 18, 2007 at 6:15 PM
I like the sound of all of this. It sounds to me like we should start with an "outdoor" engine, and then once we're happy with it we can being adding interiors. As long as the design is flexible enough that we can add in an interior component or something similar when that time comes. Don't get me wrong, I'm excited to work on interior stuff, its just that the outdoor stuff is much further along and I believe it will be an easier transistion at this point. And if we work on both simultaneously it will increase the current design time, and with out limited resources it will increase the time until we have something that really works.
Nov 18, 2007 at 6:25 PM
Is there anything you would conceptually add/subtract? We need to all agree on the initial concepts up-front before we can get a good grasp on the technical design.
Nov 18, 2007 at 8:09 PM
Stuff I can think off might be too low-level for this discussion. Like realistic water that darkens depending on depth, with settings for wave length, height, reflectivity, fogginess, using volumetric fog. Water could be dynamic and react to physics collisions with it. I could write an entire paper just on the water component of the engine.

I think the entity system should have animation integrated with it that is as easy as possible. Such as a system to transition between animation states. And speaking of states, we will likely need a state machine setup for entities. State machine is invaluable with A.I., which we will also need. I'm looking forward to working on A.I., but we really need physics, entities, and animation before it will be realistic enough to really have fun with.

That is all I can think off of the top of my head.
Nov 19, 2007 at 5:59 AM
Those things aren't necessarily too low level. Ideas like that are good. Keep the ideas/concepts coming!

I'll start hammering out of a high-level interface design if you're okay with these concepts.

On a side note, anyone else starting to get disillusioned by the whole XNA Game Studio thing, especially reading the XNA board threads over the last week or two? I was really hoping they'd get some sort of Xbox distribution plan at least announced for 2.0, and now you need a Gold account AND a creator's club membership on all endpoints for LIVE functionality on Windows, not just Xbox. What's 2.0 really giving users, besides VS 2005 integration? The only reason I started to look into XNA when it came out was Xbox support. Now, all I keep hearing is "it'll be announced when it's done" for the Xbox distribution issue, and no one seems willing to discuss the Xbox CLR, the one part of XNA I actually truly care about. All I ever hear when an Xbox CLR issue/question/request comes up on the XNA boards is, "Well, that's a different team, and they don't interact on the community sites." At the very least, I would at least like some feedback on issues/requests. Honestly, it's hard to sit down and do real work anymore when I know real technical data on the Xbox CLR is very difficult if not impossible to find, not to mention any work on the Xbox right now is purely an academic exercise.
Nov 19, 2007 at 8:03 AM
Before you start off doing a high level interface design, I think the major issue here was that we didn't have a plan in the first place. What I'm missing here is the overall road map. There should be a plan on what we are going to implement for then specific versions of the engine. This is also invaluable for those using the engine as they would know when certain features would go in.

You can put as much detail in it you want, bacially the closest releases have the most detail, and later versions less detail. This will give a good image of how much work is left in order to release a version.

Of cause a versions shouldn't have something like this:
  • Prototype with all the current template code
But rather something more like this:
  • Event mechanism
    • Central eventing mechanism for distributing messages to all systems
  • Sub system model
    • Graphics system for rendering
    • Input system

And the more detail you put in it the more the scope of a release will become appearent.

Gold - That goes without saying, this is the only way to connect to other players on XBox
Creators Club - Not surprisingly, but a little annoying. But the reason for this is clear as there has to be a way of differentiating Xna from regular Gold games.
Nov 19, 2007 at 8:50 AM
The only benefits I'm thinking 2.0 will give us is just a few performance optimizations, and a few goodies like occlusion culling. I was looking forward to the networking but knowing the requirements I'm not too excited about it. On the windows end I would rather use sockets, but it wouldn't be compatible with the 360. I'm starting to think however that even if it worked for the 360 we'd have more Windows users anyhow, and I believe there are less Windows users looking to spend $150, especially considering some of them don't have a 360.

I know Shaw that your big thing is 360, it is just sad how limited it is on the XNA end right now. Windows is pretty much unlimited access anyway we can make it, which makes this engine more feasible for so many things, like actual games. Unless there is a way we could abstract the networking well enough to use sockets on windows and live on the 360, although that is a pretty big abstaction. If the networking system is designed well we could write a version for Windows and another for the 360 in which we use Live networking. Or better yet simply a Sockets version for Windows non-Live customers, and a Live networked version for Live customers.

Big pain in the ass is what it is.

Sturm, I think we all agree on the current template not just fitting right into the new prototype. The way I figured it would happen is we all agree on a design, and from the design, we can decide how the template 'features' will fit into the new engine. So we can get to a stage in the prototype where it looks similar to the current template, but may run very different (internally). For instance, I've removed input handing from the input component and given it to the objects that need it. This is a step in the right direction I believe, the next step would be to have those objects handle messages from the input system rather than have the input system directly call the HandleInput() function within those objects. I'm trying to be flexible as possible with the template, but of course much will need to be changed to get it into the prototype. I figure I'll start simple when I start moving stuff over, like the camera system, this way I can clean up anything I need along the way. Also, if we start with the small things then we can all move components at the same time. The template is basically something to look at while we're rewritting it into the prototype I feel.
Nov 19, 2007 at 9:33 AM
What I'm trying to say it that if you laied down a road map for the project, I think everything would become much more clear for everyone. It would also show where you envision the project, and shows that we are indeed planning on working on this for a long time. This will make others more interested in the project and hopefully spawn a development community at some point.

With regards to the wrapping networking, we would have to do that anyway as the networking API is way to low level to be used directly for a game engine. Also there has to be a mechanism for providing custom data provider/consumer as different games will have different requirements. This has to be abstracted away anyway, so not really any big overhead here.

Where did you get the $150 for windows users, the platform is still free for window developers, it's only when you want to get it to the 360 platform, or Live, it actually cost money ;)

Nov 19, 2007 at 1:44 PM
Edited Nov 19, 2007 at 1:45 PM
That makes sense about the roadmap. Though you need some sort of high-level design in place so you know what needs to be done and can write release A such that implementing release B is natural and doesn't require a complete refactoring of the code.

Where did you get the $150 for windows users, the platform is still free for window developers, it's only when you want to get it to the 360 platform, or Live, it actually cost money ;)

What this means is that Live is completely inaccessible to non-developers. You can't write a game based on Live then release it. At this point, using Live will make your game "pretty," but otherwise unusable. If you didn't pay attention to the information at Game Fest, the evolution of Live in XNA went something like this:
  • At Game Fest, Live networking was announced for Windows and Xbox. Xbox of course still had the Creator's Club fee, but nothing was ever said about any Windows fee. It was implied that the Windows Live functionality would be free and redistributable.
  • A week or so later, it was said in the XNA boards that the Live functionality would not actually be a part of the XNA redistributable. So, anyone wishing to distribute Windows XNA games with Live functionality must require the end user to download/install the full XNA Game Studio package.
  • A week ago, it was further said that Live on Windows is not free, and that a Gold membership with Creator's Club membership is required. This is actually even worse than the Xbox restriction, as on Xbox you can do Silver + Creator's Club.

That's just one of the issues I'm having with XNA at the moment. I just honestly feel it's taken a turn for the worse lately.
Nov 19, 2007 at 3:34 PM
Also, who's going to work on the high-level design issues other than myself? The next few weeks are going to be hell for me (2 weeks of class, 1 week of finals), so if you're relying on me to flesh out this design/roadmap, then you might be waiting awhile :)

Nov 19, 2007 at 4:59 PM
I had gathered from the gamefest announcements/creators forum that:

*XNA networking uses the Xbox/Windows Live service, and as such you need a Gold account to play online.
*You need a Creators Club subscription to do anything with the Xbox360.
*You can play local network games for free (with CC to deploy on the 360 ofc)

What I'm not sure about is currently, windows live only works on vista, so does the XNA networking require vista too?
And where did you hear that you need CC to use networking on windows? That just doesn't make any sense..
Nov 19, 2007 at 6:30 PM
Edited Nov 19, 2007 at 6:34 PM

Aphid wrote:
What I'm not sure about is currently, windows live only works on vista, so does the XNA networking require vista too?
And where did you hear that you need CC to use networking on windows? That just doesn't make any sense..

Windows Live works on XP just fine. If XNA started to become Vista only, that would completely turn me off to it.

As for the CC requirement, see the XNA blog post about it

Now what's really interesting is that you need Live GOLD on Windows in addition to a Creator's Club membership. I've been using a Silver account on Xbox (with a creator's club membership) just fine. If this changes, I'm gonna be upset!
Nov 19, 2007 at 7:24 PM
I'm not entirely clear on the versioning scheme LordIkon has been using on the template code, but I would shoot for 0.20 for the first public release of the "new" engine framework. Basically, most of the template code must be integrated to make a public release worthwhile. Of course there will always be versions in source control, but the first move to a Release should not happen until the engine is on par with the current template code.

To hit that goal, I'm envisioning a two-stage internal release schedule. Namely, critical system design/implementation for 0.19 (strictly internal and in source control, no release), then template porting (within reason) for 0.20. I see this further breaking down as follows:

  • 0.19
    • QuickStartGame base class implementation.
      • Full encapsulation of XNA Game class, may need to wait for 2.0 beta before we design this as the Game class is probably going to change.
      • Encapsulate event system. (see next item)
    • Fully-functional event system.
      • Handle all inter-engine messages.
    • Basic input control.
      • Query for specific keys, mouse axes, gamepad thumbstick axes.
      • Optional ability to query on logical input "events," which are assigned to one or more physical input events, i.e. query if "FireWeapon" input event was triggered, which is mapped to left mouse button and/or Ctrl key.
    • State/Screen System
      • Games exist in a certain state at all times.
      • Loading screens will be thrown in here somewhere.
      • There's been discussion about whether to keep separate state and scene management classes, or just have a "master" class that handles what's visible and let the clients explicitly "emulate" state based on what screens are visible (main menu, game world, etc.). The majority vote here seems to be to ignore state and let the client just control visible screens.
    • Basic menu-ing (is that a word?) i.e. main menu from data/script.
      • Can be a very simple GUI that responds to input commands for now, but eventually we will need a good GUI.
      • Game will create a "main menu" class that exists as a state and creates/draws a GUI, and responds to GUI-generated events like "button A clicked" or "button B begin hover."
    • Basic renderer
      • Simplistic at first, will evolve as needed with the terrain components.
      • For now, just processes simple render queue commands and implement the XML based material system.

  • 0.20
    • With the framework laid down in 0.19, the template can now be integrated. Namely, we integrate the terrain classes, cameras, etc. into the framework.

Obviously we need much more detail and design here, but this is how I see it playing out. Comments?
Nov 19, 2007 at 7:40 PM
Something else that may help is a clearer understanding of roles. I joined this project to help develop the physics code and parts of the graphics code. That's still my primary goal, and I don't see myself as having the time or experience to actually lead the overall design of the project. What I'm trying to say is that I have no problem helping to design the project at this stage, but I also don't want it to be my responsibility to flesh out the entirety of the design. My specialties are graphics, physics, and general high-performance computing. Someone else with more experience in other fields will probably do a better job of designing the GUI system and audio system, to name two.
Nov 19, 2007 at 8:58 PM
I think that we should get the overall picture in place first, before we start designing. I think setting out a roadmap is first priority as this will show where we should focus our efforts. As you said yourself we do not have much time. This only makes it even more important to focus on those areas which are important for the current release, and then focus on those.
Nov 20, 2007 at 1:43 AM
I'm not disagreeing, I'm just saying that I can't be the only one working on it.
Nov 20, 2007 at 4:44 AM
For me to flesh out a design (just using this as an example) of an engine, I need to know which components require what. Shaw, you will know best what the physics portion would need for instance. I believe we should design our respective components first, and list which things each will require from other components. For example, if I am designing the terrain component I will let you guys know that I will need physics, light, camera, and rendering interactions. I could then of course go into more detail. By listing the things I need from physics, shaw could determine what he would need to expose or do on his end to have the physics component compatible with the terrain. This would occur for all components. I could pick a few, and you guys could do the same. One we have a layout of the things each needs and what we believe the solution between them is, we can then concentrate on the central parts of the engine, like messaging system.

Just an idea to spreadout the workload.
Nov 20, 2007 at 6:42 AM
I'm looking for something like what Shaw did. Just remember that we are very few devs with limited time so each release shouldn't have more than 4 or 5 features. Also there prob should be a expected release cycle, lets say one new release every 2 month? So that we will release 0.19 around january and then every second month from that point on we will release with more features.
Nov 20, 2007 at 6:47 AM
I'm not talking about design here. It goes without saying that you shouldn't design everything, that would take too much time. What you should do as a project coordinator is to define the goal and scope of the project, and individual releases. You can do that without having to design anything. Then once you've finalized that you could create an issue for each area and set the appropriate release.

This will also help those dl the engine to see what's in there, what will be in for the next release and they could vote on which features should go into the next release.
Nov 20, 2007 at 6:15 PM
For now, I'm much more concerned with graphics than physics. I'm not even going to consider anything beyond basic collision detection until after 0.20, there are just too many more important things and there are not enough people to delegate everything else.

I've already outlined what I think is important for the first internal release, 0.19. I was hoping for some comments on the outline (additions, removals, etc.), but in their absence I'm just going to assume that its a sufficient list. We need to now split up the tasks. On a high-level, we can subdivide the tasks into core engine back-end and core systems.

  • Core Engine Back-End
    • QuickStartGame base class
    • Event/message system
    • Engine interfaces (i.e. how systems talk to the engine back-end, not everything can be messages)
    • Entity system

  • Core Systems
    • Renderer
    • Input
    • World Manager
    • Screen Manager

At the least, these components are what we must implement for the first release, what I dubbed 0.19. I know it's a lot for one release, but it doesn't make much sense to release something that can't do anything. For this release, it's not necessary to fully implement the terrain code, scene culling, or such; we just need to lay a proper groundwork for their implementation in 0.20. This gives us a good understanding of the goal for 0.19, with a general idea of what to do for 0.20. Once that work is done, we'll have a much better understanding of what we can realistically do given a release time frame.

Unless there are any objections to the feature list for 0.19, I suggest we start individually going through some high-level design for the components. While splitting up the component design between people with experience with that component is a good idea, we still need to coordinate our designs. As a group, we need to identify exactly what each component will be responsible for, and how their dependencies and shared responsibilities cooperate. It'll be up to individual implementers on how to do the implementation, we just need to flesh out the interfaces and, using software engineering terminology, create the contracts for the components.
Nov 20, 2007 at 7:25 PM
It's all fine, so if you and LordIkon take all this information and from that define the features we should have for the different releases. Just keep in mind not to have more features than there are developers. Also make sure to plan 2 or 3 releases, as this will allow us to pull in features in case a feature is finished early.

Nov 20, 2007 at 8:14 PM
You mean even split this initial work into different releases, like a couple things for 0.181, a couple more for 0.182, and so on until 0.19? At this stage, I'm not sure how we can really separate like that without considering at least the interfaces for all of this as a whole. For instance, if we say we're only going to focus on the QuickStartGame base class and the messaging system for the first release, then we could easily shoot ourselves in the foot if we don't consider the interfacing to the other systems and how the messaging system will be used by the systems.

Once the groundwork is laid, I definitely agree that small, fast design/implementation iterations are the way to go. But at this stage, I see a lot of problems if we concentrate solely on individual components without considering the entirety of the "core engine."
Nov 20, 2007 at 9:27 PM
A start would be to determine what is the absolute minimum bar for releasing, i.e. QuickStartGame/Messaging/Render/etc, then split it into possible features, if they contain multiple features, Render could contain more than one feature where as Messaging wouldn't.

Having that group of features you know the work needed to complete the first minimum engine. The divide by the number of devs (here assuming every dev is suited for every task). That number (Rounded up) gives you the number of features which each dev should implement.

Now that you know the number of features and the timeframe (lets say 14th of januar) you have to ask yourself can you do that number of features before that date? If yes fine, if no then you either have to cut on features or have a later release (14th of feb??).

I you think that you won't have time because of xxx or yyy but the rest of us might, we still aren't going to hit the date and need to mitigate. Besides I don't think that anyone will have more or less time. We all have a real life to take care of and can only assing sparetime to this project. I would expect each dev in general will have the time to implement one feature per month (size equal to messaging not a full terrain/physics). and as such should be able to finish one feature before 14th of januar. If time permits we can always pull in features from the next release.
Nov 21, 2007 at 12:57 AM
The list above is pretty much what we need before any public release. Like I said, it's not necessary for the renderer/scene manager to be fully-functional, just started and capable of doing something. The parts that need to be mostly complete are the core engine components, like the Game encapsulation and messaging system.
Nov 21, 2007 at 1:02 AM
Make a list of the features (make it public somewhere on the site) and then create the issues related to the features, then we can start assign people to them :)
Nov 21, 2007 at 1:15 AM
That's fine, as long as its understood that we're still in the brainstorming stage, and an "issue" assigned to someone doesn't mean they need to go and independently implement it.
Nov 21, 2007 at 8:52 AM
If we are going to plan for a release for january then we should get into design/implementation phase soon. And yes I don't expect that just because you've been assigned to a feature that you'll design/implement it alone, but you'll have primary responsibility for it. If you feel someone is better suited for a specific feature agree to sign it over, either for the current release or the next and then find a feature to work on.
Nov 21, 2007 at 3:26 PM
I believe we should define what we expect to see (as a user) in the first release. For example, we can say that the program should start up and at the very least display a 3d model and have a camera responding to input.
Nov 21, 2007 at 3:41 PM
Yes, and this aligns very well with Shawns list. So who is going to create the roadmap, define the features and create/assing the issues?
Nov 21, 2007 at 3:45 PM
At the very least, yes.

So let's look at what it'll take to accomplish that:

  • QuickStartGame implementation to drive the entire engine.
  • Event system to handle messaging between input system, renderer, world manager, and the core engine.
  • Input system capable of querying the keyboard, mouse, and gamepads.
  • Renderer capable of rendering a custom geometry format with materials.
  • World manager that defines world geometry (in this case, "the model"), lighting, and camera settings for the renderer.
  • A Screen system that will define how the world rendering (if even present at certain times) is composed with other screen elements, like GUIs, menus, etc.
  • Some sort of game logic handler (i.e. scripting system) that translates user events into world events (i.e. StrafeLeft key translate the player along the local -X axis.), as well as handling game-specific world logic.
  • A primitive client-server loopback interface. We do not want to code everything then "add" networking later on. If we are going to support networking, it has to happen from step 0.

Anything else I'm missing?
Nov 21, 2007 at 8:00 PM
Networking is a big item, whoever takes that on should probably do ONLY that, and no other component. This is especially true if we're going to support sockets and Live networking.
Nov 21, 2007 at 8:08 PM
It's on par with graphics, or physics, or scene management. Doing any of these things efficiently is not trivial.

The biggest problem with networking is going to be testing the Live code. System link is doable in a local network if you have two or more computers, or a computer and an Xbox with a Creator's Club membership, but testing between developers over the actual Live network requires all testers to have Gold + Creator's Club (close to $200 a year), and an Xbox just to get a Creator's Club account even if the testing is all Windows-based.
Nov 21, 2007 at 8:10 PM
Taking your list I see the following list of features:

  • QuickStartGame implementation
    • This has the highest priority as nothing else will be able be implemented
  • Event system
  • Input system
    • A Feature for keyboard
    • A Feature for mouse
    • A Feature for gamepads.
  • Renderer capable of rendering a custom geometry format with materials.
    • This is multiple features which I think initially should just load standard models, as it takes a lot of time to define/write custom importers
  • World manager that defines world geometry (in this case, "the model"), lighting, and camera settings for the renderer.
    • A Feature for Camera
    • A Feature for Lighting
    • A Feature for creating SceneManager
  • A Screen system that will define how the world rendering (if even present at certain times) is composed with other screen elements, like GUIs, menus, etc.
    • Initially simply a Screen which can be opened/closed and display some text. Creating GUI is a big feature and should be punted.
  • Some sort of game logic handler (i.e. scripting system) that translates user events into world events (i.e. StrafeLeft key translate the player along the local -X axis.), as well as handling game-specific world logic.
    • PUNT: I would love to see a scripting mechanism but not for the initial release(s)
  • A primitive client-server loopback interface. We do not want to code everything then "add" networking later on. If we are going to support networking, it has to happen from step 0.
    • This would tie nicely into the messaging system so def a must have. But the initial version is a "dummy" single player client-server setup.

So this game 11 features (might not be all) and since we are 10 people on the project this should be doable for january. Anyone else got comments?
Nov 21, 2007 at 8:22 PM
Do we actually have 10 people though? Besides us three, I've seen Aphid and conkerjoe post on the boards, but I haven't heard anything from the other developers on the list. Are they MIA, or just don't post on the boards? To those people, please check-in here by making a post with your interests/specialties so we know you're still interested in contributing!

With the custom models, I'm not talking about writing custom importers for other third-party modeling packages, I'm talking about a custom geometry container that is populated by a custom processor. We can use the existing importers. I need the custom model processor anyway for the material binding and generation of tangent/binormal data. I practically already have this written in the prototype code, I need to convert it to 2.0 beta though.

Also, the Screen and GUI systems are not the same thing. I realize the GUI system is a project for another day. The Screen system is just a way of compositing the images for the game window and allowing the game clients to maintain some sort of state. You have a "main menu screen", an "options menu screen", a "multiplayer matchmaking screen", a "gameplay screen", etc. This is the SceneManager/ScreenManager debate we had awhile ago. Game logic is explicitly tied to a certain screen, i.e. you can bind a GUI menu to the main menu screen, and bind the scene manager to the game play screen, and even composite them at run-time if you want to by drawing the game play screen under the main menu screen if say the player hits Escape while playing the game.

Scripting is another of those things that has to be a part of the initial designs. It doesn't have to be very functional, we just need a separation between engine and game logic. All I'm really imagining for this is client-defined classes that control game logic.
Nov 21, 2007 at 11:11 PM
I don't know if that's the correct list, I would expect that either you or LordIkon would keep track of it, and update it as needed. If you are saying that it's just the five of us, then there is no way we are going to release by january. So we should target Feburary.

Looking through your comments, I've updated the list a bit and now I'm getting:

  • QuickStartGame implementation
    • This has the highest priority as nothing else will be able be implemented
  • Event system
    • Messaging system
    • Object pool system
  • Input system
    • A Feature for keyboard
    • A Feature for mouse
    • A Feature for gamepads.
  • Renderer capable of rendering a custom geometry format with materials.
    • Render standard model using standard processor
    • Custom model processor
  • World manager that defines world geometry (in this case, "the model"), lighting, and camera settings for the renderer.
    • A Feature for Camera
    • A Feature for Lighting
    • A Feature for creating SceneManager
  • A Screen system that will define how the world rendering (if even present at certain times) is composed with other screen elements, like GUIs, menus, etc.
    • Screen which can be opened/closed and display some text.
    • Screen which can overlay the main game
  • Some sort of game logic handler (i.e. scripting system) that translates user events into world events (i.e. StrafeLeft key translate the player along the local -X axis.), as well as handling game-specific world logic.
    • PUNT: I would love to see a scripting mechanism but not for the initial release(s), still believe this. For the first version(s) a dev would need to inherit from our QuickStartGame, and Entities in order to implement game specific code.
  • A primitive client-server loopback interface. We do not want to code everything then "add" networking later on. If we are going to support networking, it has to happen from step 0.
    • This would tie nicely into the messaging system so def a must have. But the initial version is a "dummy" single player client-server setup.

Nov 21, 2007 at 11:37 PM
That should about cover it.

I thought LordIkon was keeping track of the developer list. I just know there are several people on the CodePlex list that I've never heard from. Whether or not they have been in contact with LordIkon, I do not know.

The QuickStartGame and event system should be top priority right now. Of course, we need to consider all of the systems while designing these two, especially networking, but that should be our focus. Then we can add in the Screen system and Input system to actually display something, then add in the renderer, world manager, and game logic.
Nov 22, 2007 at 12:29 AM
So who is going to finalize this and publish it under the release section? Also who is going to create the Issues releated to these features? My initial thought would be LordIkon as he should know the people available.

But now that we got the list for this release, what should be the target features for the next release?
Obviously Scripting has been punted, but I still don't think that it should be implemented even for the next release more likely 2.1

Here's a overview of what I could imagine:

Release 2.0:
  • GUI - More Gui controls and ability to build some kind of menu
    • GUI Control Base
    • Basic GUI model
    • Basic GUI Control hirarchy
    • Basic GUI Interaction model
  • Multiplayer support - Improve the server to actually pass data to different players, still on the same box
    • Split screen cooperative - Everything is still send within the same box, no need for DirectPlay/Live/etc
  • Render - More advanced optimizations, create managers to track/manager resources
    • Shader Manager
    • Texture Manager
    • Model Manager

Release 2.1
  • Scripting System
    • Basic scripting on predefined events
  • In game editor
  • Advanced GUI Controls
    • Combobox
    • Tree View
    • Grid
    • Dialogs (Load/Save)
  • Multiplayer Support
    • Using Sockets
    • Using Live
    • Max 2 player

Release 2.2
  • Scripting
    • Create custom events
  • Multiplayer Support
    • Max 16 players (Can we do more??)

That's what I can think off, comments?
Nov 22, 2007 at 1:06 AM
That's roughly what I had in mind as well. Though, I would change the version numbers. We don't want to hit 1.0 until we have a polished product. Also, the shader/texture/model managers should be a part of the renderer from the first design iteration. It can just be a simple List<> at first, but we need to have the interface in place for them.

With the networking component, we need to seriously consider if we're going to support multiple networking implementations. It would be nice to have, but at the same time we need to consider the feasibility and the amount of the time it would take to abstract all of the networking components. My concern isn't so much the actual communication as it is the support of GamerTags, player profiles, and other Live features. Not to mention having to develop all of the code for discovering clients, matchmaking, reliable transport over UDP, etc. I would consider long and hard if we really need to go beyond Live. That only thing that sucks about it now if the Creator's Club fee, but it's not like you're going to see much non-developer usage of XNA until the distribution problems are worked out.

That reminds me, I need to check how hard it is to set up a simple client-server loop in the same game using system link.
Nov 22, 2007 at 7:49 PM
So are we planning on supporting more than one local player?
Nov 22, 2007 at 8:23 PM
Yes, one feature I think every XBox game must have is split screen. I even know a few who will not buy a game if it does not support split screen.
Nov 22, 2007 at 9:04 PM
Edited Nov 22, 2007 at 9:07 PM
The new 2.0 release adds splitscreen support as part of the networking. It apparently makes no distinction between communicating between a local player and a network player. At least thats what it says in the help files.
Nov 23, 2007 at 3:15 AM
I don't see any reason not to support multiple local players.
Nov 23, 2007 at 5:10 AM
It can make it difficult for rendering complex scenes, if we setup a decent LOD system for models as well as terrain, we can probably lower the detail in 4 player mode for example.
Nov 23, 2007 at 5:20 AM
Something else I've been giving some thought to lately is the QuickStartGame <-> Render System interaction. One of the "features" introduced in XNA 2.0 is that the GraphicsDevice instance is always available as a property in the Game class. Anytime you register an IGraphicsDeviceService instance as a service, as the GraphicsDeviceManager class does, that property is internally set. We have to register an IGraphicsDeviceService service with Game, or else none of the Live Guide drawing or graphics content pipeline readers will work. This in turn causes the GraphicsDevice used by the engine to always be available to clients through Game, no matter how hard we try to abstract the render system from the game client code. Also, since this is all implemented privately in the Game class, I see no way of overriding this behavior.

Ultimately, this leads me to the conclusion that we need to accept the fact that the GraphicsDevice will at all times be visible to client code. Consequently, I'm debating if it makes sense to implement the renderer functionality as a library of operations on a GraphicsDevice that is ultimately controlled by QuickStartGame/client code instead of a separate system that manages a GraphicsDevice instances and ultimately controls how a frame is rendered.

Nov 23, 2007 at 5:39 AM
Can we not make our Game class private, and require that the client's game have their own game derived from our private Game class? If so we could create an accessor so that they never have direct access to Game's GraphicsDevice. Just a thought.
Nov 23, 2007 at 6:14 AM
Well, no. The client's game class will derive from QuickStartGame, and that forces QuickStartGame to be public. Also, outer-classes must be public or internal. In order to be private or protected, a class must be an inner-class (defined inside of another class).

The only way I see to sidestep this issue is to make QuickStartGame non-inheritable, then make clients implement a custom interface that connects it to our QuickStartGame. In other words, instead of inheriting from QuickStartGame, clients would work with some custom interface other than Game.

Unless there's some C# trick here to hide the GraphicsDevice property... Sturm?

Nov 23, 2007 at 6:29 AM
I also tried hiding the property with 'new.' For example, in QuickStartGame:

   public new GraphicsDevice GraphicsDevice
            throw new Exception("Client should not access GraphicsDevice.");

Then, in the client Game class, accessing this.GraphicsDevice or base.GraphicsDevice will invoke this property and cause the exception. However, semi-intelligent client code can easily get around this with some casting:

    Game game = this as Game;
    GraphicsDevice dev = game.GraphicsDevice;

where dev is now the real GraphicsDevice instance.
Nov 23, 2007 at 6:46 AM
Also, what are your thoughts on sticking solely with Live instead of abstracting the whole networking component (including the Live functionality) and implementing two network implementations. I just don't think we have the manpower to re-implement the functionality given to us by Live in a way that we can easily swap between the two implementations, nor would it be worth the effort.
Nov 23, 2007 at 12:08 PM
Well in its current state, Live on xna is practically system link only as few people have a CC subscription, even amongst developers.
Going for both Live and our own networking would be a huge task, especially if we try to implement all the extra features of Live. We could just use Live and hope microsoft fix it soon.

About the graphics device access, would it be possible for QuickStartGame to instead contain a private (or internal) instance of Game, and hook into its update/draw/load events (I assume there are some).
Nov 23, 2007 at 4:15 PM
Like it or not, Live the only way for networking the xbox, so If the engine is too support both, we must surely use live?
Nov 23, 2007 at 6:36 PM
I'm sure we are going to use live, as it's the only thing available on Xbox. The question here is if we should create our own "limited" networking and then simply map it to Live/Sockets.

I think this would be the most sound idea, but it does increase scope.
Nov 23, 2007 at 6:37 PM
On Xbox, yes, Live is the only option for networked multiplayer. We definitely need to support Live if we're doing networking. The question is whether or not to support a custom sockets-based network component as well. The XNA team keeps promising to release information about the Xbox/Live distribution plans that will enable sharing with non-Creator's Club people, though I'm slowly losing faith in that.

Aphid, I don't see any events for hooking into Update/Draw. It would have to work in the reverse order: we subclass Game, and internally hold a reference to a custom-defined GameClient (or some other name) base class, which clients will derive from and treat as their version of Game.
Nov 23, 2007 at 6:54 PM

Sturm wrote:
I'm sure we are going to use live, as it's the only thing available on Xbox. The question here is if we should create our own "limited" networking and then simply map it to Live/Sockets.

I think this would be the most sound idea, but it does increase scope.

The problem I see with supporting both is that they're fundamentally different in how they enable matchmaking. With Live, you either setup system link (one client creates a lobby, others search the local network for it) or connect to a centralized matchmaking server that manages game sessions. There is no "connect to IP x.y.z.w" functionality. With our own network solution, it would be the reverse, you can connect to a specific IP, but there's no centralized matchmaking. Creating a centralized matchmaking server would just be too hard for our limited resources, not to mention the hardware we'd need for a dedicated centralized server.

If we build a common interface using the lowest common denominator between these two systems, we lose the following functionality given to us by Live:

  • Centralized matchmaking
  • Voice chat (as far as I know, there is no other way to extract voice stream data from the Xbox 360 controller in XNA)
  • Roaming gamer profiles

We would also be forced to implement the following ourselves:

  • Reliable packet transmission over UDP (TCP is too much overhead in general for faster-paced games)
  • Packet encryption
  • Game discovery on the local net

I just feel we stand to lose more by abstracting Live than we gain from creating a custom sockets solution.
Nov 25, 2007 at 1:39 PM

Sturm wrote:
Yes, one feature I think every XBox game must have is split screen. I even know a few who will not buy a game if it does not support split screen.

There is a problem with this and Windows compatability. Although like I said 2.0 adds splitscreen support, this is only relevant to the Xbox, as Games for WIndows - Live only supports one signed in player at a time. This means no co-op on one PC. Take a look here for the post:

Should split screen be implemented at loss of compatability, seeing as it will only work on an Xbox?
Nov 25, 2007 at 4:38 PM
Well, 2.0 doesn't give you inherent support for split-screen, it only enables you to treat the local players as you would network players so you can logically separate the two players in the game code. However you decide to implement local multiplayer (split-screen, all players on screen like a sports game, etc.) is up to the developers and XNA doesn't provide any mechanisms for that. All of the input handling, multi-viewport rendering, etc. you have to do yourself.

That said, the real wrench in the works here is the inconsistency between Windows and Xbox. We need to decide if local multiplayer is important enough on Windows that we need to support two distinct networking types: Live for system-link and Live matchmaking for a single local player in a multiplayer game, and some sort of local loopback for multiple local players in a multiplayer game. You might be able to still you Live and logically create multiple players per end point on Windows, but that would be a big undertaking and would have a huge impact on our packet structure design.

Nov 25, 2007 at 5:45 PM
By 2.0 supporting splitscreen, I only meant the whole viewport thing. I agree though that 2 forms of networking would be a major undertaking. I personally think we should only use live at this stage, as the extra work needed for multiplayer on windows would not be compatible with the Xbox. I dont think its justified.

Alos, Shawn Hargreaves of the xna team has given his view on multiplayer live here Like he says hes not aware of it, but if the devs think its a good feature, it may be added at some point, invalidating our existing code. Thats my view at least
Nov 25, 2007 at 6:00 PM
If we just code to Live, nothing we do should be invalidated if the multiple sign-ins are implemented in Windows. Our code just won't allow more than one player per machine on Windows, and support up to four per machine on Xbox, all with the same code. All we're concerned about is how many local gamer profiles are signed-in. How they sign in (Xbox Guide / Windows Guide) is not really of concern to us.

By 2.0 supporting splitscreen, I only meant the whole viewport thing.

I'm still not understanding you. XNA 2.0 has no built-in support for split-screen rendering, as far as I know. It's up to the developer to implement a multi-viewport renderer that can render the scene with one camera in the top half of the screen, and with another camera in the bottom half of the screen.
Nov 25, 2007 at 9:16 PM
I agree with you on the Live thing. If we used a condition if !XBOX, we can reuse the same code, just limiting the windows sign ins.

As for the viewports, Im probably exagerating. The how to article is here ms-help://MS.VSExpressCC.v80/MS.VSIPCC.v80/MS.XNAGS.1033/XNA/Use_Viewport_For_Splitscreen_Gaming.htm (open with xna documentation). All I mean is the multi viewport built in. As far as im aware this simplifies things compared to 1.0.
Nov 25, 2007 at 9:26 PM
I don't think we'd even need conditionals. Just code it assuming multiple sign-ons, and the Windows version will automatically just be limited to one as there's no way to sign on more than one.

I see what you mean by the viewports. I thought you were talking about something specific to XNA in 2.0; viewports are just a standard feature of Direct3D and have been in XNA since 1.0 beta.
Nov 26, 2007 at 5:10 PM
Edited Nov 26, 2007 at 5:13 PM
My bad, I thought it was new!

On the subject of the actual network engine, I was thinking something along these lines:

We use Live to set up the session etc (not important at the minute, im more concerned with what data we send and how)
Data is sent keeping only entities synced
COntinues until sessione ends

Ive thought of two approaches to keeping objects synced, having the program that created the object handle all physics for it, and then send it new posistion to the other games. This is low CPU load, but high traffic. Alternatively all objects are simukated idependantly, with no data sent, except the computers 'check in' wit each other every minute or so. The exception to this is critical entites such as gravity emitting ones, and ones bound to a player are always synced. This is high cpu load, but a few large updates leading to low network usage. Which is most appropriate, or is there a better way ive missed?

Also I was thinking that to tell the network system an object has changed, the physics engine throughs a ObjectChanged event with some for of ID and change info, and the enginge than decides to tell the others if appropriate (based on the methods above)

Whats does everyone think? Would this suffice as a start to networking?
Nov 26, 2007 at 5:37 PM
I've done a little networking before, and this is how I see it:

(Bear in mind I'm not factoring LIVE into any of this, so there might be an easier way).

Physics should be done by both the client and the server, and every so often (5-10 frames), if the client is too far off of the server's physics model for something, the client updates to match the server.

If you "trust" the client's physics at all times it makes it easier for somebody to hack their version to do different things. The server should act as the authority.

Most games will allow the user to run their game as the server as well. In this case I believe they end up running a server and client both.

I wrote a paper on it that had some good links as references, however I'm not home right now, I'll post it if I find it.
Nov 26, 2007 at 6:56 PM
You definitely want to keep the server as the centralized authority. .NET applications in general are just too easy to "modify" with Reflection.

Ideally, we need to keep the network traffic as minimal as possible while still maintaining synchronization and security. There are lots of things to keep in mind here and design for, including when to use reliable packet transmission, how much data we should send per packet, what data should be sent per packet and per frame. Obviously you want to keep an update tick going several times a second, but you also have to factor in instantaneous events (weapon fire, triggers, etc.) that can't wait until the next network update cycle.

We need to break these discussions into separate threads if we ever hope to keep these discussions meaningful and understandable.

mikelid109, start a new thread here outlining your thoughts, in detail, on the networking component. Specifically, we need to know what the network component will need resource-wise, and what it provides to the engine back-end. We also need people to do the same for the other components. When I get some time, I'm going to start a discussion for the graphics component. That way, we can keep these discussions more or less separate and on-track.

By the way, this is my last week of classes and then I have finals next week, so don't expect much participation/response from me for the next week. :)
Nov 26, 2007 at 7:22 PM
Ok will do. I hadnt really considered network topologies yet, more what things need to be updated and when. I think your right though when it comes to security. Ill design it so that at some stage we could add some basic encryption if wanted. Ill also break this into a new thread, but probably not today. Ill get my ideas straight first!

Oh, good luck on your finals shawmishrak!
Nov 26, 2007 at 7:28 PM

As far as I know, all Live traffic is already encrypted by the API.
Nov 26, 2007 at 7:41 PM
Yes, good luck on those finals. How many terms do you have left?
Nov 26, 2007 at 8:57 PM
I figure it's probably best not to start counting, especially since it's so variable. I finished my undergrad work last Spring, and this is my first quarter of grad school. So, at the very least, 4-5 years or so, most likely 6.
Nov 26, 2007 at 9:08 PM
Good luck anyway!

Can someone give me a quikc runthrough fo the Message system? I think I get the jist, but I just want to make sure. How does it get sent to the right sub-system, and what Types and values are currently in use? The Design Doc just refers to a EngineEventType, which I cant find

Nov 27, 2007 at 7:08 AM
This is for the prototype, if you were asking regarding the Template someone else might be able to answer this.

There isn't any way to send a message to a specific sub system (Yes there is an optimization to do here). All messages are sent to all systems as it's just an event.

The event is raised during update, which actually sends the update message as the first message. This might actually not be the right approach as now all messages come after the update, but this is the current implementation.

If you need to send a message to a subsystem you have to do a few things (I've reworked most of this locally and made it a easier, but I'm waiting on Shaw to check in some code as he had some interface changes as well).
  • Add an identifier to the MessageType class (You don't have to do this, and can create your own class for this as it's just an integer that is passed)
  • Make sure that EngineMessage contain have the needed information otherwise add the information to the class
  • Create an instance of EngineMessage and set the needed parameters
  • Invoke QuickStartGame.SendMessage

In the sub system you then have to:
  • Hook up to the QuickStartEngine.EngineEvent event
  • Usualy in the event handle method you would verify that your system is active, before processing
  • Switch on the eventParam.Type to filter on your message id
  • Get the information from eventParam (remember to validate as it could pass a lot of data)

Some of the changes I've done:
  • Events are now string based (externally, internally it still uses int) and registered on the QuickStartGame (Makes debugging a dream)
    • There is an virtual on the QuickStartGame called InitializeEventMessages where you are supposed to register your custom event types using RegisterEventType
  • EngineMessage is now templatized which allows custom data to be passed.
  • Messages can now be send to a specific sub system

There are some other changes, that I can't remember right now (I'm not at home so can't check). Also I'm considering making an interface for EngineMessage and pass that around. Since you have to typecast no matter what, it's not that great an overhead, and does make the implementation a lot eaiser.

I hope this clear things up a bit. If you have more questions please feel free to create a new topic, as this thread is getting out of hand ;)