Network Re-Design?

Jan 26, 2008 at 5:19 PM
Firstly, sorry for my silence. Ive been busy with coursework for a while. Ive been around, ive just had nothign useful to say. :)

I want some opinions on the network please. Put simply, its not working. THe amount of waste being sent by sending every message, it just pointless. SO I want to propose an alternative. As such, can we bump the network to 0.20, or do you still want what ive got? The main bit should stay the same for the user, with my changes being engine related.

I would like to have a plyer class added to the engine. All input could then be sent to this, and it can be queried by the Scenemanager for rendering. The benefit would be that the network can deal with only that, and update it as necessary, meaing the amount of messages are reduced. Is this something everyone would be happy with, or is there a better solution?

Side note, Adribeiro can you post here please? Ive had nothing off you, and you mentioned you had sent me the code youve been working on. I want to check your getting my messages as Ive heard nothing.

Sorry if this is wasting your time, its just I cant see how I can handle it all so genrically. I need a concrete way to interface with the other systems.
Jan 26, 2008 at 6:41 PM

mikelid109 wrote:
Firstly, sorry for my silence. Ive been busy with coursework for a while. Ive been around, ive just had nothign useful to say. :)


It's ok not having time as real life tends to take over, but please do reply on post requesting a status on the feature or update the issue with the current status.


mikelid109 wrote:
I want some opinions on the network please. Put simply, its not working. THe amount of waste being sent by sending every message, it just pointless. SO I want to propose an alternative. As such, can we bump the network to 0.20, or do you still want what ive got? The main bit should stay the same for the user, with my changes being engine related.


The feature is already being punted to 0.20. If you could explain in more detail what is going wrong would help us provide some more help.


mikelid109 wrote:
I would like to have a plyer class added to the engine. All input could then be sent to this, and it can be queried by the Scenemanager for rendering. The benefit would be that the network can deal with only that, and update it as necessary, meaing the amount of messages are reduced. Is this something everyone would be happy with, or is there a better solution?


We can't add a single class for player, as there might not be only one player, or only one player object. Also there should be others who need to react to player input, controls or other entities, we can't know as this is determined by the individual games. I can't understand why you can't filter out those messages you do not want to send, alternatively you could add a RegiserMessage on the network class which you register the messages who should be send through the network service.


mikelid109 wrote:
Sorry if this is wasting your time, its just I cant see how I can handle it all so genrically. I need a concrete way to interface with the other systems.


It must be generic as we will not know what the game developers will require. They can create anything they want and shouldn't be limited by the engine.
Jan 28, 2008 at 1:22 PM
Edited Jan 28, 2008 at 1:23 PM
Hi,

When you will code freeze to version 0.19 and 0.20?
Jan 28, 2008 at 2:43 PM
The only missing feature for 0.19 is the MousePad integration which should be finished this week. The release should then be ready by monday 4th of February
Jan 31, 2008 at 5:32 AM
Mike can you give an status on your progress?
Jan 31, 2008 at 1:04 PM
Sure thing. I have coem up with a possible solution to the problem, whihc I will reoutline. The problem is the netowrk system is too generic. The netowkr cannot differentiate messages, and has trouble on the recieving end. It is alos inefficient, as just serializing the data would lead to extra padding.

Solution: Stay Generic and hands off, as asked :)
The user creates a cutom link between the engine and the netork system, using methods provided in a base class. This interpets engine messages as well as game specific ones and converts it to a format it decides is appropriate for the network, and then decodes it at the other end. This means the network can concentrate on only sending raw byte messages efficiently and securily, not worrying about how it interfaces. Bonus is if the core engine changes, only the interface needs to be changed to reflect this.

Im partly pushing this area to someone else as I dont have intimate knowledge of where we are going with the engine to develop the interface. I can handle the network communications, but the interface should be worked on by everyone. Who better to write the physics interpreator than the guy who first made the system, for example? If this isnt okay with everyone, I can give it a go, but cnat promise as to how good it will be.

What does everyone think? Its only a brief outline as Im away from my dev machine at the moment. Be brutal :)

Jan 31, 2008 at 2:49 PM
While I can see that this can be highly efficient, I'm affraid that it also imposes a lot of complexity. But I would need to see some exmaples before I can determine anything.
Jan 31, 2008 at 6:07 PM
I honestly cant see an alternative. I could just send the engine messages if that is what we want, but I cant guarentee how well it would work without actually coding it up. How about a compromise of both? I create the interface, but it merely passes the messages on in byte format. If this proves inefficient/inelegant/unworkable then we can re-code the interface. WOudl this be a suitable compromise do you think?
Jan 31, 2008 at 7:27 PM
Prototyping a few ways of doing this will probably help. Just write a few functional prototypes and see which is more natural and efficient.
Feb 1, 2008 at 7:21 PM
Well from the description alone I can't really help in any way, I would need a more specific scenario, if you want help from me.
Feb 1, 2008 at 9:12 PM
No problem, no one can be expected to work with thin air. :)

Il work on the interface, but tied into the messaging at the moment. Then when its uploaded, people can modify it to fit with their own systems, if they decided they need the improved efficiency/control over transmission. I leave methods in as place holders for the current systems. Is this ok with everyone?
Feb 1, 2008 at 9:44 PM
Without any pattern/issue/code description it's not possible to come with any suggestions. So I would say create a patch and we can review this.
Feb 9, 2008 at 12:30 PM
The scene managers arent final yet, are they? It seems like if we passed the responsibilty to the scene manager for notifying the netowkr of a change in object, and the network could notify the scene manager, it should work. Any other systems could still just send messages for any data they require to send. Is this a better solution than an interface compatible with all systems?
Feb 9, 2008 at 2:55 PM
I think you need to be more specific if you want help, currently nothing is set up for the networking so everything is possible.
Feb 9, 2008 at 3:22 PM
OK, Ill make a suggestion :)

I assume the scene manager will contain a list of objects, that will have been added by the user. Some form of identifying which ones should be synchronised with remote systems would be in place, be it a collection or flag on the object. Note only certain objects would need to be synced as most would change as a reaction to others. eg ai moves away from player. Only player is synced as all will simulate the same reaction. Every update cycle the scene manager informs the network of those requiring an update, either by passing in the objects new set of properties, or by a byte stream (whatever we determine is most appropriate). The network then transmits this. When data is received, it is stored in a buffer. AT the start of each cycle the scene manager could query this buffer, which is then cleared for the next set of data.

Any other systems needing to send messages would use the normal message system, which is monitored. The network system sends it, and the remote system receives it and changes the message range to ensure it is not sent again.

What do people think? Its a rough outline, but I feel it would work. This means the network can be unconcerned as to input, physics and even to an extent AI.
Feb 9, 2008 at 8:16 PM
I don't see the scene manager being the one being able to determine and send messages for entities. simply because entities might have child entities which would have to send network messages, while the parent entity wouldn't. Also IA entities would need to sync across the network as they can easily get out of sync.

We should make it so that the network system would just take the entity id which should be synced and then let the network do the sync of the entity, this would make it much easier for developers to create game entities.

You also have to consider that it's not only entity states which are sent over the network, there are messages which needs to be send as well, i.e. server shutdown.

How is it dertermined if a message should be send over the network?
Feb 10, 2008 at 10:55 AM
Well the current system just has an event handler that's copies any message to a buffer if it falls into a certain message range. Otherwise the message is not buffered, and only the buffer is transmitted. I'm at a loss for suggestions then. I pretty much agree with what you're saying Sturm, could you post an outline of how you see it working please? I disagree with the AI though. If all the players are synced, and so is the random seed, surely it should generate an identical result on all machines? I would still do a full sync every couple of seconds, but it would save on bandwidth to do it less.
Feb 10, 2008 at 12:02 PM
Hi,

I recently got in touch with the engine (.19) for creating a game. After browsing the topics I noticed this topic (i've been thinking about a "good" network idea for some months now - it's hard to make it perfect .. )

Anyways, I might be out of line on this topic, since I didn't participate in this discussion from the beginning.

Here are my 2 cents:

A network game always consists of two elements: a server (master) and client(s) (slaves).
The master should, in my opinion, always be the one who takes decisions, eg: AI, death decisions, hit decisions and so on.

This means that the clients are normally receiving more data then sending it. They shouldn't be burdened with sending data which is not ment for them to sent.
wrong situation the client notices a hit on an enemy (message) and considers it dead, thus sending this signal to the server
good situation the client shoots a bullet towards an enemy, sending the "fire bullet" message to the server, which then calculates the hit and maybe a dead.

Considering the above, this means that each client sends data about the entities he's involved with, while the server sends additional data, like AI positioning, death messages, etc.

I don't have a clue how this should be implemented in the current ideas, like the scene manager. It might be a good idea to brand each entity with a "player id" so the scene manager knows if they need to sent it across (?).

Furthermore, torque used the following principle: a single player game always consists of a server and client (local queries). This might be a good choice, since you make the engine network-ready, even when not using networking abilities.

I hope to get this point clear, if not, i'll be happy to explain them in more detail.
Feb 10, 2008 at 12:05 PM
Even when you have the same seed and the same initial setting, the law of chaos dictates that nothing changes identically. You might have different systems running at different speeds, which means that the faster system gets a more pricise physics calculation than a slower, over time this difference might calculate to the two entities moves in different paths. You need much more frequent updates than a few seck, because the sync could force the entities to be moved quite a few pixels so much more likely to update once a frame.

How do you define which messages will be sent, can the users simply hook their own messages into the system?

It's difficult to dao any meaningfull outline as it's dependent on the actual implementation, various solutions would require different implementations. So it's prob easier if you would ask the questions.
Feb 10, 2008 at 12:21 PM
Hi MOever

We are going to use a similar client server architecture, so basically every game is a client server game, even single player games.

For your comments above I would say that in a perfect world you are absolutely right, but unfortunately we do not live in a perfect world. This means that we need to compromize and have some kind of preminitive guess on what's most likely going to happen, so in your above scenario we would do:
  • Client shoots a bullet (it might be towards a friendly too :evil:)
  • Client sends a message to the server
  • Client registres that the the shot hit a player (AI/Human) and plays the hit animation
  • Server gets the message and calculates the result
  • Server sends message to all relevalt clients
  • Client gets message from server and can now do
    • Play the death animation for the character
    • Complete the hit animation as the player was hit but didn't die, maybe update the player model with blood slattering
    • Cancel the hit animation as the player wasn't hit at all

So there is more to networking than just sending the required messages, sometimes the system needs to make a best guess as to what most likely is going to happen and introduce some intermidiate states which can make the transitions less obvious.
Feb 10, 2008 at 12:36 PM
Hi MOever.

I already have the lcoal code sorted to some extent, meaning the same server can be re-used remotely or locally without any major rewrite if desired. Also, this has the benefit of not using a 127.0.0.1 loopback, cutting down on lag in single player game. I mainly agree with you, but feel your approach is to server heavy. I personally favour a hybrid. The server has the 'correct' version of events, and can overrule clients (security considerations?). The clients and servers run independantly, but the server sends a periodic update, that the cleints must accept. I understand what youre saying Sturm, would an update every second be suitable in your view (if we have 30 update cycles a second)? This should keep in all in line, yet cut down on network usage.

The network message system is very much tied to the engines. If its possible to send a message via the engine, its possible to send it via the network also.

This may work out fine If whenever an object is changed an engine message is sent with the details. The recieving computer will then recieve this message also, and process it. If whatever controls the entites can support this, I think it would work. All other message like kill/gun fire etc would be handled in custom client and custom server code.
Feb 10, 2008 at 1:16 PM
Thanks for the explanations, I had no idea you guys were already this far :)

I totally agree with Sturm and his example, a thing i forgot to mention myself was the network prediction, which is covered in his example a bit, thus making the server load less intense.

The example doesn't cover non-player events, like AI. Have there been some ideas about this? ( as stated above, the server could be the one responsible for such decisions).

Is there a possibility to join this project as a developer, I feel there's alot to learn for me here :)
Feb 10, 2008 at 2:37 PM
AI is still a bit vague at the moment, but I believe Lord Ikon has plans for it, but not necessarily on the network side. I have no authority as to if you can join, but more the merrier I say. Get in touch with Lord Ikon, Sturm or Shawmishrak, they re the coordinators.
Feb 10, 2008 at 3:00 PM
MOever I would suggest that you have a look at our issue list and pick one where you would like to help and contact the person assigned, or if you don't find anything where you feel you can contribute then have a look at our planned release for 0.20 and look if htere is anything there. We are goingo to implement some GUI and I could imagine that there were plenty of tasks to get started on. If there is somethin else then make a post amd lets discuss it.
Feb 10, 2008 at 3:36 PM
Once the network basics are done, would you be interested in prediction? I would like someone else to do this, allowing me to work with efficiency and optimization of the core network (and possibly encryption and other features)
Feb 13, 2008 at 12:54 PM
Sure thing!

Is there any way to get me involved in your current plans? albeit just for the looks. ( I browsed the CSV, but there's nothing there - yet).
Feb 21, 2008 at 6:40 PM
Is it fine with everyone if I post a patch of the generic network system, minus the work in progress sockets? I want to get it finalised and added to the project. Then I can not worry about having to re-add it when the next checkout appears :)
Feb 21, 2008 at 8:28 PM
You are always welcome to post patches to the engine and we'll review it. This also helps everyone understand the issue you have with the code. It's be far easier to understand it everyone can see it. Just remember to follow the guidelines for patches defined here Uploading Patches