I read through the whole series of posts in one sitting - impressively thorough!
As someone who spends a lot of time writing and structuring regular old REST APIs, it's a lot of fun to see an approach to API design that is rooted in the foreign convention of game design, even if the implementation language is in the ballpark of ones I use.
Interestingly, once you get past the authentication, the reliable UDP implementation, and into the "meat" of the behavior, it's the same sort of call-response stuff that anybody else is doing, with the same sort of fairly basic little payloads. The "special sauce" is all the components getting to that point - and the fact you're getting binary data and not JSON or something. Not the most exciting thing, if you're a game developer.
Sometimes I wonder what somebody who paid as thorough attention to systems I've worked on would say about them. What would the weird stumbling blocks be? What would the "magic" be that seems to make no sense? ("What are the security holes they find?" goes without saying, but that's feedback I get from our internal pentesters anyway.)
> As our encryption is asymmetric, you may be asking how the client decrypts messages send by the server if it only has a public key. To do this the game exploits an interesting quirk of RSA cryptography, its possible to encrypt with the private key and decrypt with the public key. This is exactly what happens when the server sends the client message. For anyone contemplating doing this though, don’t, it’s terrible for a number of reasons. You can read some of these reasons here, written by someone far more knowledgeable about cryptography than myself.
Great effort reverse engineering the protocol and producing an enjoyable write up.
> its possible to encrypt with the private key and decrypt with the public key
Isn't it just signing/verifying? I.e. "encryption" with a private key is how RSA is being used to sign, and "decryption" with a public key - used to verify the signature.
This could well be it, though if the plaintext isn’t sent (unclear from the article if it is, I think not?) then all the client can really do is decrypt with public key and check that the output appears well formed, which may be good enough.
It would seem odd, however, that client -> server is encrypted but unsigned whereas server -> client is signed but unencrypted. My guess, for what it’s worth, is that somebody at the company insisted that all comes must be encrypted and so they cobbled this together along with the obfuscation of the public key the author describes.
That quote doesn’t seem to appear in the article, or at least not anymore?
I was curious to read the reasons. The ones that come to mine are the obvious ones that EC is significantly faster and more secure and that you should only use asymmetric to negotiate a symmetric key exchange (eg ECDH) since symmetric is going to be even faster and suitable for real-time streaming.
I've been reverse engineering another online game and encountered very similar themes in terms of encryption. Lots of weird obfuscation, pseudosecurity, and obvious vulnerabilities. Instead of choosing something battle-tested like TLS they roll their own crypto and fail miserably.
One fun project I did a while ago was reverse engineering one of the games from my teens, Runescape 2 (now called Old School Runscape, where the game is a backup from August 2007 with many updates since then).
There is a huge interest in the game, with many players still playing it, I had fun for a couple of months for a few hours a week.. but then noticed there was a field in a client you use (nobody uses the official client, they use Runelite, which is an amazing opensource client for OSRS).
I reverse engineered the login protocol, it was a lot of fun. Erlang pattern matching on binaries with Elixir made this incredibly easily and very fun to implement.
Depends what you want to do. But good debugger is absolutely essential.
Reversing communication? MITM the communication.
Want to understand the game's logic / find entrypoints? Memory scanner/editor -> breakpoint to a address in debugger.
Combine above with disassembler with good code analysis tool and you'll be able to follow the code even without actually having the source.
In the end reversing is basically poking around and experimenting, but it is very rewarding and will teach you a lot how software work and how easy it is to actually make them bend to your will. Be wary, you'll need lots of free time for this.
Quote: "However all multiplayer traffic in FromSoftware’s games is actually sent entirely peer-to-peer."
This is very important to understand why currently DS3 PC server is closed for half a year now due to cheaters. FromSoftware does a childish attempt at creating a server hence the plague of cheaters. And what they do? They released Elden Ring instead of working to properly create a server. FromSoftware knows how to create console games, but a PC game is outside of their expertise.
What exactly is your point here? Console games need (mostly the same kind of) netcode as well. They are just generally harder to exploit, which dramatically lowers the amount of people trying to do so.
They are capable of writing a netcode that is good enough for an environment where it is relatively safe due to the restrictions of the devices it runs on but not for any true open environment. - So they just write bad netcode, that's all.
My point is that if you want to create a proper server you do the heavy lifting on server, not on client. The second you let the client decide who does what (location / damage / item drop etc etc) you get cheaters that will modify their client to bid their rule. Hence why I consider FromSoftware servers childish.
> Hence why I consider FromSoftware servers childish.
Servers in FromSoft games are only for matchmaking. The actual gameplay has always been peer to peer. The design of the games mean that making the multiplayer entirely online server based would not work. They're peer to peer because that's the best way to support the hybrid singleplayer / multiplayer design where players drop into and out of your world. Otherwise your world state would need to be consistently synced to the server, which introduces major problems -- in particular, the fact that you would be dealing with lag in a game that you spend mostly singleplayer.
They would have to rework the game design to be able to effectively make the multiplayer run on dedicated servers instead of P2P.
> Servers in FromSoft games are only for matchmaking. The actual gameplay has always been peer to peer. The design of the games mean that making the multiplayer entirely online server based would not work. They're peer to peer because that's the best way to support the hybrid singleplayer / multiplayer design where players drop into and out of your world. Otherwise your world state would need to be consistently synced to the server, which introduces major problems -- in particular, the fact that you would be dealing with lag in a game that you spend mostly singleplayer.
This isn't quite true, they could have on-demand spin up of a "server" and hand off relevant gameplay decisions to it but it would be a ludicrous overcomplication and expensive for one small part of the actual game.
Switching to a rollback system for their peer-peer networking would do a lot to prevent exploits though but would likely be a significant and difficult change with the determinism requirements.
genuinely lmao. you're not saying anything fromsoft devs aren't aware of. for the amounts of players they have, running typical authorative multiplayer servers that guarantee available spots for every player is an expensive undertaking, both in development time for all the backend stuff, as well as the actual running of the servers.
it's a business decision made by real grown up adults. you're being horribly childish yourself.
PC gaming unfortunately never evolved in form where games would run in a sandboxed environment. Linux + flatpak steam + proton is probably the closest you can get without too many practical issues.
While the game not being sandboxed creates risks for the consumers (as demonstrated by FROM with Dark Souls 3) and allows people to more easily exploit the game, it also makes modding and often times supporting a game long past its original lifetime possible.
Personally, I would love to have the benefits of sandboxing but I would not want to live without modding and community supported servers. The console market shows us how consumers are abused in an environment where they have no say in what happens to the software that they purchased.
How does sandboxing kill modding? You still can modify anything inside the sandbox, the running process just can't escape the sandbox. (Meaning any damage is mitigated, or more controlled) By a sandbox, I don't mean we should lock our systems behind a jailed garden like consoles do it.
Sandboxing itself won't. The way publishers and distributors will implement sandboxing will.
Someone will develop a reasonable, possibly quite open sandboxing technology. That technology goes on to be adopted by one or multiple major publishers. Over the years, anti-cheat and other proprietary detection mechanisms get more and more prevalent within it. Connection to multiplayer servers will no longer be possible without the publisher-provided sandbox. Modding will be more and more restricted until eventually, some form of paid-mods or user-store within the now completely publisher-controlled runtime environment appears.
Call me pessimistic but that's what I gather from my last two decades of gaming with highlights such as Origin scanning your whole machine, DS shadowbanning people, etc.
Right, this is a real fear on closed platforms like OSX and Windows. Microsoft is actually going to this direction with the pluton platform. But this also means the "personal" from PC gets taken away. The thing is, software developers themselves can't achieve lock-in and take over the control how the code gets executed on your PC if the platform you are running does not provide this feature in the first place.
So typical sandboxing on Linux will never go in the ways of DRM, or give control to the publisher of the software. Your fear will become true, if software starts targetting or even preferring a trusted platform instead of a open platform like the PC we currently have. But this is not fault of a sandboxing technology itself, it's the fault of companies looking to have control over their product and their users.
Unfortunately, I can see future going towards with trusted platforms, where users on non-trusted system will be heavily handicapped, for example netflix not allowing 4k streaming, multiplayer games not allowing you online, or putting you on different servers etc... Valve's linux + proton push may make developers keep targeting the proton WIN32 subset in the future, but we'll see...
Regarding modding, many online games do not allow it even in the current form, requiring you to play offline if the game is modified, or connect to a custom server. By the way, on linux you can easily achieve a sandbox using bwrap https://github.com/containers/bubblewrap
How is sandboxing relevant here? A sandbox limits the capabilities of untrusted software to affect the outside system. It does not do anything for cheating, the system administrator can affect the sandboxed software in any way they want.
I read through the whole series of posts in one sitting - impressively thorough!
As someone who spends a lot of time writing and structuring regular old REST APIs, it's a lot of fun to see an approach to API design that is rooted in the foreign convention of game design, even if the implementation language is in the ballpark of ones I use.
Interestingly, once you get past the authentication, the reliable UDP implementation, and into the "meat" of the behavior, it's the same sort of call-response stuff that anybody else is doing, with the same sort of fairly basic little payloads. The "special sauce" is all the components getting to that point - and the fact you're getting binary data and not JSON or something. Not the most exciting thing, if you're a game developer.
Sometimes I wonder what somebody who paid as thorough attention to systems I've worked on would say about them. What would the weird stumbling blocks be? What would the "magic" be that seems to make no sense? ("What are the security holes they find?" goes without saying, but that's feedback I get from our internal pentesters anyway.)
This part, yikes:
> As our encryption is asymmetric, you may be asking how the client decrypts messages send by the server if it only has a public key. To do this the game exploits an interesting quirk of RSA cryptography, its possible to encrypt with the private key and decrypt with the public key. This is exactly what happens when the server sends the client message. For anyone contemplating doing this though, don’t, it’s terrible for a number of reasons. You can read some of these reasons here, written by someone far more knowledgeable about cryptography than myself.
Great effort reverse engineering the protocol and producing an enjoyable write up.
> its possible to encrypt with the private key and decrypt with the public key
Isn't it just signing/verifying? I.e. "encryption" with a private key is how RSA is being used to sign, and "decryption" with a public key - used to verify the signature.
I thought so too. Encrypt the hash with private key -> decrypt with public key and check the hash.
This could well be it, though if the plaintext isn’t sent (unclear from the article if it is, I think not?) then all the client can really do is decrypt with public key and check that the output appears well formed, which may be good enough.
It would seem odd, however, that client -> server is encrypted but unsigned whereas server -> client is signed but unencrypted. My guess, for what it’s worth, is that somebody at the company insisted that all comes must be encrypted and so they cobbled this together along with the obfuscation of the public key the author describes.
Considering it is the basis for the entirety of RSA PKI / RSA signatures, I wouldn't call being able to decrypt with the public key a quirk.
That quote doesn’t seem to appear in the article, or at least not anymore?
I was curious to read the reasons. The ones that come to mine are the obvious ones that EC is significantly faster and more secure and that you should only use asymmetric to negotiate a symmetric key exchange (eg ECDH) since symmetric is going to be even faster and suitable for real-time streaming.
The quote is from the second article in the series.
You can find it here: https://timleonard.uk/2022/06/02/reverse-engineering-dark-so...
I've been reverse engineering another online game and encountered very similar themes in terms of encryption. Lots of weird obfuscation, pseudosecurity, and obvious vulnerabilities. Instead of choosing something battle-tested like TLS they roll their own crypto and fail miserably.
Lots of f2p mmos still do the good old xor encryption https://git.cloudef.pw/sw-server.git/tree/src/packet.c#n32
One fun project I did a while ago was reverse engineering one of the games from my teens, Runescape 2 (now called Old School Runscape, where the game is a backup from August 2007 with many updates since then).
There is a huge interest in the game, with many players still playing it, I had fun for a couple of months for a few hours a week.. but then noticed there was a field in a client you use (nobody uses the official client, they use Runelite, which is an amazing opensource client for OSRS).
I reverse engineered the login protocol, it was a lot of fun. Erlang pattern matching on binaries with Elixir made this incredibly easily and very fun to implement.
Amazing write-up! Any good resource how to do this kind of reverse engineering? Maybe with simpler/older games first.
Depends what you want to do. But good debugger is absolutely essential. Reversing communication? MITM the communication. Want to understand the game's logic / find entrypoints? Memory scanner/editor -> breakpoint to a address in debugger. Combine above with disassembler with good code analysis tool and you'll be able to follow the code even without actually having the source. In the end reversing is basically poking around and experimenting, but it is very rewarding and will teach you a lot how software work and how easy it is to actually make them bend to your will. Be wary, you'll need lots of free time for this.
Quote: "However all multiplayer traffic in FromSoftware’s games is actually sent entirely peer-to-peer."
This is very important to understand why currently DS3 PC server is closed for half a year now due to cheaters. FromSoftware does a childish attempt at creating a server hence the plague of cheaters. And what they do? They released Elden Ring instead of working to properly create a server. FromSoftware knows how to create console games, but a PC game is outside of their expertise.
What exactly is your point here? Console games need (mostly the same kind of) netcode as well. They are just generally harder to exploit, which dramatically lowers the amount of people trying to do so.
They are capable of writing a netcode that is good enough for an environment where it is relatively safe due to the restrictions of the devices it runs on but not for any true open environment. - So they just write bad netcode, that's all.
My point is that if you want to create a proper server you do the heavy lifting on server, not on client. The second you let the client decide who does what (location / damage / item drop etc etc) you get cheaters that will modify their client to bid their rule. Hence why I consider FromSoftware servers childish.
> Hence why I consider FromSoftware servers childish.
Servers in FromSoft games are only for matchmaking. The actual gameplay has always been peer to peer. The design of the games mean that making the multiplayer entirely online server based would not work. They're peer to peer because that's the best way to support the hybrid singleplayer / multiplayer design where players drop into and out of your world. Otherwise your world state would need to be consistently synced to the server, which introduces major problems -- in particular, the fact that you would be dealing with lag in a game that you spend mostly singleplayer.
They would have to rework the game design to be able to effectively make the multiplayer run on dedicated servers instead of P2P.
> Servers in FromSoft games are only for matchmaking. The actual gameplay has always been peer to peer. The design of the games mean that making the multiplayer entirely online server based would not work. They're peer to peer because that's the best way to support the hybrid singleplayer / multiplayer design where players drop into and out of your world. Otherwise your world state would need to be consistently synced to the server, which introduces major problems -- in particular, the fact that you would be dealing with lag in a game that you spend mostly singleplayer.
This isn't quite true, they could have on-demand spin up of a "server" and hand off relevant gameplay decisions to it but it would be a ludicrous overcomplication and expensive for one small part of the actual game.
Switching to a rollback system for their peer-peer networking would do a lot to prevent exploits though but would likely be a significant and difficult change with the determinism requirements.
genuinely lmao. you're not saying anything fromsoft devs aren't aware of. for the amounts of players they have, running typical authorative multiplayer servers that guarantee available spots for every player is an expensive undertaking, both in development time for all the backend stuff, as well as the actual running of the servers.
it's a business decision made by real grown up adults. you're being horribly childish yourself.
PC gaming unfortunately never evolved in form where games would run in a sandboxed environment. Linux + flatpak steam + proton is probably the closest you can get without too many practical issues.
This has its own up- and downsides.
While the game not being sandboxed creates risks for the consumers (as demonstrated by FROM with Dark Souls 3) and allows people to more easily exploit the game, it also makes modding and often times supporting a game long past its original lifetime possible.
Personally, I would love to have the benefits of sandboxing but I would not want to live without modding and community supported servers. The console market shows us how consumers are abused in an environment where they have no say in what happens to the software that they purchased.
How does sandboxing kill modding? You still can modify anything inside the sandbox, the running process just can't escape the sandbox. (Meaning any damage is mitigated, or more controlled) By a sandbox, I don't mean we should lock our systems behind a jailed garden like consoles do it.
Sandboxing itself won't. The way publishers and distributors will implement sandboxing will.
Someone will develop a reasonable, possibly quite open sandboxing technology. That technology goes on to be adopted by one or multiple major publishers. Over the years, anti-cheat and other proprietary detection mechanisms get more and more prevalent within it. Connection to multiplayer servers will no longer be possible without the publisher-provided sandbox. Modding will be more and more restricted until eventually, some form of paid-mods or user-store within the now completely publisher-controlled runtime environment appears.
Call me pessimistic but that's what I gather from my last two decades of gaming with highlights such as Origin scanning your whole machine, DS shadowbanning people, etc.
Right, this is a real fear on closed platforms like OSX and Windows. Microsoft is actually going to this direction with the pluton platform. But this also means the "personal" from PC gets taken away. The thing is, software developers themselves can't achieve lock-in and take over the control how the code gets executed on your PC if the platform you are running does not provide this feature in the first place.
So typical sandboxing on Linux will never go in the ways of DRM, or give control to the publisher of the software. Your fear will become true, if software starts targetting or even preferring a trusted platform instead of a open platform like the PC we currently have. But this is not fault of a sandboxing technology itself, it's the fault of companies looking to have control over their product and their users.
Unfortunately, I can see future going towards with trusted platforms, where users on non-trusted system will be heavily handicapped, for example netflix not allowing 4k streaming, multiplayer games not allowing you online, or putting you on different servers etc... Valve's linux + proton push may make developers keep targeting the proton WIN32 subset in the future, but we'll see...
Regarding modding, many online games do not allow it even in the current form, requiring you to play offline if the game is modified, or connect to a custom server. By the way, on linux you can easily achieve a sandbox using bwrap https://github.com/containers/bubblewrap
How is sandboxing relevant here? A sandbox limits the capabilities of untrusted software to affect the outside system. It does not do anything for cheating, the system administrator can affect the sandboxed software in any way they want.