Inside the Spudworks – DDoS and ping
Tl;dr: DDoS attacks are bad. They make players mad, which makes us sad. We’ve made some changes, which we think are rad.
Hey all! The infrastructure J-Mods are back with another Spudworks blog. This time, we’re diving into DDoS attacks and ping. As usual, this will be a tech-focused post, but we’ll do our best to explain the concepts as best as we can – and if there’s any follow-up questions let us know on our socials and we might cover them in a future blog!
Traffic Cops
Over the summer, our beloved games reached new heights. As J-Mods and fans, this was lovely to see. Unfortunately, it also brought with it some bad actors and at a few points RuneScape and Old School RuneScape became unavailable to players due to Hyper Volumetric Distributed Denial of Service (DDoS) attacks.
Part of our job here in the Spudworks is to ensure that your tick-perfect clicks reach us in a timely and consistent manner, even under adverse conditions. So, let’s dig into why these attacks were so disruptive.
What is a DDoS Attack?
Let’s start with what a Denial of Service (DoS) attack is. These involve a malicious client communicating with a server in such a way that it prevents the server from responding to other legitimate clients. There are three broad categories of DoS attacks:
- Volumetric – Flooding the server’s network interface with more traffic than it can handle. Imagine a blocked toilet and you get the picture.
- Protocol – Abusing how legitimate interactions work to consume server time or resources. Imagine an old lady paying the cashier in pennies and holding everyone else up.
- Application – Targeting application weaknesses or spamming legitimate requests. In the above example, paying normally at the cashier but only buying one item at a time over and over.
Denial of Service (DoS) attacks are easy to mitigate because the attack comes from a single source: one bucket of Ugthanki dung, one old lady, or one loopy customer. They are easy to detect and easy to block.
Distributed Denial of Service (DDoS) attacks spice this up by using the same methods but from lots of different sources – often fleets of compromised IoT (Internet of Things) devices such as the Mirai Botnet – to make it harder to distinguish the bad traffic from the legitimate traffic.
What does Hyper Volumetric mean?
A hyper volumetric DDoS attack is the largest and hardest to stop, exceeding a Terabit per second of bad network traffic. Attacks of this class can be more than 1000 times the volume of our normal traffic patterns.
You can imagine Jagex’s network capacity as a four-lane local highway, leading to Jagex’s game servers. The cars on this road come from a global road network and represent packets of data sent by lots of different clients.
To enter the Jagex highway, cars must pass through a checkpoint – where the good traffic is filtered from the bad. These checkpoints are highly effective but are not perfect. A small percentage of bad traffic will slip through.
On a normal day most cars are legitimate traffic, requiring just one lane of our highway’s capacity. However, during a hyper volumetric DDoS attack, imagine it like the worst traffic jam you’ve ever seen (for our British players, think of the M25 at rush hour)! The checkpoint is still doing its job, but so much traffic is arriving that allowing even a small fraction through is enough to block the local highway. All four lanes are congested and traffic waiting to get onto the road is queuing up.
It takes so long to get through the queues that legitimate traffic starts to give up and go home. This is expressed in the real world as a network timeout.
How does Jagex respond to DDoS attacks?
Jagex is targeted by hundreds of DDoS attacks each year. In an industry wide trend, they are getting both more common and more disruptive. Almost all attacks are automatically mitigated by Jagex’s defences.
The Jagex network is monitored and protected 24/7. When an attack impacts our players, infrastructure engineers – along with engineers from our partners – jump into action. We can’t go into the specific mitigation steps we take for obvious reasons, but engineering teams will actively try to analyse and counter the attack.
What is Jagex doing to protect against future attacks?
There are three main ways in which infrastructure can be expanded or improved to handle these kinds of attack
- Increase our global traffic scrubbing (the checkpoint from our road analogy) capacity and refine its rulesets to reduce the amount of illegitimate traffic which leaks onto our own network (local highway) during an attack.
- Increase the capacity of our own network (local highway) to provide extra headroom during an attack.
- Increase the capacity of our backend networks (private roads) to better handle stampeding herds of legitimate players spamming us with logins/retries during or immediately after an attack.
In response to the recent attacks, infrastructure J-Mods have:
- Overhauled and significantly increased our global scrubbing capacity (checkpoints).
- Expanded the bandwidth of our transit connections (local highway) by 300%.
- Expanded the capacity of our backend networks (private roads) by 400%.
So, no more DDoS then?
We can’t guarantee that. Our previous infrastructure and mitigations were sufficient, until they weren’t.
Protecting our games is an arms race, but we will continue to evaluate, invest, and improve.
What about my ping?!
“But anonymous potato peeler who writes this blog, you said you’d talk about ping? Why was my ping so bad for a few weeks after the attacks?”
Well ever-curious player, that is an excellent question.
Let’s start with a convenient potato-based segue: Hot-potato routing, also known as early-exit routing. This is a routing strategy commonly employed by ISPs to ensure that your packets take the cheapest, not necessarily the quickest, route out of each intermediate network.
In the example below, ISP A could get your packets to their destination faster but due to the hot-potato strategy will prefer a slightly longer route which minimises their own network costs.
To provide players with the best routes to our game worlds, Jagex employs “Anycast” addressing to advertise every game world from many regional Points-of-Presence (PoPs). This way, hot-potato routing encourages ISPs to exit as early as possible into the Jagex network so that we can carry your packets directly to their destination using our backbone.
We noticed that some attacks targeted only one region at a time. Due to anycasting, high volumes of bad traffic were hitting all our Points-of-Presence impacting players in regions not targeted by the attack.
As a temporary measure to combat this, we disabled anycast for a few weeks. This isolated regional attacks and took the pressure off our backbone network but came at the cost of increased latency for many users.
As we brought our new defences, expanded Points-of-Presence, and increased network capacity online, we brought anycast addressing back into service. This will have restored normal ping for most players, but expanding our POPs will change routing for some ISPs and will not benefit everyone evenly. We have more plans in this area to share in a future post.
FAQ
Here's a quick Q&A based on questions and comments that we've seen crop up frequently in the past!
Why can’t you just use Company XYZ?
We evaluated multiple DDoS mitigation partners, both indie and enterprise scale, as well as improving internal tools and processes. Due to the unique nature of our games, what works for one company may not work for us but we’re confident we have chosen the best solution for our problem space.
Who was the attacker?
It is very difficult to attribute the hiring of botnets to groups or individuals. For significant attacks, Jagex engages with law enforcement to investigate.
What about the Cloud worlds? Will they help with this?
More capacity always helps, but DDoS mitigation services like AWS Shield Advanced or Azure DDoS Protection are not necessarily an improvement on our current approach.
We've been making progress on Cloud based worlds. Stay tuned.
Why didn’t you have better defences in advance?
We deal with hundreds of DDoS attacks a year. Most of the time our defences automatically protect Gielinor without manual intervention, meaning very few of these attacks cause major disruption.
Attackers are constantly adapting and scaling their methods. We’re always learning and will continue to invest in this area.
This sounds like a really fun challenge to solve, can I help?
If you find challenges like this exciting and would love to be involved, keep an eye out for a vacancy that suits you at https://www.jagex.com/careers.
Who is mod 🐹?
Computing power measured in squeaks per second and the gnashing teeth of righteousness (DDoSers beware)!
“Can you write a blog about XYZ?” / “My question isn’t covered here!”
Drop us a note on the subreddit or the official discord and we’ll try to cover it in future.
Mods Kraken, Vxp, Bash, Maniac, Vallcore, Haydon, Cky, Qwert, Drax, M0iqp, Ibex, Adad, Roman ... and 🐹