We're posting this newspost as a way to collate outgoing communications regarding Tuesday's service outages, as well as provide a brief update on any ongoing issues players may be experiencing.
Thanks for being patient whilst we finalised our plans for 'Making Good' following the recent downtime experienced on November 22nd when one of our external Data Centre providers suffered a site-wide power failure. We know downtime can disrupt your valuable spare hours to play the game, and we'd like to make up for what was lost.
As a result, all Active Members who have logged in at least once within 28 days of Monday, November 21st will be receiving a 2 Day Membership Voucher Passcode. We will begin rolling these codes out to player inboxes today, and we expect all players will have received their code by Tuesday, December 6th. We'll confirm when all codes have been delivered, so please do not contact support before this time as you may just be later in the queue! Please note, the Voucher Passcode will be valid for use until March 31st, 2023.
In order to redeem the code once you've received it, follow these steps.
- Step 1: Go to your Messages via your account settings
- Step 2: Follow the advice in the message titled ‘Data Centre Outage Follow Up’ to claim and redeem the voucher code (or gift it to another player)
- Step 3: Enter the code on the Code Redemption Screen
- Step 4: Confirm you want to claim the membership on the account you are logged into
- Step 5: You will be shown confirmation of a successful redemption
Reported Lost Items
We've seen reports from various players stating they lost items during the outage, some involving items they were trading at the time, others involving their Group Iron or Clan Hall storage system. While we restored player's profiles, Clan profiles and GIM Group profiles to the most recent backup points available, it's entirely possible that items could have been lost if they were traded or taken in/out of those storages moments before the servers went offline. Player profiles, Clan profiles and GIM Group profiles are stored across multiple different servers, and cannot all back up their data at the precise same moment. For example, if a player deposited an item moments before the outage, the player's profile might have been updated to reflect that they no longer had the item, but the Group Iron and Clan systems might not have been updated to reflect that the item was stored. Thus, when the outage hit, this difference could result in the item being lost.
Unfortunately, due to the nature of the outage and the limited data available in its aftermath, we are unable to validate which players lost items, let alone the actual items that were lost. Our only alternative at present would be to give any player whatever items they might claim to have lost, which would generate items with an extremely high risk of abuse that would directly harm the game's economy for everyone, so we have had to make the difficult decision to not return any items that have been reported as lost in this way.
We're very sorry we are unable to help further in this matter, but rest assured we're actively investigating how we can avoid similar issues like this occurring in the future when a disaster of this scale occurs, and we'll inform you of any progress we make as we're able to!
So What Happens Next?
One of the things we've taken from this incident is the investments we've made into security and safety of player data have really made a difference. After the multiple failure point issue that caused the Login Lockout for RuneScape, we committed to doing even more to look after player data under any circumstances. In the time between then and now, we've been modernising our systems and implementing more robust recovery options in the event a disaster should occur.
As a result of that work, even with the game being down for a significant amount of time and a series of issues to resolve once we got back online, only 6 accounts were corrupted by the outage, all of which were identified and returned back to their normal state.
However, we're not done. While this issue is with an external partner, we are working with them to ensure a failure of this level cannot happen again. We are also considering further backup options that may enable a quicker migration of services should a similar incident occur, alongside improvements to the Group Iron and Clan storage systems.
Finally, we'd just like to say a big thank you once more for the patience you've showcased since last week. The calm, supportive, and eager to play attitude on display really does make a difference to team spirit as we've worked flat out to get you back playing at all costs.
The Jagex Team
Following recent server outages which caused disruption when attempting log-in from certain regions, we've been working closely with our system administrators to ensure all players are now able to log back into the game as normal.
The team has made some minor adjustments to our authentication services to ensure that further reports of disruption when logging in have been rectified. Game stability over the last 48 hours looks promising, and we will continue to monitor stability over the weekend to ensure that things are fully back to normal and to fix them promptly if anything more happens.
Before the weekend, we would like to let you know that we are still in the process of reviewing our make good options for players. Next week, we'll set out and communicate our plan for this as we're still working out some of the technical and delivery details to make this happen.
We'd like to thank you for your patience as we work on bringing the servers back to normal. We look forward to sharing further details on our make good plan with you next week!
On Tuesday (November 22nd), one of our external Data Centre providers experienced a site-wide issue involving a power failure which resulted in full downtime of any services housed there. Our engineers worked to get you back online as quickly as possible, and services were restored earlier yesterday morning at approximately 02:05 GMT.
Unfortunately, while most of our systems have recovered from the hosting centre's outage, the login systems have since struggled at times and players may therefore have found themselves unable to log in. We have identified what was causing these login systems issues and we believe that it should remain stable barring anything unforeseen happening.
We'd like to reassure you that as we continue to recover from the outages, our systems administrators are automatically alerted to any other outages when they happen, and can usually restore access to the game within minutes. However, in the meantime, you may prefer to avoid risky activities, in case you log out and find yourself unable to log back in quickly.
Unfortunately, we'd also like to inform you that when restoring the Polling System, the data we had stored for the ongoing 'Mysterious quest boss rewards & Revenant weapons poll' was corrupt, meaning that we no longer have your votes counted. We've since restarted the poll, and we'd like to ask you to cast your vote once more. It now closes on Tuesday, November 29th.
We're very sorry for any ongoing disruption that's followed Tuesday's hosting centre outage, and are working to restore stability. We understand that you're probably very curious about the reasoning for the outages, but we don't have anything more to share at this time. Discussions are ongoing regarding 'Making Good', as previously announced.
The Jagex Team