Mod Kitty's here to tell us about how our brave QAers test the NXT client, and what we've learned from the last beta weekend.
Don't forget - there's another Closed Beta Weekend to come, starting on Friday 18th March at 12:00 UTC!
This week's dev blog is a bit different, as it's written from the QA Team's perspective. Ever been curious about just how we assess the quality of a game as huge as RuneScape, re-vamped in a shiny new engine? Read all about it here!
Spot the Difference
As we saw the RuneScape world taking shape in NXT, it was noticeably different to what we were used to in the old client. The dynamic shadows, the new lighting system, the increased draw distance and all the other new additions created a whole new gaming experience, but we had to make sure it was still immediately recognisable as the game you know and love.
We've spent a lot of the time testing the Java client side-by side with the new one, and to help with this I developed an automated test that actually went through 30 different locations to take side-by-side screenshots of the two clients. This was handy for showing us how the environment changed, and we used them to closely match the mood of each location in terms of colour and lighting.
 |
 |
Content bugs and engine bugs
The environment and models that were made in the last 15 years were built within the constraints of the Java client. These limitations no longer apply with the NXT client, and once development was underway we found that a lot of elements in the world seemed a bit out of place. For example, many locations seemed a lot closer to each other than you might expect them to be; locations or NPCs would seem to float in mid-air because the building was set to be loaded a lot later; and certain materials seemed a lot shinier, because the lighting was different.
We've come up with a programmatic solutions for default lighting and environmental effects, but this won't work well enough for all environments, so we've given artists the ability to customize them. We'll get there, but it'll be a fairly slow process – particularly working alongside weekly updates. Please bear with us!
Bugs Reported over the last Beta Weekend
First thing that needs to be said: thanks for all the huge volume of awesome feedback you gave us over the Beta Weekend. Secondly: we've read every bit of it!
The NXT developers and the QA Team have all put in time to go through the enormous amount of data that we collected after the Beta Weekend. That includes:
- 15,000 in-game bug reports
- 50 forum threads
- Server logs detailing client crashes
- Loads of feedback on Twitter, Reddit and YouTube.
We went through everything in fine detail, not to just spot new bugs, but to also make a heat map of bugs and issues that were reported most often.
Here's what we found out:
Some players were not able to actually start the client, or had problems installing it
We've learned a lot about these issues from players who posted on the 'Critical Bug Reports' thread, and have tried to tackle all of them. By the next Beta Weekend, there should be far fewer people who report that they can't install or start NXT – hopefully none!
Crashes!
Re-writing the engine in C++ has allowed our developers to closely control and optimise the PC's resources, gaining a serious performance boost. However, rebuilding something so complex from the ground up means that crashes are inevitable.
Thankfully, we've prepared for this and we do have a crash reporter implemented, so most of the time the client crashed, our servers have picked up the errors and a lot of them are already fixed.
From your feedback, we've seen four types of in-game crash:
- Crashes on transition from the Lobby to the game – Those were mostly compatibility issues.
- Crashes on launching the application - Mostly related to missing libraries, necessary operating system updates, or wrong permissions.
- Crashes after an hour or so of playing – This means the player actually encountered a condition for a crash.
- Runs the first time but crashes the second time due to "Error Saving File" - Probably there's a residual rs2client.exe process still running and preventing the new one from accessing its resources. This issue should now be fixed.
Compatibility issues
This means the game behaves differently on certain combinations of hardware and operating systems. We've always been invested in delivering the game to as many players as possible, and that means assessing the quality of a player's experience with RuneScape on as many PC configurations as possible. This was ensured in three main steps:
- Making use of our in-house compatibility testing lab currently made up from more than 30 machines – desktops and laptops - each with different combinations of operating system, CPU, GPU, etc.
- Opening NXT for the staff to play at home, allowing us access to an even broader spectrum of computers.
- Employing third party testing companies and commissioning compatibility tests for those machines as well.
Testing that our client works as intended on dozens of PCs can be a very time-consuming and repetitive job, but we've made it easier by setting up an automated testing framework using Python. Pairing this with a comfortable reporting system, it allows us to set up and trigger tests as often as needed on multiple machines simultaneously, and closely monitor NXT client's performance each build at a time.
This is nothing new, as we've been using automated testing for the Java client for years. It was just a case of scaling up both tests, and increasing the number of machines running the tests.
The Beta weekend threw up a number of compatibility issues that we didn't see in our staff testing, or by our external testing company. We've investigated these issues, and found a few common causes. Follow these steps for the best NXT experience:
- Don't run RuneScape as Administrator: This can lead to essential files not having read/write privileges.
- Update your operating system: Ensuring your OS is as up to date as possible helps to run NXT, and improves the security and stability of your PC.
- Update your GPU drivers: With older drivers, some integrated Intel chipsets or older Radeon GPUs can display graphical glitches like white intermediary frames or long shimmering polygons. For everyone, it helps get best performance.
We also saw some issues with older versions of OSX. We'll be running further, more thorough tests on these versions of the operating system.
Performance issues
Our goal is for RuneScape to look and play like a modern-day game. Any 3D game released in the last few years will make your GPU work for its money: it's the nature of a 3D graphics applications, and the very reason GPUs exist.
I have read on the forums that some of you are concerned about your machines overheating. As long as you take due care with any overclocking, it should be no problem at all – NXT won't tax your PC more than any other modern 3D game (and a good deal less in many cases).
Another common debate on the forums was: "I run Java on High, why should NXT run on Low?" If you've read the previous tech blogs, you'll realise that the High setting in NXT is not exactly like the High setting in Java. Actually, just a quick glance will suffice:
 |
 |
Umm...Where's my tower?
It's important to note that if you're running the NXT client with Low settings, you'll still have more features enabled than in the old client set to High. And the best thing is that you'll still get a better frame rate, while using less CPU!
 |
 |
Notice the dynamic shadows under all animating entities.
And this is how the performance graphs look for a laptop with an Intel(R) HD Graphics 4000, an Intel(R) Core(TM) i5-3210M CPU at 2.50GHz on Low Graphics Settings (while just resting near the Lumbridge Castle):
 |
|
| FPS - NXT | FPS - Java |
 |
|
| CPU % - NXT | CPU % - Java |
Lag
A lot of people have been reporting that the client is sluggish, unresponsive and slow to load. We have identified some cases where we need to optimise the way we load assets, especially for CPUs with 4 cores or fewer. This is currently being addressed, which should help players who're experiencing these issues.
These loading issues have mostly affected the look and feel of the animations and combat, and a lot of bug reports came in about them: freezing after playing one animation (like Surge), playing in place, or not playing at all.
Sprites corruption/icons not looking okay
After playing NXT for a long period of time, many players reported seeing duplicate items in their inventory, dark shadows around objects in the world and other similar graphics errors. This was a particularly hard one to fix, especially since it only happens after quite a few hours of continuous gameplay. We have added some code to prevent such issues, but further stress testing is required to ensure the fix works perfectly.
Combat issues
The users that tested NXT during the beta weekend have had quite a few complaints about how unreliable combat feels. This was mostly affecting boss fights, where players reported that hit splats were not rendered as always on top, bosses were sometimes out of place, and some abilities didn't look or behave like they were supposed to.
In the few weeks after the beta weekend, we prioritized fixing these issues, so you won't have to worry that testing NXT means risking your bank and items.
Other issues
We've been focusing on fixing how the minimap behaves. It had problems updating for areas that were set to fade in on teleport, and it sometimes failed to update on transition, if the client was taking a bit too long to load in the new area.
Other problems spawned from our drive to optimise and improve the way the old system works. Therefore, picking is still a work in progress. Due to our increased drawing distance, this means a user could click on an NPC that is one town away, if it's visible on the screen.
This requires pixel-level calculations, on multiple levels of screen depth, and that means lots and lots of searches. We've tried our best to come up with various smart solutions for this, but some edge cases, like NPCs that change shape when they spawn, it still needs improving.
The legacy
I hope that this post managed to paint a better picture on the work we do in QA and what you can expect in the future from the new client.
We use the word 'legacy' a lot in our team. In most instances, it refers to the way old content works and how we should improve on it, but it also means the legacy of players that have stayed with us thought all these years, and the pressure to deliver the same awesomeness they are used to. We're well aware that our users have been used to a certain quality of user experience and we're striving to deliver it and much more in order to meet their expectations.
Mod Kitty
RuneScape Tech QA