QA Survival 101 — How to prep yourself before getting your games tested

Get your game ready for a QA round and learn how to get the most out of testing.

written by: Szymon Fratczak, Game & Monetization Designer

Introduction

Patching bugs is an unavoidable part of game development. While it is hard to believe that, almost every game contains them. Even if we had spent the last year and a half on fixing our creation and think that it cannot be broken, there are over 7 billion people living on this planet ready to prove us wrong.

Each and every one of us plays games differently, so it is hard to predict what each person will do in a given situation. If you place a huge mountain in a world of your making and then tell the player to go around it, I can guarantee that there will be people who will try to take a shortcut by climbing itin a straight line (and the brave lot will even do it by horse). A developer might have not expected such development. The result? Our hasty would-be-mountain-climber finds a spot with no environment loaded in, falls right into an endless void, and if that was not enough, their save file gets irreversibly corrupted. Oops.

There are endless examples of what could go wrong, so it is best to find as many as possible even during development. The perfect people for that are the Quality Assurance (QA) testers.

In this article, I will attempt to explain procedures of quality assurance using my own experience as a tester and as a test lead in companies specialising in outsourced tests.

Preparing the build

In what state does one send a build for testing? If anything, it has to run. It is the most basic of requirements that one has to be ready for. The game can be full of bugs from the beginning of it to the very end — after all, the QA testers are there to report those. You should, however, check if your project does not suffer from defects (“blockers”) that halt progression on the main route. These have the possibility to stop the testing team from accessing an important part of the game.

Another matter is implementing debugging tools. While the game does not have to have them implemented, unlocking routes, changing levels, teleporting or changing currency values are always good time savers when it comes to a QA tester’s work. In other words, it is a good idea to spend a few hours on adding such tools so that others can save a few days of work.

The Plan of Action

It’s a good idea to plan tests quite early. Doing so a few weeks, or a month, ahead of the preferred testing cycle date allows you to discuss the necessary details before it all begins. Additionally, it is an opportunity to ensure that you get to pick the period of time that fits you best and, possibly, has the most QA testers and platforms to test on available.

At the beginning, it is quite important to establish when a testing cycle will take place, how long it will take, and how many testers are going to be needed. (However, the last bit can be handled by a specialist on the side of the Quality Assurance team.) It all depends on the size of the game and how you wish to set up the workflow. The usual way of handling all that is to have a week of testing, which is followed by a week of fixing the reported bugs. After that, a few days are spent on going through the applied fixes, so as to ensure that everything was done properly and there are no adverse effects.

When we know in what state the build will go to QA, it is a good idea to take a moment to think on what should be tested. In other words — talk about your priorities with the people responsible for organizing QA tests. Remember to add detailed information about which platforms are supported so that the team leaders can decide on the most effective testing plans.

At the end of the list are the technical details. What you need to decide on is what tool will be used to report and track bugs, through what (FTP, Dropbox, Google Drive etc.) the reports will be sent and accessed, as well as the preferred way of daily contact. While e-mails are usually good enough, some might prefer Slack or Discord. It’s also a fair idea to consider whether you will want to keep in contact with the team leader alone or if you’d prefer to have direct contact with the QA testers themselves. Do not forget about potential time-zone related differences in contact availability, as you might need to prepare for difficulties related to that.

And while we are on the topic of contact with QA, it’s a worthwhile idea to settle on the form of reports that will summarise the tests. Daily reports are a frequent choice. They usually include a summary of work that was done, reports of defects, suggestions, and results of retests (if those took place). If needed, it’s possible to set up weekly reports as well as a finishing report that binds all of the aforementioned information together.

Bugtracker and reporting bugs

The market has a fair number of free and paid tools used for reporting bugs, and a lot of them can be pretty similar, which can make the choice of which to pick a bit of a problem. It’s a good plan to check the available free versions so as to make up your mind as to which software fits you best. While outsourcing QA, it’s quite possible that the testers will recommend their own bugtracker.

One of the most popular choices in such software is Atlassian’s JIRA. It is fairly easy to use and allows you to tackle a fair amount of possible problems related to tracking bugs. The obvious downside is that you need to pay for it.

Below, you’ll find the most frequently encountered elements that can be found in tickets issued by QA testers. Before the start of the testing cycle it’s a good idea to settle on details of your requirements regarding such things as the template, attachments, or even the account of a person that the bugs will be assigned to once they are created and fixed.

1. Summary — In a nutshell, it is a short description of a given defect in the game. The QA can use specific tags that allow for easier browsing of all the reports. For example, the tester can point out which platforms suffer from the given bug, as well as its location within the game (e.g. the name of a level) or the type of it (UI, Gameplay, Multiplayer).

• Example: [iOS][Android] Gameplay — The player is unable to use the lever at the end of the tutorial

2. Severity — As the name suggests, this shows the impact the bug has on the game. Stratification values used will differ from person to person, but the most often used scale goes from 1 (a critical progression blocker) to 5 (suggestion/a trivial bug).

3. Priority — Often interchangeably (and wrongly) used with Severity, Priority is more of a subjective point used to show the importance of a given task in comparison to other ones. It is fairly fluid as the given task may be not as important during the first week of development, while it could be critical during the last week before the game release. Priority is most useful in larger projects where Severity might not be enough to plan the development altogether.

4. Description — A detailed description of the bug. It includes all of the required information regarding the defect. It’s an expansion on the Summary.

5. Steps to Reproduce — Step-by-step instructions in how to reproduce a given bug.

6. Actual Results — Description of what happens after one follows the instructions of the aforementioned ‘repro steps.’

7. Expected Results — Description of what should happen once the bug is fixed — according to the QA tester.

8. Repro Rate — Frequency of the bug after executing the fifth point. The frequency is usually expressed in percentages (100% — always happens, 50% — happens every second attempt etc.) or in a form that points out the number of bug’s appearance against the number of attempts, e.g.: 5/5, 3/10.

9. Attachments — Files attached to the report. Logs, crash dumps, saves as well as screenshots or recordings which show results of execution of ‘Steps to Reproduce.’ Try to let the testers know what you need and they will deliver.

10. Notes — Additional information which, according to the tester, should be helpful in fixing the given bug. This field is most often used for things that did not fit the previous points, such as the game version in which the bug occurs.

When testing is done

First of all, congratulations! You’ve survived your first QA cycle.

Your database is filled up with new bugs and your e-mail box already has a fresh report for you to read. Take a look at it, as you will be able to learn what the testing team encountered over the last few days, and verify the state of the game before adjusting priorities.

According to QA testing procedures, bugs which were marked as fixed should be taken through the process of retesting as well as regression testing on a newer version of the game. What does that mean? In short, testers confirm whether bugs were eliminated properly as well as whether the implemented fixes did not result in new defects. If everything went well, the bugtracking ticket gets closed. Otherwise, it is reopened with additional information for the developers. These processes are just as important as the initial testing, therefore it is not a good idea to underestimate them.

Tips

1. Try not to send new builds too often. It is especially important when it comes to larger titles. Having to upload them to the cloud so that they can be downloaded and placed on multiple machines can take up a lot of time, which could be spent on combing through the game in search of new bugs. Additionally, if the game uses a save system, the old game version save files may become incompatible with the new one. Testers then must erase their progress so as to avoid finding bugs that will not make it to the final version of the game anyway while still having to report them. It’s a waste of time, so the best compromise is to send a new version every 2–3 days. It’s usually a good idea to do so more frequently in the case of finding and patching critical bugs that block the progress of testing itself.
2. Communication is important. While the statement is not particularly groundbreaking, it’s commonplace for testers to have questions regarding the game as they work with it. I recommend that you reply to their questions with detailed information (or even share documents that might help), as a good QA tester will try to verify every element.
3. Leave precise commentary within tickets. I’m mainly speaking about notes which you leave under bugs that you have fixed and sent back to the tester so that they would retest them. The most important piece of information is the build’s changelog number, which points to the version that contains the fix (or at least the day of sharing the build on which the fix will be there — depending on your workflow). Regardless, descriptions of what you’ve fixed, which grant information regarding the game’s mechanics, allow for testing procedures that are broader and more precise. Also — they are fun to read!
4. Leave room for suggestions. Many testers are gamers, who tried out tens if not hundreds of different titles. They will be keen to share opinions on your game.
5. Add feedback. Share a good word or a bit of constructive critique at the end of the cycle. We all make mistakes, even though we do not always spot them. Getting feedback from a customer happens fairly rarely (especially, if you do not ask for it directly), but it is always a good thing that helps fix imperfections and makes the QA team become better.

Summary

I was once told that game testers are the silent heroes of game development. They sacrifice many hours of arduous searching through virtual worlds so as to find as many bugs as possible. They can show the developers what awaits them once they swim out onto the seas of Google Play or Apple App Store.

Well organized cooperation with QA will benefit not only the dev, but also the gamers. After all, everyone wants to achieve the same goal: creating a game that everyone can have fun playing.