martes, 30 de diciembre de 2014

Game Testing 101: Reports and Methods



So, you want to be a Game Tester? Or perhaps you are already one, but want to share your experiences and see others'? Game Testing is a perfect way of entering the Game Industry. Being "paid to play games" is every child's dream. 

This is my second #TestingTuesday post, a continuation to the guide Game Testing 101. Every #TestingTuesday I will write a Game Testing 101 article to complete the Game Testing 101 series.

A walk in the past

Last Testing Tuesday highlights:
  • Game Testing is methodical attention to detail applied to finding mistakes and incoherence in a game.
  • A videogame has many stages, and the testing must correspond to the current stage of the game. In the Pre-Alpha phase the basic game architecture is programmed, not much experience-oriented testing is needed, however technical testing is needed. In the Alpha stage experience-oriented testing is normal in terms of game mechanics and basic game elements. The Beta phase, in which every feature is included in the game, is the most testing-intensive phase, both user-experience oriented testing and technical testing. 
  • While it is true that every game is different, giving some structure to testing maximizes the results, with this purpose we should guide our testing with some specific categories:
    • Gameplay and Mechanics: The rules of the game.
    • Art: All visual-related items.
    • Interface: all first plane user communication items.
    • Music and Sound Effects: All sound-related items.
  • The complete article can be read here

How to Report

 There are many software in which bugs can be tracked and reported, from a self-made google sheet to most complete software such as Bugzilla or Mantis. The main advantage that I have detected in using these tools is that they ease the communication with the rest of the team by sending emails when an issue is assigned to them and when specific actions are made on that bug issue.

More important than what software to use is what we should say about each bug. While some fields of a report may vary in each company or project, there are some other fields that we should always strive to keep in our reports so that the team for which the report is directed receives as much information as possible without having to write a long text per each issue.

For example purposes we will be reporting a game crash in the pause menu in level 3 o an imaginary game. A game crash means that the game has become unresponsive and has frozen, stopping further gameplay to occur normally.

Summary: this is the news headline, we need to give as much information as possible in just a few words, it is different to say: "There was a baseball game last night" than to say "The Yankees beat Red Sox 4-2" In our game crashing bug, we should report: Game doesn't load in Internet Explorer.

Description: A 1 paragraph description of the bug. We should only describe what is happening. For our game crashing bug: When trying to load game in Internet Explorer 11.0.1 in Windows 8.1 the game is stuck in the loading screen at about 50%. Nothing else happens. Browser doesn't crash.

How to Reproduce: A step-by-step guide about how to reproduce the bug, we should be careful about the level of obviousness that we handle in this piece of information, we don't want to be too obvious or too careless. If the programmers needs to enter the pause screen in level 3 and press 2 simultaneous buttons in the controller to activate the crash, we should make emphasis in the bit corresponding the pause menu in level 3. So for example:
    Step 1: Open the game normally.
    Step 2: Play normally until level 3 is reached.
    Step 3: Midway through level 3, press Pause to Pause the Game.
    Step 4: Press buttons X and A at the same time and observe. Game crashes.

Additional Information: Any additional information should be given at this point, browsers, controllers, specific characters that trigger the bugs, and so on. For our game crashing bug, we could, for example, write: See references (2 attached images of the state before and after the crash), happens with both Xbox One and PS4 controllers in both Xbox One and PS4.

Expected Behavior: When a bug happens is because something is not behaving as it should, and if the tester detects it is because he has, consciously or not, compared the game against an improved version of the game that exists in the tester head. This field is to be filled with the behavior of this improved version. In our pause crashing bug, the expected behavior is: Game should not crash in pause.

Platform: This field should be filled with all the information of the platforms in which the bug happens, browser versions, operating systems, consoles, mobile devices, browsers in mobile devices, specific controls, specific brand of keyboard and devices, and so on.

Assignee: This field should be filled with the name of the team member who is responsible of the solution of this bug at the moment, for example if it is an art related bug, the immediate assignee is the artist, but when the artist has done his job as expected, the assignee should change to the programmer who will implement this change.

ID: A unique identification number.

Attachments: All the images, videos, screenshots, sounds or any other related media files that support the bug description and that are mentioned in the "Additional Information" field. The naming of all these should be obvious.

Frequency: This refers to the reproducibility of the bug, a crash may happen all the times, some times, 50% of the time, only once or the tester can simply not know. Either way, this field should be filled with the frequency of the reproducibility. For small games, it is ok to test 10 times to measure this frequency. For bigger games, 100 times is recommended.

Severity: This field refers to the importance that this bug represents to the end user, for instance a crash that is reproduced 50% of the times or more has a very high severity, but a small pixelation in a button has a low severity.

Priority: The priority represents the immediate importance of the bug fix. It is important not to confuse it with severity, a grammar bug has a high severity always, but it doesn't have a high priority in the pre-Alpha and Alpha phases, in which final text are yet to be defined and most of the game functionality is being implemented.  

Testing Methods 

We now know about what to test and how to report, it is time to learn a bit about how to test. A clean testing method will give us more results in less time. It is very easy to lose sight of bugs in big games, or when testing for hours in one sitting, and one person can only see so much. The following methods serve as guidelines to test each of the categories. 

Smoke Testing

It is the first test that should be done and it is the first test that is usually done, consciously or not. Smoke testing is in many cases automatized, but some manual smoke testing is always needed.

When smoke testing, the tester needs to verify that the game is not crashing, the screen flow is correct and that no crashes occur. Smoke testing can block the rest of the testing, and this is why it must be done first, even if it is a fast run of the game, if smoke testing is not properly done we can encounter, for instance, a crash in level 1 will logically bock the testing of further level.

Even if we have tested the game 100 times and it is ready to deliver, we should always smoke test to make sure that everything is running as it should be. 

Soak Testing 

Soak Testing is very simple to do and it can detect problems in saving data and memory leak. In order to soak test, we should leave the game running in a specific device, platform or console for long periods of time in a critical screen, for instance in the pause screen for 8 hours, and go back to it to observe the game behavior, if the game performance is slow, if the level continues where it has left off, with the same amount of coins, lives, ammo, if the enemies are spawning correctly and so on.

Specifically, if the game performance as been affected, it could be a sign of memory leak, this means that some trash data is not being properly destroyed and it is being constantly stored in the device, the longer the game is running, the more amount of data that it is stored in the device. 

Play it Right 

This means to play the game as the game developers intend the players to play, if the game is a platformer scrolling from right to left (meaning that the player moves left to right), then we move left to right. If the player is supposed to press only one button at a time, then the tester presses only one button at a time. 

Play it Wrong 

In Play it Wrong, the tester job is to play the game in a completely opposite way to what was designed, if it was designed so that the character moves from left to right, we should move from right to left, we should press all the buttons at the same time, try to go through walls or jump a hundred times in a row. 

Sometimes I like to pretend that I'm a hyperactive toddler fiddling with controller to Play the Game Wrong, this has given me amazing results in finding bugs that are difficult to find but common to reproduce in gameplay. 

Upcoming in the Next #TestingTuesday

Now that we know how to report and how to test, we will go deeper into each of the testing categories.

 
In the next #TestdayTuesday we will be seeing what does gameplay mean, what does it mean to test gameplay and how it is related to the player's psychology.

No hay comentarios:

Publicar un comentario