In the last year, the User Research team at Sprung Studios faced a significant barrier to typical research. The pandemic made us question how we could support both our internal teams and external partners effectively in this new context.
Under normal circumstances, we would be running regular Usability Tests with live participants. Yet health concerns made in-person studies unfeasible and remote testing posed too high a risk to confidentiality. Due to the restrictions, we leaned on our methodological roots by establishing and iterating on a research method known as the Expert Review.
This method now sees regular use at Sprung, and researchers are available to conduct Expert Reviews on a multitude of games at any given moment. That game could even be yours.
Why Expert Reviews?
The goal of an Expert Review is to assess a game from the perspective of the player. Researchers at Sprung do so by applying research and evidence derived from the fields of Cognitive Psychology and Neuroscience.
It’s a frequently used method in Games User Research that bears resemblance to another method known as the Heuristic Evaluation. Yet, rather than a simple checklist, the Expert Review at Sprung is an in-house workflow that applies the trained eye to your game.
Process
Expert Reviews at Sprung are conducted using a process aimed to emulate the player’s state of mind. The process revolves around three key aspects:
Player Goals
-
Often specific to the game itself, these require a high degree of familiarity with the game being tested and the associated genre conventions.
Cognition
-
This aspect concerns the brain functions at play when we perceive and interpret new information.
-
In other words, researchers view the game as it relates to attention, perception, and memory.
Context
-
Understanding what the game developers are striving for is paramount to properly contextualizing the findings.
These three aspects form an iterative feedback loop, in which each aspect informs the others. This process not only ensures that interactions are implemented well, it also can also help the project stick to its original vision.
Personnel
At Sprung, Expert Reviews are always conducted by two researchers for two main reasons:
-
Two researchers will find more issues than a single person could
-
Both researchers can check each other’s biases
Having multiple researchers work in tandem introduces rigor and reliability. We are emulating the peer review process, minimizing the subjectivity of the findings. Not only does this ensure the quality of the report, two researchers are better able to prioritize the findings and draw attention to what is most important.
Advantages
1. Expert Reviews can uncover the highest priority issues for less cost
Whether conducted as a standalone method or the first round of research before a Usability Test, the Expert Review can save developers money by calling attention to obvious issues. In the latter scenario, the developer gets more value out of a Usability Test when it is conducted.
2. Expert Reviews can be conducted earlier in the development cycle
Even when the game is in a conceptual or grey box state, an Expert Review has a place in the iterative design process. Researchers are trained to focus on what’s most important, while participants may not be able to ignore the bugs or incomplete sections of an early build.
3. Researchers are guaranteed to see more of a game build than any participant
Whereas most participants only get a single session with a game, researchers get to sit with the build for longer. This means that an Expert Review is guaranteed to uncover a wider breadth of issues.
4. Expert Reviews can save developers from unexpected costs
Resolving issues becomes exponentially more costly as development progresses. Identifying and patching these early on can help developers avoid costlier solutions later in development.
Misconceptions
Expert Reviews are the opinions of the researchers
-
Researchers conduct Expert Reviews by following a process based on cognitive science principles.
-
Conducting research means setting opinions aside and advocating for the player.
Anyone can conduct an Expert Review
-
User research might be an interdisciplinary field, but that does not mean just about anyone can conduct an effective Expert Review.
-
To uncover issues and translate them into actionable findings takes time and training.
-
Someone working closely on a project is more likely to be blind to its faults, due to their unparalleled familiarity.
Expert Reviews are of less value than Usability Tests
-
As a standalone method, the Expert Review is capable of finding high-priority issues and more.
-
The Expert Review can and should also be used to inform the scope of a subsequent Usability Test when possible or appropriate. In either capacity, they are just as valuable as Usability Tests.
Types of Expert Reviews
At Sprung, Expert Reviews can come in several different forms. We apply the same process to all, even if each one requires a different lens. A few examples include:
Usability Expert Review
This type of Review is often the first filter for any project. The Usability Expert Review can help any game developer align their project’s goals with its features. It is recommended to carry these out as early as possible, so obvious issues get addressed before they are committed to code.
First Time User Experience (FTUE) Expert Review
Onboarding new players and teaching the game’s core rules takes effort and expertise. That makes tutorials a prime candidate for iteration. As with a Usability Expert Review, conducting research on the FTUE is best done as early as possible. The FTUE Expert Review could either be conducted on early documentation or a first playable tutorial.
Accessibility Expert Reviews
This type of review can help any developer fill this gap in expertise if it exists at their studio. Researchers at Sprung Studios are trained to identify and draw conclusions from potential accessibility barriers, first and foremost. Any single barrier could affect many people from a range of disabilities, such as those related to cognition, hearing, visual perception, mobility, or speech.
Final word
When game developers work on a project for weeks or months on end, they gain an unparalleled familiarity with the product. In turn, this can blind them to the average player’s experience.
Bringing in trained researchers to conduct Expert Reviews is one way to uncover biases and root them out well before it is too late to make changes. At Sprung Studios, the User Research team is available to do just that. We partner with game studios to advocate for the player and take the guesswork out of game design.
For more information about our services, please contact: