After some talks with the project managers and analytics team I set up to create a simple playtesting process that our games could undergo whilst on the building stage, so we could have at least a couple of build iterations tested by real users and provide accurate feedback to the developing studio.
The process was simple:
1- 10 users recorded their screen and voice when playing the game for at least 15 minutes, talking out loud about everything they were experiencing.
2- I processed these recordings, listing on a spreadsheet all the relevant observations for each video.
3- Then I grouped similar observations and categorise them into mechanics, UI, economy, narrative, general feel and bugs. All bugs were immediately addressed.
4- Together with the product manager we looked at the observations and established a priority order based on the number of people raising it and overall impact on the experience.
5- The 5 more important observations were reported to the developing studio along with suggestions on how to improve them for the next iteration.
6- Once the next iteration was ready, start another playtesting round paying special attention on the newly made improvements.