Disclaimer #1. Nothing new here. Just relaying the audio notes from the recent Pantheon Rise of the Fallen VIP Roundtable with programmers Kyle and Rob.
Disclaimer #2. This blog is neither associated with, nor employed by, Visionary Realms. I am a fan!
Disclaimer #3. This is not word for word what was said in the video. I tried to sum it up and include the most important parts in a concise way.
00:49 - When you find a bug in Pantheon how do you view severity of an issues verses priority of features when it comes to testing Pantheon Builds? How does your input let the devs know what to focus on? Mark - We typically focus on the scope of what's being testing unless it's game breaking. Priority of features is usually like a deep scan rather than a targeted hunt.
01:37 - How do you determine what needs to be in the test plan of a build? What kind of factors are there in determining pass/fail in Pantheon? Kim - We check with the devs to see what's being handed over soon and base our build on that. Pass and fail depends on the issue. Some issues are bugs and some things are feedback from our testers.
03:28 - How are you assuring the quality of testing with such a limited number of testers? Kim - We are a tight nit group of developers and we do a lot of internal testing.
05:03 - Are you capable of forming full groups on a consistent basis for testing? Kim - Yes we test technical issues and also get feedback from group play.
06:01 - Do you test from different group makeups with different skills from different angles? How often? Kim - Constantly. We approach every game system and quest from the mindset of 80 different people. Mark - Our team has a lot of good diversity and different play styles. Kilsin - We have a lot of really smart testers.
08:48 - Do you use a formal system to do QA or do you just play the game? Kim - It's some of both. We have formal methods such as smoke tests but we also just form static groups and play the game without GM tools. Mark - Sometimes it's focused. Sometimes it's more organic.
10:35 - Do you get bored doing the same thing over and over again? Kim - After the 200th time I've done the same quest .... YES.
11:25 - When PA testers discover issues or bugs what information are you looking for to fix the problem? In short how can we make your job easier? Kim - A step by step set of instructions on how to reproduce the bug is the best. Also when you submit a bug our system captures some data including your location and also what you are targeting. Keep that in mind as you submit bugs. Be as detailed as possible. Mark - It's all about reproducing the issue. Screenshots also help.
14:32 - How frequently do you communicate with the rest of the team when you are testing? Kim - Constant communication with creative and programming.
15:45 - What's the process for when you find something wrong or broken? Mark - Game breaking we address right away. Minor things we submit a ticket and the devs get to it when they get to it.
16:30 - What is your favorite or most memorable bug that you've found while testing Pantheon? Mark - I always get a kick out of the weird animations or graphic glitches that happen. Kim - My favorite is when we grouped in BRK and there was a glitch where when we attacked mobs everything would disappear except for their hair.
18:51 - How often are you involved in pre alpha tests? Kim - I'm usually there for the PA Tests. Sometimes I group up with people as a GM or incognito. Mark - The PA sessions are my most enjoyable time. Nothing beats people enjoying the game.
19:59 - Kilsin have you ever been a play tester? Kilsin - Not officially. But I do it on the side.
20:43 - Who play tested the infernal chicken in the starting area? Mark - Sometimes the devs sneak stuff in without testing it. Kim - I did talk to Joppa about it and he liked it and said we should see more of it.
22:48 - What does QA do? Kim - That's a huge question. We test every single system, zone, class, ability, and everything else in the game. We test for bugs and give feedback. Mark - PA ticket investigation. Organic testing and focused testing. Any new content.
25:54 - What is a pre alpha tester verses a QA tester? Kim - QA testers have the insider information such as the game formulas. We get very technical in that regard. PA testers are more general and asked to give feedback on how the game feels. Mark - The PA testers really help us as we use them as a compass to make sure we are close to what the community needs.
28:19 - What should pre alpha testers be focused on? Mark - Joppa does a good job of telling everyone the scope of each test session. Take the time to read through what he says each test session.
29:19 - What parts of the game need more focused testing? Kim - That is really dictated by Joppa's team and Kyle's team. Whatever they need.
30:18 - What parts of the game have been thoroughly tested and have minimal issues? Kim - The builds are in a really good place. Mark - It changes per day.
32:55 - As a QA testers how far in advance of the PA VIP tester sessions do you test that content? Kim - We test PA stuff right before PA.
33:54 - How are the PA testers doing in providing feedback? Mark - All the feedback we get is valuable. Multiple accounts of the same issue are super valuable.
35:08 - Triple A games often have hundreds of QA testers. At launch how many do you think Pantheon will have? Mark - 3 (joke) Kim - 3 counting Kilsin ... we will scale as needed.
37:01 - Can you as a QA tester confirm if the code is good or bad? If so how does that feel compared to other games you have QA tested? Kim - I think I can tell when the code is bad ... even gamers can tell when games bad coding. Pantheon seems very stable and runs amazing. Mark - The coding that Pantheon has is out of this world. I tested for Vanguard and saw many issues with it. I don't see those problem with Pantheon.
42:17 - Can you talk about games you have QA tested in the past? How does that compare with testing Pantheon? Mark - I did a lot of alphas and betas. I worked with the devs in Vanguard. Kim - I've done regression testing before (not video games). I've also done a lot of beta and alpha testing.
44:31 - What's the funniest issue you have found? Have you made a gag real? Kim - I do have a folder of funny pictures.
45:41 - Where do you want to go from QA? Other areas in the gaming world that interest you? Kim - I'm very rooted in QA. Mark - I really enjoy testing.
47:27 - Can you describe how it's different to test for bugs verses testing for what is fun? Kim - We have different types of testing. If we are focused on something then it's very narrow and we are focused on that. We also test everything during play sessions making sure it's fun.
50:20 - Do you mostly work with the programming team or the design team? Mark - It's pretty equal.
51:00 - What kind of tools do you use to communicate and follow up on issues? Kim - Chat systems and also a ticket system that is connected to the in game bug reporter.
53:21 - What would you describe as a perfect bug report? Mark - Reproduction steps with a screenshot. Kim - walks through an actual ticket creation.
57:13 - Kilsin what MMORPG did you play most as a fan? Kilsin - Old School Vanguard: Sage of Heroes. I played from beta till it was shut down. Currently playing Elder Scrolls.
57:46 - How are Kim and Mark liking the barbarian testing? (joke?) Kim - I think I know who asked this question ... (no answer)
58:09 - Does knowing what to expect make it more difficult to test bugs? Kim - No it doesn't. You have to look at things from different perspectives constantly. Mark - The scope helps us focus on what we need to.
59:54 - What games do you enjoy playing in your downtime? Mark - I'm a longtime EQ Player, FF, and Swords of Legends Online. Kim - I don't have a lot of time to play games. When I do Diable 3 and also Dark Alliance.
01:01:11 - How are you helping each class have it's own structure? Mark - I love class identity. I try to use the design of the dev to test things. Kim - I don't want to speak for design, but I think we are for sure going for unique features for every class.
01:04:41 - Has working from home been a challenge? Mark - Kim and I are in the same zone so it's not an issue. Kim - There is a lot of overlap in times. We make it work.
01:06:11 - How closely do you work with the world builders and specifically the dungeon designers? Mark - It's pretty standard. The same as everything else. We try and break it.
01:07:28 - When testing encounters how often do you have to bug Saicred? How do you communicate with him? Kim - All the time but sometimes he puts me on mute. Mark - Saicred gives us a target.
01:08:30 - How often does fixing one bug lead to fixing bugs in other places? Mark - All the time. 6 degrees of separation :).
01:09:11 - Have their been tools specifically built for QA to help with testing? Kim - I have programming create new commands and tools for us all the time.
01:10:00 - Has Mark found any fun Monk bugs like he did in Vanguard? Mark - Nothing to that scope yet.
01:10:24 - Minus wants to know who tested the power of the majong and pyramid spiders? Mark - It's safe to blame Saicred in all these situations.
01:11:00 - Which team illicit the most eye rolling - coding or design? Kim - I like to think they all love all our feedback. I'm sure they all roll their eyes at me.
(PA Testers when VR Puts out the next PA Date)