Beta Testing Discussion

[Tahazzar]

This is a far too common misconception that has no validity in the reality of iterative games development.

If we switch to software development terms isn't this just waterfall vs agile? And as far as I know, agile is universally known as just better. Rare are the cases where pure waterfall could be recommended instead of a more agile method.

Now I have a bit of a gripe with the large number of "play testers".

Hopefully I'm not just fanning up the flames again, but...

See this article for example.

They seem to indicate that, yes, at the beginning you only need around three to five people for playtesting. However, closer to release (ie. beta), it's pretty much "more is better" - especially when testers can be acquired for free (like in these forums). That, and probably bunch of those testers fail one way or another to provide good feedback (are busy, lazy, or whatever). Even if you had "too many" you could still just ignore the "rest" if you felt you had gained enough information for the next iteration / release.

[Grimlock]

It's a good read, thanks.

I do see that I've overlooked one aspect of good play testing.  In order to remedy that I'm going to have to get all of the volunteers to fork out their home addresses so I can get all creepy and silently hover over them while they play the game:

"Sit next to the person playtesting your game

Ask them what they think of each level or game element as it appears

Observe (do NOT talk to the player - let them play and don't interfere) and record the issues you find, as well as any important factors (e.g. play length, number of deaths, frustration and emotional response)

Do this with a few people, then fix the issues you think you need to.
Repeat"

[Quietus]

* Quietus prepares a Most Dedicated ROM Hacker award. :cool:

[Tahazzar]

I do see that I've overlooked one aspect of good play testing.  In order to remedy that I'm going to have to get all of the volunteers to fork out their home addresses so I can get all creepy and silently hover over them while they play the game:

:lol:

I mean, yeah, that would be the "optimal way" to conduct it, but understandably it's rather hard in the context of these forums.

This is why I think a bunch of (potential) playtesters would be recommended in this case since I would assume that the absolute best you can hope for is some video (maybe extensive notes if you lucky), that you then have to try to interpret. Ie. "Why is this baboon trying to climb that wall for ten minutes??"

Although I have to admit I have no experience in this area... :<_<:

[Thirteen1355]

Right now, I and a bunch of other people are beta testing a pretty famous Mario 64 hack (Last Impact).
There are about fifteen beta testers in general. Three never did anything. I did the first two Worlds and found most bugs in these worlds, but due to lack of time, other people are progressing much quicker, so I now just test random places in the game (making them all accessible with a cheat).

Three people beat the game 100% already, and the creator is convinced he will be releasing the final version of his hack soon.

I don't know, but I get the idea not many beta testers are needed.

[Quietus]

I'm generally of the opinion that quality over quantity is the way to go about beta testing.

[Thirteen1355]

In beta testing, quantity IS quality. More testers won't make good testers less good.

[Quietus]

I disagree. I'd rather have three dedicated bug hunters playing thoroughly than twenty people blitzing through the game, and passing on one or two bugs.

[Thirteen1355]

And what's the connection between having twenty people and them blitzing through the game?
What about twenty bug testers playing thoroughly?

There's absolutely no connection between having many testers and them suddenly becoming bad.

[Quietus]

No, but you also have no guarantees that any of them will be any good. I'm saying that I'd rather pick three people known to me, and know that they would do a thorough job, than throw it open to everybody, and hope that some would do a thorough job.

[Thirteen1355]

I think you need as many different people as possible, for different experiences.
It's not only about bug hunting, it's also about testing difficulty.

If you would let your three speedrun-loving friends test the game, they will experience the difficulty differently from people who only played Zero Mission.

And again, why not three friends you know they would do a good job, along with 17 other people?

[Quietus]

Because to me, beta testing would be targeted toward bug hunting. Once the bugs are mostly ironed out, I'd move to a public release, and adjust the difficulty from the feedback given. Many hacks have done this before. A great recent example would be Redesign: Axeil Edition, which was shredded by public outcry over its insane Metroids, and was rapidly changed (more than once, I think).

speedrun-loving friends

I don't see how speedrunning is relevant.

[Thirteen1355]

It was just an example.

If you're part of a community, and you let only your Metroidconstruction admin friends test the game, which all have beaten NESTroid about 45 times, you will experience that they all think Rogue Dawn is easy.

But if you let 20 people, who've only played one Metroid game, play Rogue Dawn, they might say they had a hard time beating it.

[Quietus]

Of course. I would think that much would be obvious, but for bug hunting, I'd want those most versed in the games glitches and hacking pitfalls.

As I mentioned, difficulty would most certainly be flagged upon release, whether it be too hard or too easy, and in this case, the more opinions, the better, as it'd give you a better average.

[Thirteen1355]

Exactly.

[snarfblam]

I do see that I've overlooked one aspect of good play testing...

I mean, yeah, that would be the "optimal way" to conduct it, but understandably it's rather hard in the context of these forums.

An internet equivalent might be having a few testers (privately) stream their gameplay.

[Thirteen1355]

I mostly record my test runs.

[AuroraxPhilic]

Holy shit. I should be taking notes here. This is video game philosophy thinking at it's finest up in this piece! XD

Given enough thought, I've come down to a bit of a compromise. Have certain playtesters try the break the game, and use another group, who don't know anything about your game, and see how they manage (RD is take a similar approach in a way).
When Super Metroid was play tested, ordinary people who never had much experience with video games playtested it, and look what came of it. Definitely, it is irrefutable that you need to have some people who don't know what they're doing play your game. This is your super casual audience of course, and could make a game very polished and flow better just by indeed seeing people play. (#ThoughtBoner)

[Zero One]

Went ahead and split this into its own thread, because it seems to be pretty able to carry a separate discussion, not necessarily only for Rogue Dawn.

[Quietus]

One thing that I am curious about is how popular a game has to become, or how big a game has to be, before bugs are jokingly considered 'features'. A good example of what I mean would be Skyrim. Now, a load of the bugs in this game would have people up in arms if it were another game, but with Skyrim, players seemed much more willing to laugh it off, and accept it as-is. I'm curious as to when that line is crossed, or whether it's the size of the game that makes players think "well, they'd never be able to fix them all." :^_^:

[Zero One]

There's definitely a point where a bug becomes a feature. As an example, strafe-jumping in Quake. It's absolutely a bug in the original Quake engine, but it gained such popularity with the fanbase that as of Quake II, it was a feature, and in Quake Live, is a taught technique.


Though with Bethesda, I think it's every game since Morrowind having some daft bug that they just don't fix. Bethesda's engines are pretty garbage. Fallout 4 completely falls apart if your framerate exceed 100FPS. Quite frankly, Fallout 4 falls apart regardless of framerate.

[Quietus]

Fallapart 4, from Bettheresda Softlocks