Photo by Justin Marty via Creative Commons

Both Mike and I have run beta programs before, but never for a two person effort. In the past, my typical engineering team has been around ten software engineers and ten QA engineers for about 15 months, but I’ve been responsible for teams with as many as 25 software engineers + 25 QA engineers for 2+ years, and as small as 2 engineers + 1 QA.

For Note*spark, we had a team of two engineers + 0 dedicated QA engineers, which is similar to the smallest sized team I’ve ever worked with. But instead of taking a year to develop, we sprinted the whole way, working long hours for about two months to finish the server, web and iPhone pieces.

Once we were done with feature development, we almost decided not to do a beta at all, on the theory that two months of engineering was small enough for us for us to QA ourselves. And besides, we had written a bunch of unit tests, so there probably weren’t that many bugs left, right? Boy, were we wrong! By running a “real” beta, we found a ton of bugs that we never would have found ourselves.

Along the way, we made some decisions about how we thought we should run a small beta. We also learned some things specific to running betas for the iPhone. None of this is rocket science, but I thought I’d post what we learned in case it’s helpful to anyone.

~

How we found beta users

Our first issue was how to get beta testers. We started by asking friends and family. Some of them were active beta testers, and some weren’t, which is to be expected. After this, we had to go searching for more beta testers.

Because our particular app was a general purpose app, we just needed people who had some interest in taking notes. So we could cast our net widely.

By far, our #1 source of beta testers was through a posting I did on my personal blog, which had been mostly focused on technology stuff. Even though I haven’t posted regularly in a while, there were a good-sized group of people who were still subscribed who already had some connection with me. And because I’d been posting about tech, these people had some amount of interest in the stuff I did for work.

So if you have an existing blog, that’s a great place to start.

Our #2 source of beta testers was similar: facebook. I let people know that I had an iPhone app to test, and some curious long-lost friends contacted me to sign up. I’d imagine you would get a similar response through twitter, or other means of communicating with friends.

Our #3 source of beta testers came from some postings we did on a few iPhone forums. We could probably have gotten more people this way if we’d pushed harder, but we didn’t want to be annoying. Also, in general, I think most people don’t want to randomly test stuff for random people they don’t know.

In the end, we ended up with around 50 beta testers which is what we were shooting for.

Qualifying beta users

In doing a small scale beta, we had to make some decisions on what to do formally and what to do informally. We decided to do a formal signup questionnaire on the theory that it would help us ensure we were getting adequate test coverage across platforms, browsers, etc. We also used it to calibrate people’s interest in the product going into the process. Betas can help you do market validation along the way, and it was useful for us to know who came into the process being interested in a note taking app in the first place.

We used wufoo’s free form service to do our questionnaire. It’s easy and it’s free. You can see the questionnaire here.

How we track bugs

We decided not to use a “real” bugbase for now. All of our bugs are stored in a Google Docs spreadsheet, which lets us both access the same info without doing anything complicated. We are, at least, being careful to record things like bug IDs, statuses, etc., so we can eventually move to a real bugbase if needed. But given that we are just two people, it didn’t seem worth it.

The main reason was set-up time. If we had found a free (or very cheap) hosted solution, we probably would have gone with that, but we didn’t want to take the time to install Bugzilla. If anyone knows of a quick and easy solution for bug tracking, I’d love to hear about it.

What we learned

Like I said above, we found a ton of bugs we wouldn’t otherwise have found:

1. Things that beta users found confusing that would never have occured to us
2. Bugs based on beta testers using the product in ways that we hadn’t anticipated
3. General, bone-headed bugs that we hadn’t caught for some reason

In the end, we spent two months in the beta cycle after thinking that “we didn’t need a beta”. Ha! It proved once again a rule of thumb, which is that the closedown period usually lasts about as long as the development period. During this time, the beta testers found roughly as many bugs as our in-house testing. Some of these bugs were critically important bugs that I don’t know if we would ever have found. By the way, this rule of thumb obviously doesn’t apply if you are doing periodic lockdowns and using an “always shippable” engineering process. But we weren’t doing that. We were just coding as fast as we could.

We also proved another rule of thumb, which is that 20% of the beta testers find 80% of the bugs.

Finally, we found that continually adding new beta testers was the key to finding fresh bugs. When you are testing a small app, as all iPhone apps are, you don’t need someone to spend 100+ hours on the product to find the deep bugs. Most of our beta testers found the bulk of their bugs within the first few days, and we found that by adding new beta testers along the way, we could keep the inflow of bugs coming at a good pace.

If you have any other tips on running small betas, I’d love to hear them. Meanwhile, our beta is alllllmost done. (crossing fingers)