How Many Testers Do You Need for an iOS App?

Quick answer: how many testers you need depends on the job. For usability feedback, about five testers per round is enough. Run three small rounds and fix the problems between them. This is better than one big round.

For beta testing, the goal is to find crashes and bugs that occur only on some devices. For this job, get as many testers on as many different devices as you can. If you need real numbers, such as a task success rate for a report, Nielsen Norman Group recommends about 40 participants.

These are three different questions, and each one has a different answer. If you mix them, you make one of two mistakes. You test with two friends and stop, or you wait months for 50 TestFlight testers who never send feedback.

This guide shows where each number comes from and how to plan your rounds.

Where the "five users" rule comes from

In 2000, Jakob Nielsen of Nielsen Norman Group (NN/g) published Why You Only Need to Test with 5 Users. The article uses a model by Nielsen and Tom Landauer. In this model, each new test user finds a share of the usability problems in a design. Across many projects, this share was 31% on average.

The curve becomes flat quickly. The first person you watch shows you almost a third of the problems. The second person repeats much of what the first person did and shows a few new problems.

By the fifth person, you have seen about 85% of the problems. After that, each extra person mostly confirms what you already know. Each extra session teaches you less.

This is about qualitative testing. You watch people try to do tasks in your app, and you write down where they have problems. You do not measure anything. You collect problems to fix.

Why three rounds of five beat one round of fifteen

The article does not say "five people and you are done." The same curve shows that you need at least 15 users to find all the problems. Nielsen's advice is to divide those 15 users into three studies of five, and to fix the design between them.

There are good reasons for this:

• Fixes need checking. Your fix for a confusing onboarding screen can cause a new problem. The next round shows you if the fix works.
• Big problems hide small ones. If nobody can complete your sign-up form, you cannot learn about the problems on the next screen.
• Fifteen people with the same problem teach you nothing new. After the third person fails at the same step, you already know what to fix.

Nielsen repeated this advice in a 2012 follow-up. If you have a bigger budget, spend it on more studies, not on more users in each study.

When five testers are not enough

Your app has distinct user groups

Five testers in total are too few if very different people use different parts of your app. For example, a marketplace has buyers and sellers, and a school app has teachers and students. Each of these apps is two apps in one.

The 2000 article suggests 3 to 4 users from each user group when you have two user groups. When you have three or more user groups, it suggests 3 users from each one.

You need numbers, not a list of problems

Quantitative studies measure things. Examples are task success rate, time on task, or which of two designs performs better. Five people cannot give you a reliable number.

The advice from NN/g about this changed over time. A 2006 article said 20 users. The current recommendation, from 2021, is 40 participants for most quantitative studies.

The authors say that NN/g gave different numbers over the years. Most indie apps do not need this type of study before launch.

The problems you care about are rare

The 31% figure is an average from projects in the early 1990s. A 2021 NN/g article explains that some problems affect only a part of your users. To see these problems, you need more people.

You need 9 users to find 85% of the problems that each affect 20% of users. When each problem affects 10% of users, you need 18 users.

A bug that occurs only on one older iPhone model is the same type of rare problem. Five usability sessions will probably not find it.

Usability feedback and beta testing are two different jobs

When developers ask "how many beta testers do I need," they usually ask about two things at the same time. When you separate the two things, the numbers become clear.

For usability feedback, more people mostly give you more of the same findings. For beta testing, more people give you more devices, more iOS versions and more unusual real-world conditions. So for beta testing, the number of testers is important.

How many TestFlight testers? TestFlight lets you add up to 100 internal testers from your team. You can also invite up to 10,000 external testers. The first build that you share with external testers must pass TestFlight App Review.

As an indie developer, you will not get near these limits, and you do not need to. A few dozen active testers on different devices are usually more useful than hundreds who install the app once and never open it again.

TestFlight testers also share crash reports with you automatically. So testers who send no feedback still help you with crashes. Our guide on how to find beta testers for an iOS app tells you where to find them.

A simple test plan for an indie developer

Here is a plan that fits a normal release cycle:

1. Choose two or three tasks. For example: complete onboarding, create the first item, and find the setting that people ask about most. Write the tasks as short instructions for testers.
2. Round 1: five sessions. Watch all five sessions before you change anything. For each problem, write down how many of the five people had it.
3. Fix the top problems. Start with the problems that stopped people from finishing a task. Make small improvements later.
4. Round 2: five new people. Do not use the testers from round 1 again. They already know the app, so they cannot show you if a new person still has problems. Check that your fixes work. Then look for the problems that the earlier problems hid.
5. Round 3: five more. If this round finds no serious problems, stop and release the app. If it still finds problems that stop people, fix them and run one more round.

Before a major release, run the beta test for crashes and device coverage at the same time as these rounds. The two jobs do not compete. One job does not replace the other.

What this plan costs on Raizy

Raizy is a free feedback exchange for indie iOS developers. When your app is live on the App Store, you can request feedback sessions.

In each feedback session, another indie developer installs your app on their own iPhone and uses it. Then the tester uploads a short screen recording or a screenshot and writes notes.

You can add instructions for testers of up to 500 characters. That is enough to list your two or three tasks.

Each feedback session costs 1 credit. You earn 1 credit for each session that you give and that a moderator approves. New accounts start with 0 credits.

So for a round of five sessions, you first test five other apps and get those sessions approved. Three rounds mean 15 sessions given and 15 sessions received.

There are no fees and no subscriptions. You can pause a request between rounds while you work on fixes.

Know what this type of round can do and what it cannot do:

• The testers are other indie iOS developers. They are not always your target users. They are good at finding confusing flows, unclear text and bugs. They cannot tell you if your audience wants the product.
• Recordings are silent. There is no think-aloud voice track. You learn from what happens on the screen and from the written notes.
• You do not see who the tester is. So you cannot make sure that each round has new people.
• Testers can test only apps that are live on the App Store. For a pre-release build, use TestFlight.
• You cannot choose testers by device or country. So Raizy is not a tool for device coverage. Use your beta test for that.

See usability testing for iOS apps on Raizy to learn how a feedback session works.

Common questions