The Crowd Sourced Testing Difference

When the founders of crowd-source testing pioneer Applause set out to start a new company, they didn’t want to simply provide a cheaper or faster form of testing. If they were going to invest the time and energy into this venture, they wanted to provide something fundamentally different, something previously unavailable. Something unique.

There are two basic problems with traditional testing. First, testing is based on what you expect the end user to do with your product. Every test plan ever written is focused on the anticipated use case. But if there is one thing the explosion in mobile adoption has shown, customers hardly ever do what you think.

Second, when you test in the lab, it is virtually impossible to recreate all of the scenarios and conditions your users will encounter “in the wild”. Simply trying to build out a lab with a small subset of the various phones, mobile devices and web browsers on the market today can be exceedingly expensive. But that doesn’t even start to address the complexities of networks, signal interactions, intermittent signals or geographic and international conditions.

In recent years, companies like Google and Facebook have famously embraced rapid release cycles and have pioneered leaving it to their customers to do their testing. By releasing rapidly and actively listening to their fanatically loyal users, they benefit from real-world feedback and have been able to deliver products that delight their users.

But this approach is not entirely viable for companies in industries like banking, insurance and healthcare. These apps need to work…and increasingly, “delight”…before they are released. Erroneous balances, faulty logins and inaccurate medical advice have serious consequences and real liabilities.


The basic idea behind crowd sourced testing is to leverage a diverse community of freelance testers to execute tests on an almost on-demand basis. Much like other crowd source models, this allows organizations to tap into a potentially otherwise idle workforce while providing workers with the flexibility to work where and when it is convenient.

Multiple consumption and compensation models have emerged for crowd sourced testing. Some are completely self-service, allowing you to post opportunities to a portal where individual participants bid on opportunities. Others are more “white glove”, managing the tester recruitment, vetting, ranking and educating testers, compensation and actual triage of uncovered issues.

Another benefit of “the crowd” is that they come with a real world variety of devices. Simulators and device clouds are nice, but they can not effectively anticipate the complexities introduced by waning battery life, conflicting software, erratic Wi-Fi signals and other side-effects of reality.

And ideally, you want real, professional testers who know how to follow instructions, know where to look for relevant bugs and know how to effectively document their findings.


Given the variety of solutions now available, here are some things you should consider when considering crowdsourced testing:

Managed Service – crowd sourced solutions, whether for testing or other services, are offered in a variety of access methods. Systems like Amazon’s Mechanical Turk are extremely self-service. It simply provides access to a pool of individual resources interested in bidding on tasks. To use them for testing, you would need to assemble a team, vet the testers, verify their credentials, etc.

Other services manage all of this for you, provide team leads as part of the offering and are managed by a professional project manager. Clearly, these additional services can come with a price tag, but if you are looking to actually take work OFF your plate, they are definitely worth investigating.

Multi-disciplined – testing often includes tasks beyond functional / regression testing. Does the vendor offer exploratory testing? What about usability? Accessibility? Localization? Security? Can they help you actually create test cases? Will they triage the bugs before they get to you? Deduplicate them?

Vetted testers – who are these testers anyway? Can they follow directions? Can they be trusted? How long have they been working for the vendor? Can I interact with them directly? Can I ask for them to stick around for my next test?

Unlimited access – when people see dollar signs associated with an individual test, they associate a cost with the activity and consciously (or unconsciously) ask “is it worth it”. Models which provide unlimited access to testing promote a guilt-free environment. More testing gets done and leads to better products.

Being able to re-run a test, tweaking some missing condition if needed, allows for more informal test plans. It doesn’t need to be perfect the first time. This enables fewer delays in the test definition phase.

Finally, guilt free iterating is a great way to improve your process and make consistent progress.

Elastic demand – the vendor’s consumption model should allow for unpredicted ups and downs in activity levels.

Rapid response – you should be able to start a test within hours of the request, not days or weeks.

Large community – Global scale is what makes crowd sourcing work. You need access to a large, vibrant pool of tester. They need to be kept busy. They need to be learning from each other…and policing each other.


In the enterprise space, there are initially a lot of concerns over the idea of using crowd sourced testers in their environment. Typically, these revolve around security and access controls, but often risk, liability and IP infringement concerns are mentioned, too. Since companies like MasterCard, Citibank, MLB and Johnson & Johnson have all leveraged crowd sourced testing at one time or another, it is clear that these hurdles can be overcome.

Access Controls – every company needs to assess their individual access control and risk tolerance, but the crowd does not really introduce anything new in this area. Technologies (like VPNs, access control lists, biometric sensors) exist to provide controlled access to systems and data.

Sensitive data – make sure you have a valid non-disclosure agreement with your vendor and they have one with their testers. For special projects, make sure your vendor facilitates a direct NDA between you and the testers. While limiting the pool of available testers, it is possible to background check individuals. If you have concerns about highly sensitive data, make sure you vendor is willing to go this extra step.

Unknown crowd – the crowd does not need to be anonymous. Make sure your vendor has controls in place to verify the identity of testers. Also, make sure they have a Code of Conduct for testers which holds them to well established standards of behavior. Ideally, the community should also police itself, leveraging the power of the crowd to protect their reputation.

Test data – if the community is large enough, you should be able to find existing customers to test their own accounts. Not only does this do away with the need for test data (in most cases), it also means there is less personally identifiable information (PII) lying around on your systems. If you have test data, the crowd can use that, too.

Liability – if something should go wrong, you don’t want to be going after individual testers in court. Make sure your vendor has insurance and is big enough to provide ample liability protection.

Complex Internal Apps – clients often comment that their apps are complex, have small user bases and require special subject matter expertise. But when you have a huge community, it is possible to find these skills.

You should also be able to segment the community into project teams trained on your app. The beauty of having a large pool of app experts is that it become self-healing. If you lose 1 or 2 people from a team of 30, the old team members can train the new people. If you lose 1 or 2 people from a team of 4, you are devastated.

Risk averse – a this point, it’s little late for this concern. Gartner recognizes crowd sources testing as a viable market and virtually every major brand in every industry segment is using the crowd.


Not every use case can benefit from the crowd. If you have an application that runs on a single (or small number) of platforms and can be effectively tested in a remote lab, crowd sourced testing is not likely to compete with traditional offshore resources. If your app requires you to be physically located at your corporate location in order to be tested, it is not a good candidate for the crowd. (In this case, you might be a great candidate for automation. Does you vendor offer any cloud-enable automation services?)

But that type of application is becoming more and more rare. So, if your app needs to work across a number of diverse platforms, needs to work in-stores and on the couch, need to work on the subways of NYC or the mountains of Tibet, you might want to consider crowd sourced testing.


I am sure there are benefits, concerns, and use cases I have missed. Please feel free to leave questions in the comments and I will do my best to answer them in a timely manner.

Share this article

Leave Your Reply

Your email address will not be published. Required fields are marked *