go back to the blog

Testing with the crowd

  • 23/11/2012
  • no comments
  • Posted by EuroSTAR

Nowadays, everyone is talking about “the cloud”. The cloud is cool, the cloud is THE solution! Change two letters and there’s another web 2.0 phenomenon waiting for you. “Crowd” a.k.a. “crowdsourcing” is a term you may not know by name, but you have probably encountered it in your daily life. In this blog you’ll find out what kickstarter, Wikipedia and lay’s chips have in common and what day have to do with testing.

Well… what is it?

“Crowdsourcing is a neologism, used to explain a recent development in which organizations (government, companies, institutes) or persons use a large crowd of individuals (professionals, volunteers, interested people) who are not specified beforehand for different tasks like consultancy, innovation, policy and research.”

Wikipedia NL

Cloud and Crowd aren’t only related in words. In real life they also have a lot in common. To make crowdsourcing successful you need the cloud. In this blog however, I want to emphasize the crowd. Those clouds in the sky are regarded as obvious in this case.
According to the English Wikipedia (a perfect example of a successful crowdsourcing project by the way) the first crowdsourcing project was from “The Oxford English Dictionary” (OED). An open call was made to the community for contributions by volunteers to index all words in the English language and example quotations of their usage for each and every one. They received over 6 million submissions over a period of 70 years.

crowdsourcing_147x116Crowdsourcing looks a lot like outsourcing. The difference is in the character of the company that’s taking on the job. The smart brains in the example above could have outsourced the work to the university which would probably have released a horde of students to do all the work. How long would it take for them to collect that many words? Instead, they reached out to the normal people. Maybe there were some linguistic geniuses among them, but nevertheless there have worked many more people on this project than when it would have stayed closed. In addition, less money was spent this way than when it would’ve been outsourced. The only reward the crowd probably got was the honor of contributing to a famous dictionary.
When internet arrived, crowdsourcing became a lot easier. Only a few clicks and keystrokes and people from all over the world can be reached to help on a job or take some other advantage from the crowd. Some examples will follow.


Crowdfunding: When you have a great idea it’s often hard to find somebody to invest in your invention. Kickstarter is an example of a website that makes sure clever people without money can bring their ideas under the attention of an audience. People can order the product before it is made or invest (small amounts of) money. When there’s enough cash the clever mind can start building his product.

When you buy more, things become cheaper. People who like sales in supermarkets know this. That’s is why consumer organizations in the Netherlands organize auctions for the energy market. Which energy company doesn’t want see a large group of customers becoming their client? They would love to give these people a discount on their energy bill.
lays_127x107Crowdsourcing: As a potato chips company you have to introduce a new flavor to the marked now and then. Why wouldn’t you ask your own customers to create this new flavor? Thanks to the crowdsourcing project initiated by Lay’s we can now buy “patatje joppie” chips. It’s based on a mayonnaise/curry/onion sauce. At the moment we can try the new finalists. What’s it gonna be? Ailo e Aglio, Whiskey Cocktail or Spicy Reggae Chicken?
Another example is the mobile app “Roamler”. A company that wants to know if they products are placed correctly in a supermarket can send some employees to investigate, which costs a lot of time and money. Instead this company can send the people who are already there. During shopping the smartphone users make a picture of the productplacement and maybe describe the placement in the store. Roamler employees check the evidence and send the results to the company.

In crowdsourcing it’s all about one party that hands an activity to another party, the crowd, to execute it. Both parties benefit. The first has his job done in a smart and usually cheap way. The other gets money or another reward, like points and/or appreciation.

Crowdcourced testing

Now I hear you asking: “But isn’t there a way we can use “the people” for testing?”. The gaming industry is doing this for many years, Squerist (the company I work for) experienced this when they were investigating the possibility to send functional testers to the videogame creators. Why would they hire expensive testers when they can send a beta version to gamers that will spend many free hours testing the game, only to be the first one to be able to experience the gameplay of this highly anticipated videogame?
Well, I understand there’s a big difference between a videogame and a high-risk financial application. You can’t just let some random people take a look at it. You need a decent risk analysis and good planning. There are a few companies that saw this dilemma and thought of it as a challenge. UTest is one of them.


UTest wants to be the bridge between the internal testteam and the crowd. These are some benefits UTest mentions on their site extended testing coverage. Instead of testing indoors, the application to be tested is sent to real users in the whole wide world, with different languages and different hardware setups. More combinations that any other (outsource) company could ever make possible. Other benefits that are mentioned are: Maintain control, increase app quality, tester diversity and improve efficiency. They don’t give pricing information, because for every job the pricetag is adapted to the demands of the cutomer. Anyway, I assume the price will be less than when the same amount of testers is hired through outsourcing. It’s impossible for me to make a statement about price/quality ratio. Proper research is needed at companies that have experience with UTest.
utest_228x85As a tester at UTest you will get your own web portal with information about tasks, reported bugs, ranking and financial information. Yes, you will get paid when you work for UTest. When you find bugs you will get paid for every approved bug and if you’re writing testscripts you will get paid for every script you deliver. If the rate is $4,- for every bug, you can calculate for yourself how many bugs you need to get approved every hour to earn the same amount of money you do now.
UTest’s clients also work through a portal. A UTest advisor can help them to make the right choice regarding region, scope, testapproach, etc. Testers who meet the requirements get an email invitation and the testrun can begin. The client can watch during the test activities which bugs are found and if necessary he can ask questions to clarify the bugreport, so communication runs fast without UTest in between. They only run the platform.

What can we do with it?

I am a test engineer at a company that focuses on software quality assurance and process optimization. Should I worry that companies won’t call my office anymore to hire me, but instead ask the crowd? In some projects crowdsourcing could be the better choice over hiring external testers. For instance, when you need to test under a lot of different hardware configurations or when you want to let normal people test your website after the professionals are done testing. But it’s not always the right decision and there are moments you need normal testers in your building. Maybe the application that needs to be tested can’t run outside the office. Another possibility is that the project is very secret and you want to see the autograph of the tester on a secrecy contract and not UTest’s. What if a crowdtester from Somewheretown runs to the competitor and all we know is his email address? There are many more examples in which an oldfashion tester is preferred over a crowdtester. These were the threats and defences. What are the possibilities?
Testers can gain experience in the business of testig by participating in UTest projects. They can learn how to properly report a bug and how to communicate with other testers and clients. All through the web. Also companies can learn from how UTest can find the right tester for the right job very quickly by making them fill in their profile.
One final idea for my company. When I created my UTest profile I was encouraged to participate in a test cycle for beginners. There was no money involved. It was like a game with the purpose to get used to the way UTest works. Together with some other newbies we were investigating a public website ( in our case) and searched for wrong links, label errors and other bugs. You were done when you had four accepted bugreports with right description, title and screenshots. This way of work can easily be implemented as a game for co workers or job candidates to separate the men from the boys. Who will be (or is) the first company that hires new testers through a crowdcourced testmatch?


Richard Dekker

richarddekker_97x130Is a test engineer working for Squerist, A service company in the Netherlands. Squerist focuses on software quality assurance and process optimization. Its mission is to give companies confidence in the quality of process and IT in a innovative and inspiring way.

Blog post by

go back to the blog


Leave your blog link in the comments below.

EuroSTAR In Pictures

View image gallery