I have blogged many times about mobile apps that offer cargo cult science, such as DIY pollution monitoring, or medical diagnosis. It is absurdly easy to make an app and distribute it, and there is little filtering on what is produced, or what claims are made. Thus, there are hundreds of thousands of apps related to health and diet, with scarcely any evidence that any of them do anything, let alone produce the alleged benefits.
While I’m alarmed at shoddy apps that look good and make unproven claims aimed at the general public, I’m really, really worried about cargo cult science apparently marketed to scientists, who should know better.
A case in point is an “APD Colony Counter App” from the Agency for Science, Technology and Research (A*STAR), Singapore, which was reported last year in Nature Methods  and received uncritical mention in blogs that probably should know better.
The app itself is a perfectly reasonable idea, using image processing software to create a phone app that analyses and counts bacteria colonies on a Agar plate. This is a tedious and time-consuming thing to do by eye, so automation is a great idea. It is also true that ubiquitous mobile phones now have cameras and processing sufficient to do this task. So, why not? Techs have a phone in their pocket, why not use it?
But there are several things that worry me.
First of all, there are zillions of devices and packages that already do this (e.g., [1, 3]), albeit, many are expensive. In fact, there are mobile apps that do the same thing (e.g., ) So, why do we need another one?
Even more important, does this software actually work? And how does it compare to other methods, human and algorithmic? A cheap app that produces poor results is no bargain, especially if health or money depend on the answers.
In this case, there is a short abstract, published in Nature Methods . This paper does not review other methods, or cite any such review. The paper claims that the app is “convenient”, and portable. They also note that “there are many colony counter apps”, but this one uses software that is “to segregate merged colonies better for more accurate quantification”. So far, so good.
After discussing the interface, the paper offers one table demonstrating the effectiveness of the app. The table compares “App Count” to “Manual Count” i.e., the number of colonies found by a person versus the app. There is little information about this tiny dataset (the sample size is twelve), such as the actual images and the expertise of the human comparison.
For that matter, the measure reported is “accuracy”, which is the number of colonies counted by the app compared to the human. There is no report of false positives, nor of the accuracy of the human standard, nor any indication that the two methods identified the same colonies, i.e., how well they agreed with each other. This data simply does not show that the algorithm even works, and there is no evidence in the paper offered that it produces valid results.
Worse, this data isn’t even relevant to the question of whether this supposedly superior algorithm works better that other similar apps. They specifically assert that this algorithm yields “more accurate quantification” than competing apps, but offer not one shred of evidence that this is true. Where is the comparison study?
This is cargo cult science, apparently marketed to scientists. In a press release, they make expansive, if non-specific claims
“”If we were to look at the history of science, many breakthroughs—including discovering microorganisms—were done at home or outside the workplace,” says Gan. “By having apps that anyone can access anywhere, I’m hoping that we’re going to bring back the spatial freedom for scientists to make discoveries anytime, anywhere.” “ (quoted from )
Remember, this app is offered not as a toy, it is intended to “enable the smartphone to transform into a useful scientific device for the quantification of bacterial load in clinic, research, environmental, and even food safety regulatory labs.” 
Clinical tests and food safety? Yoiks!
Look, I’m sure that this app probably works pretty well. Maybe it is better than other apps, though I don’t know if it is meaningfully better.
But I need to know that this device is safe and effective. Offering something cheap that looks good, with unsupported claims that it is “accurate”, just isn’t good enough.
The truly embarrassing thing is that this is not a particularly difficult validation study. Evan I could do it. So I don’t think it is too much to expect that people who hope to create “breakthroughs” in science do some very simple science.
- Biocompare. Colony Counters. 2017, http://www.biocompare.com/Lab-Automation-High-Throughput/11357-Colony-Counters/.
- ColonyCount. ColonyCount. 2012, http://www.colonycount.org/.
- Quentin Geissmann. OpenCFU. 2013, http://opencfu.sourceforge.net/.
- Promega. Mobile, Desktop, and Web Apps for the Lab. 2017, https://www.promega.com/resources/mobile-apps/.
- Science X network, Counting microbes on a smartphone, in Phys.org. 2017. https://phys.org/news/2017-03-microbes-smartphone.html
- Chun-Foong Wong, Joshua Yi Yeo, and Samuel Ken-En Gan, APD Colony Counter App: Using Watershed Algorithm for improved colony counting . Nature Methods Application Notes.August 9 2016, http://www.nature.com/app_notes/nmeth/2016/160908/pdf/an9774.pdf