OK, I’ve been beefing about “cargo cult” apps that use mobile devices and sensors to do DIY environmental and medical analysis. Unfortunately, it’s getting harder and harder to even know how real things are.
Case in point, consider Tekla S. Perry report for IEEE Spectrum about the “SCiO Food Analyzer” app. First of all, this isn’t a trivial toy (like, say, a “smart” hair brush). They are building a tiny infrared spectroscope that attaches to or soon will be built in to a mobile phone or other device.
Is this real, or is this just something that looks real? It’s hard to tell.
My rule of thumb is the spectroscopy is pretty magical, so this has got to be an interesting device. The question is, does this device actually work? And how do we know that?
The suggested use case is the desire to examine food in the store to get a better idea of the quality. The IR scanner can do some kinds of chemical analysis, and report on carbohydrates, fat, and sugar contents, for instance. The app uses unspecified “algorithms” to relate the measures to the flavor of produce, as well as the levels of carbs.
The article reports on a successful demonstration of the technology, which impressed the reporter. The app isn’t intended to tell you what an unknown item is, instead you tell it “this is an apple”, and it tells you the sugar content and how it falls in the range of apples it knows about. I.e., the algorithm predicts how the fruit will taste, based on the readings.
“It was all pretty magical, pointing a gadget at food and getting an instant analysis. To be fair, I can’t verify the accuracy of what I was seeing on the screen; I didn’t take the fruits and cheeses back to a laboratory to confirm the analysis using more traditional technology. But it certainly seemed real, real enough that I would be pretty excited to have this kind of technology built into my smart phone,”
Evidence? There are no obvious citations in the article. Consulting the company web site, there are some generic descriptions of the technology, but no validation study, published or not.
This being Silicon Valley, there is lots of information about awards and press reports, as well as news about funding and company alliances. Apparently, attracting venture capital and phone manufacturers is supposed to tell me that the results are scientifically valid. Sigh.
The lack of peer-reviewed evidence is a concern. For one thing, it is offered in the area of food safety (and possibly drug safety), which are potentially dangerous if users misinterpret the results or rely on them farther than they should. (“My phone didn’t say it was contaminated, so I thought it was OK to eat it.”)
It’s not that the technology is unbelievable, or implausible. But the fact that it could work does not mean that this particular device does work.
There are many questions that I’d want addressed.
The IR scan has many obvious limitations. I’m pretty sure it won’t work on frozen food, nor through foil or other IR opaque packages. I suspect it won’t work for most cooked foods, I don’t know what kinds of errors it may be vulnerable to. (Dust? Water on the lens? Sugar water on the packaging Fingers in the way of the scan? Deliberate hacking?)
The unspecified algorithms are surely some form of machine learning. What exactly were they taught? What are the limits of the data? If it knows about apples, what about pears?
For example, there are hundreds of species of apples. Have they sampled all of them? How well does the system deal with a new variant? What happens if I point it at a kiwi fruit and ask it if this “apple” is fresh?
The basic learning task is not just a chemical analysis, but also relating the chemistry to the quality of the produce. What heuristics are used, and how valid are they? In addition to variation in produce, how much variation is found among people’s tastes? How is this accounted for in the algorithm? Just how useful are the results?
And, of course, whatever it does, how reliable and accurate is it?
Having made a living as a software guy, I know very well that demos are hardly the same thing as actual validation.
Finally, I thought it was kind of funny that the motivating problem was that the food in the local store tastes blah, and “he resigned himself to occasionally buying tasteless produce or traveling 30 miles to a grocer he discovered that he could trust.”
This device addresses this lack of trust by…I’m not sure. I guess it lets you avoid the stuff you don’t want, though it doesn’t do much to get better food into your local store.
But the funny part is that the lack of trust in the store is solved by an app that does a lot of fancy stuff–that we are asked to trust on faith.
At the moment, it’s hard to know just how well this “magical” product works. Is this real, or cargo cult? I can’t say, and that means I must assume it doesn’t work until proven.
It is more than a little worrying that venture capital seems to have replaced openly published research as the method for validating technology. We know that will lead to disaster.
- Consumer Physics, “SCiO: The world’s first pocket size molecular sensor”, https://www.consumerphysics.com/
- Tekla S. Perry, What Happened When We Took the SCiO Food Analyzer Grocery Shopping, in IEEE Spectrum – View From The Valley. 2017. http://spectrum.ieee.org/view-from-the-valley/at-work/start-ups/israeli-startup-consumer-physics-says-its-scio-food-analyzer-is-finally-ready-for-prime-timeso-we-took-it-grocery-shopping