Machine Mislearning

Rock Stevens and colleagues from University of Maryland have published a cool paper on security attacks on machine learning systems [1]. Despite the ever increasing use of machine learning, and it’s near magical effectiveness, there has been surprisingly little attention to security issues in this complex and data intensive software.

Actually, the lack of attention may be due to the “magical” nature of machine learning. It works so well despite the fact that we often don’t really understand what it is doing–it’s magic. Who wants to spoil the party by thinking about evil hacker mischief?

The paper (titled, “Summoning Demons: The Pursuit of Exploitable Bugs in Machine Learning”) explores the “attack surface” of a typical machine learning system—and there are plenty to choose from. The discussion of the basic components of ML is familiar, but thinking about them as avenues for attack by hackers is a new perspective.

They use openCV as a well known, open source, example. In this cases, OpenCV is used learn to classify images, e.g., as containing a human face or not. Pretty much every component of the system is potentially vulnerable to false input or software manipulation or both.

For example,

Within the feature extraction component itself, an attack can target the input parsing algorithms and/or the integrity checks performed on the feature representation.” These hacks “could result in mispredictions, mis-clustering, arbitrary code execution or [denial of service].

And so on.

The paper reports on a study that systematically explored to find bugs in openCV. They describe their method, termed “fuzzing”: input cases are fiddled with small random changes to generate a host of inputs to be tested. Some of the mutant inputs are handled correctly, some are correctly rejected as “broken”, but some produce incorrect results that the program says are correct: these are “silent errors”. This technique discovered quite a few cases that crashed the software (which is not a great behavior), and five “exploitable” bugs, i.e., security issues.

The investigation discovered bugs that enable an attacker to insert arbitrary code that can manipulate or crash the system, and also bugs that enable an attacker to manipulate the machine learning in several ways. The researchers note that when they reported these bugs (which is standard procedure), there was a different response. The bugs that involved attacking the code were assigned identifiers and priority to be fixed.

However, “the bugs that led to misprediction, misclustering, or model divergence—including a Malheur memory corruption bug that allows the adversary to control the feature matrix, but not to inject arbitrary code—did not receive CVE numbers [i.e., recognition as priority issues to be fixed]. Many of these bugs were labeled WONTFIX.

Yoiks! “This emphasizes the fact that ML bugs are currently a misunderstood threat.”  Quite.


Thinking about this, I realized that this is really “attacking” machine learning in the same kinds of ways that you would “attack” natural human learning. You can manipulate the data available in ways that exploit errors in the learner’s heuristics, to either conceal or mislead his or her perception or decisions.

But, as this study suggests, you may also be able to “hack” the learner at a deeper level by manipulating his or her assumptions, methods, and expectations—i.e., fiddling the way they learn and perceive.

Deception of humans has been studied for millennia, but this work gives an additional perspective. Machine learning isn’t necessarily similar to any specific human reasoning or perception system (though it is designed to be analogous), but thinking about these “attack surfaces” might give us insight into cognitive and perception errors, as well as deliberate “attacks” on human understanding.

It is is interesting to think about how certain deceptive inputs might be designed to deliberately trigger “bugs” in a decision maker’s process. We see this all the time, e.g., in subtle “photoshop” manipulations of images to remove wrinkles or adjust skin tone. This ML study makes me think about what kind of “programming” is being hacked here, and what might be done to “patch” the bug.

Anyway, this is a really cool paper.


  1. Rock Stevens, Octavian Suciu, Andrew Ruef, Sanghyun Hong, Michael Hicks, and Tudor Dumitracs. Summoning Demons: The Pursuit of Exploitable Bugs in Machine Learning, 2016. https://arxiv.org/abs/1701.04739

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s