Category Archives: Technology

Ethereum faces classic software engineering problems

The Ethereum developers have been rediscovering (but certainly not reinventing) software engineering.  Under the benign dictatorship of Vitalik Buterin, Ethereum has struggled to maintain professional quality software without a conventional top down, closed organization.  (See also this and this).

This month we read about their struggle with planning and scheduling “hard forks”-significant software updates that are incompatible with earlier code [1].  In conventional software development, these are managed through a central distribution, and users must take the update or loose compatibility.  In cryptoland, “hard forks” are “voted on”, and if a significant fraction of the network does not accept the change, the network splits.  And there may be competing “forks” that address a problem in different ways.  Sigh.  This is no way to run a railroad.

While this approach is “disruptive”, at least in the sense of “less organized and much harder” than conventional software project management, But it hardly “reinvents” software development.  Amazingly enough, all the hard problems of software maintenance are found in cryptocurrency software, and still have to be solved.

Crypotoland is already famous for its non-functional planning and decision-making.  What changes should be made?  More important, what changes must be made, versus might be made?  What are the effects and implications of a proposed change?  And so on. (See this and this and this.)

The “hard forks” problem is basically the difficult question of compatibility.  Some changes are so drastic that you basically have to throw away all the old software and data—it’s effectively a whole new product.  These changes are painful for users, and the more users you have—the more successful you are—the more difficult such upgrades become.  There is too much sunk cost in the old software to blithely toss it out and redo it.

This is not just conservatism or laziness.  As the Ethereum folks recognize, there are a lot of people using the software that simply do not have money, people, or expertise to port their stuff to a new version, let alone to do so over and over again.  And if they do try to keep up, they may spend most of their time just chasing the releases, with no time for their own software or business.  (Been there, done that.)

In the case of Ethereum, they also face a classic software dilemma.  They are working on “Ethereum 2.0”, which is a pretty complete rework of the basic Ethereum protocol.  In principle, everything will be wonderful in 2.0, but it will be quite a while before 2.0 is ready—it’s already a couple of years of discussions, and probably several more years before it might be done.

In the mean time, there are many changes that might be made to the current version of Ethereum core software.  Some of these may be critical fixes, others are good ideas, and others are, well, who knows?  But all these changes will be obsolete when the great day comes and Ethereum 2.0 comes out.  (Although, some changes might be applicable to both old and new—so they have to be done twice.)

So just how frequent should these “hard forks” be?  Too few, and the software may suffer.  Too many, and downstream developers will be overwhelmed?  And everything you do before V2.0 will essentially be thrown away.

Coindesk reports that the developers are discussing setting a regular schedule of forks, every six months, or even every three months.  Of course, if history is a guide, they probably can’t hit such a target anyway, presumably because the process of testing and preparing (and, I hope, documenting) the code takes longer than hoped.  (Definitely been there, done that.)

The ultimate kicker is that unlike conventional software, every one of these updates can be a political disaster, potentially causing a schism in the network, with untold consequences for users.  Software maintenance is hard enough without having to have to worry about civil wars among different interest groups.

It is good to see that Ethereum folks seem to have some understanding of these challenges, and are taking them seriously.  It is reported that the God Emperor of Ethereum, VB, wants a conservative policy for changes, doing only those necessary for survival of Ethereum, until 2.0 is out.  On the other hand, the schedule for 2.0 is uncertain and seems to slip farther into the future every year, so there is desire to improve what exists, rather than what might exist someday.

However “innoruptive” you think cryptocurreny/blokchain is, it’s still software, and it’s still hard.  And the more you succeed, the harder it gets.

Welcome to the software biz!

  1. Christine Kim (2019) Ethereum Core Developers Debate Benefits of More Frequent Hard Forks. Coindesk,


Cryptocurrency Thursday

Hacking Tesla’s Autopilot

The folks that brought you the Internet are rushing to get you into a self-driving, network connected car.

What could possibly go wrong?

Setting aside the “disruption” of this core economic and cultural system, there have been quite a few concerns raised about the safety of these contraptions.  Automobiles are one of the most dangerous technologies we use, not least because we use them a lot.  Pretty much everything that can go wrong does go wrong, eventually.

Well, buckle up, cause it’s as bad as anyone might have thought.

The spring the Tencent Keen Security Lab report on successful hacks of Tesla cars [2].  The focus was on the autopilot self-driving system.  In fact, they were able to root the car, monkey with the steering and other controls, and reverse engineered the computer vision lane detection to make a simple hack that could cause the vehicle ot suddenly change lanes.

“In our research, we believe that we made three creative contributions:
1. We proved that we can remotely gain the root privilege of APE and control the steering system.
2. We proved that we can disturb the autowipers function by using adversarial examples in the physical world.
3. We proved that we can mislead the Tesla car into the reverse lane with minor changes on the road.” ([2], p.1)


Rooting the car is obviously bad for many reasons, and in this case they used their access to discover weaknesses in the other systems.  Taking over the steering is, well, just about as bad as it could get.  Tesla’s response is that this isn’t a “real” problem, because the driver can always override at any time.  But doesn’t that defeat the purpose of the autopilot?

The lane changing hack is interesting, if rather academic.  They found a case where just the right paint on the road could fool the algorithm.  As Evan Ackerman puts it, “Three Small Stickers in Intersection Can Cause Tesla Autopilot to Swerve Into Wrong Lane[1]. But this is really a rare case, and would probably be overridden if oncoming traffic were present.  As Ackerman comments, though, this brittleness is worrying because “the real world is very big, and the long tail of very unlikely situations is always out there,”.

To be fair to Tesla, a big part of the problem is that the car is full of software that is connected to a public network.  The hackers got in through the automatic updating system over the network.  Tesla is hardly the only car designed this way, and more companies are moving to remote updates for on board software.  Sigh.

There are many other autopilot and self-driving systems under development (including at Tencent).  They will have similar vulnerabilities.  If the received wisdom from software engineering holds true for these systems, there will be one bug for every ten lines of code—thousands and millions of bugs to exploit!

I’ll also note that a key part of the attack was that the root access allowed them to examine the system in detail and at leisure.  This raises a big question about proposals to release or open source software for cars and other systems.  Yes, this can lead to the rapid discovery of flaws. But it also means that hackers can have their way with the system.  And it certainly doesn’t mean that bugs will be fixed quickly out in the field.

My own view is that cars and other life-threatening technology should never, ever be connected to a public network, and should never do software updates over public networks.  That’s less convenient and costlier for the manufacturer, but that’s just tough.

  1. Evan Ackerman, Three Small Stickers in Intersection Can Cause Tesla Autopilot to Swerve Into Wrong Lane. 2019: IEEE Spectrum — Cars That think.
  2. Tencent Keen Security Lab, Experimental Security Research of Tesla Autopilot. Tencent Keen Security Lab 2019-03, 2019.


Robot Wednesday

Local Solar Power: Lot’s of Progress, but Some Pieces are Still Missing

In recent years, photovoltaic (PV) systems have become cheap enough that they now are cheaper than coal and competitive with natural gas.

Solar energy can be harvested at many scales, from giant arrays (potentially in outer space), buildings and campuses, individual homes, and, of course, gadgets [1].  It’s basically the same technology.

One of the most interesting things about solar power is that it really is possible to distribute the generating systems at many scales, including personal, neighborhood and community systems. For me, this means that it is possible to “put the tools in the hands of the workers”, so people can own their own electric power generation.  How can I not want this?

Achieving this vision requires meeting a number of challenges.

While PV is cheaper to install each year, and very cheap to operate, building and installing a solar array needs a significant financial investment.  Second, any scheme to share electricity (solar or other) requires distribution to the users, ideally via existing grid connections.  And  third, this requires political and economic structures for the technical systems.

In recent months I’ve reported on some developments that are addressing these challenges [2,3].

This being the US, any problem that can be solved by monkeying around with money, is a problem we can solve, yessir.  So, for instance, there is now a Clean Energy Credit Union (CECU), which offers all the advantages of an insured credit union, and is dedicated to financing PV and other clean energy for consumers and small businesses [2].

Locally, there is also a bulk purchase program, which negotiates a good deal from a good provider, and then promotes installation of PV on homes and businesses [3].  In both these cases, the institutions help pool and direct local people’s money to local projects (and local workers).

These are good things, and help everyone “Think Heliocentrically, Act Locally”.  But this isn’t the end of the story.

The vast majority of people do not own their own home or business.  To date, the only way to get PV power is through public utilities, assuming the utility has renewable energy generation and can and will “sell it” to customers (e.g., through a check off that requests renewable energy).   In some places, including my local area, cities are generating renewable energy for government and sometimes local consumers.

But how can average people, without a lot of money, invest in solar power, and reap the benefits of generating their own power?

We can see that the CECU credit union is implementing the “consumer lending” model that helped get two cars in every garage, as well as a lot of people into houses with garages  (for better or worse).

For generating power, we might look for other models from how people finance their housing, such as condominiums, time shares, and cooperatives.  The basic idea is for people who don’t necessarily own property to pool money and build PV arrays near by.  The power generated is shared out to investors, and any profits would go to them.  It is quite possible that an owner/customer’s investment might be entirely paid back in a decade or so, from reduced utility bills.

There are more than one way to skin this particular cat, but I’m particularly interested in local cooperative model for community solar projects.  There have been electric coops for a century and more, usually in underserved rural areas.  This same model can work for a small solar farm in town or on roof tops.

What does it take to do this kind of project?   I’m still learning the ins and outs of how it might be done.  In general, there are a variety of organizational and legal models [5]   Personally, this old bolshie heart beats fastest for a pure cooperative, a la People Power Solar Cooperative [4] .  Much depends on local laws.

Beyond legal charters, the key is, as usual, the right people and leadership.  Identifying and mobilizing the right people in the right way.  Easym peasey!

I have a lot of work to do before I’ll see anything coming true.

More later.

  1. Robert McGrath, Tiny Watts – Solar Power For Everyone, in Tiny Watts Blog. 2018.
  2. Robert E. McGrath, A New Option to Finance A Clean Energy Future for Everyone, in The Public I: A Paper of the People. 2018.
  3. Robert E. McGrath, Think Heliocentrically, Act Locally, in The Public I: A Paper of the People. 2019.
  4. People Power Solar Cooperative. People Power Solar Cooperative. 2019,
  5. Trebor Scholz and Nathan Schneider, eds. Ours to Hack and Own: The Rise of Platform Cooperativism, A new Vision for the Future of Work and a Fairer Internet. OR Books: New York, 2017.

CryptoTulips For the Home: “Blockchain-powered” refrigerator

The cool thing (no pun intended) about refrigerators is how simple they are.  Plug it in, put food in, take food out.  Maybe adjust the temperature setting so frozen stuff stays frozen and the milk does not freeze.  Every once in a while, maybe a light bulb needs to be replaced.  There you go.

This spring Wien Energie and Bosch are demoing a “blockchain-powered” refrigerator (quoting Coindesk [1]).  Obviously, blockchain doesn’t “power” the appliance or anything else–in fact, blockchain sucks power big time.  As in many IoT concepts, a blockchain is actually used to log the consumption and source of the electricity used, and ultimately, to use “smart contracts” to control where and how much power is purchased.  Wein Energie is, of course, a power company, so they are on the other end of the transaction.

The idea apparently is to be able to know how much your refrigerator is using, and also to be able to specify that you want power from local renewable sources, for instance, and to use the blockchain records to prove your Carbon footprint.

The fridge also has the usual IoT widgets.  A mobile device controls the settings and tracks usage.  It also raises alerts, though the only example given is if you leave the door open.  (As I said, fridges are pretty fool-proof to start with, so it’s hard to add digital whistles and bells.)

The press release says* that this makes the customers active participants in the energy market, at the level of a every device [2].  Wein is, of course, in the energy market business.  Consumers, of course, are not.  And most of us really aren’t interested.

So, despite the Coindesk headline, this is really just another iteration of the blockchain IoT microtransaction theme (e.g., this, this, this, this) The refrigerator is implementing a general purpose electricity buying protocol that can be used for any electric device.

The Coindesk report talks about controlling the refrigerator power stand alone, i.e., with a contract for where the refrigerator gets its electricity [1].  I suspect this is misleading, because it will be completely unwieldy to have to “actively participate” device by device.  Also, it seems unlikely that you’ll want to do different things for each appliance, vehicle, screen, etc.  You’ll probably want to buy power from one source, and distribute it to all your devices.

So, wouldn’t it be better to have an account that buys electricity for all your devices?

Your refrigerator doesn’t need to make a separate bargain with the supplier, and the power company doesn’t need to know how much power your refrigerator is using, I think what you really want is an (opaque) outer “shell” that gets electricity for your whole ensemble, and a separate, private system to manage what you hook up and how you allocate within your bubble.

What I’m describing is much more complicated that the Wein concept described in the demo. I’s actually a glorified “smart meter”, that acts the same way as a firewall. The blockchain might be a good fit for the house to power company market, but the within the house management should have its own private data.   The network inside home is not particularly simpatico with an open-to-the-whole-world blockchain, it is likely that a conventional database would make more sense than a blockchain for managing your own devices in your own home.

I’m not a gigantic fan of the whole idea of “smart devices” in general, and very much not a fan of having every device talking to the open internet.  So, I’m really not a fan of this kind of “transparent” appliance connected to the grid model, blockchain or no.

In my own view, I think that using a blockchain solves the problems of the power company, but seems to be a poor match to the actual needs or desires of customers. In fact, it seems to create a lot of new problems for home owners:  now you  have to do a bunch of administrative IT work to run your refrigerator and everything else.

Ick!  How is that an improvement?

Perhaps blockchain “powered” appliances will be a candidate for the Crypto Tulip of the Year this year.  We’ll see.

* I’m relying on the Coindesk translation—I’m not fluent in German.

  1. Daniel Palmer (2019) Bosch and Wien Energie Demo Blockchain-Powered Refrigerator. Coindesk,
  2. Wien Energie, Sonnenstrom kühlt Bio-Joghurt, in Wien Energie – News. 2019.


Cryptocurrency Thursday

Hydrogel Water Harvester

Pretty much everything on this planet depends on clean water, and one of the great dreams of science fiction is technology that extracts clean water from saltwater or humid air.

This spring researchers from Texas report on a new technology that absorbs useful amounts of water from the air, releasing clean liquid water. [2]  Cool!

The technology is “a super moisture-absorbent gel (SMAG)” which is a complicated hybrid of chemical s that collect more than its weight of water, and store it.  As far as I understand, the material has a mesh that capture water, and many pores that store water that is captured. Similar to a sponge, the material swell as it absorbs, and when heated shrinks and releases the water.  The cycle can repeat daily, absorbing water at night, releasing it in the warmth of the day.  Actually, it continues to absorb water as it releases.

As far as I can tell, 1 gram of this material captures something like .4 grams per hour, so if you had a kilogram or so you might get enough water to supply a household. This is quite impressive!

Equally impressive, the process is powered by solar energy or other similar head source—no need for electricity or fuel.

The UT team’s technology requires only solar power, is compact and can still produce enough water to meet the daily needs of an average household. Prototype tests showed daily water production of up to 50 liters per kilogram of hydrogel.” (from [1])

This just goes to show you that, as a chemistry teacher told me a long time ago, with the right catalyst, you can perform black magic!

Nice work, all.

  1. Adrienne Lee, Solar-Powered Moisture Harvester Collects and Cleans Water from Air, in University of Texas – Science & technology. 2019.
  2. Fei Zhao, Xingyi Zhou, Yi Liu, Ye Shi, Yafei Dai, and Guihua Yu, Super Moisture-Absorbent Gels for All-Weather Atmospheric Water Harvesting. Advanced Materials, 31 (10):1806446, 2019/03/01 2019.


Fish and Bees “Talking” To Each Other

I’ve long been interested in Animal Computer Interfaces [3], so I was immediately interested in the news reports about “Robots Help Bees Talk to Fish[1], which splashed even in the mainstream media.

As I wrote way back when, interfacing non-human species via computers is a valuable exercise, which forces the designer to think broadly and deeply about the “user experience” of non-humans [3].  And communicating across species requires figuring out what kinds of messages and behaviors can be mapped to each other.

This spring a group of European researchers report on a study in which they  created a communication system mapping the flocking behaviors and cues of the two species to each other [2]. Both species move in groups, coordinating through subtle signals.

The interface is via species appropriate “robots”, which mimic the body and behavior of the relevant species.  This is an interesting use for representational robots:  as a conduit for “body language”.  (As an analogy, I could imagine a robot arm and hand “directing traffic” for humans.)

To the degree that this works, the mapping indicates a robust understanding of the species signaling system.  Whether the bees and fish have much notion of the other species is hard to say, but there is at least a possibility that we can ask the question.

The actual research isn’t exactly using robots on both ends of the channel. There is a robot zebra fish, but the honeybees have a species appropriate interface, an array of thermal and vibration sources.  So this is a real multi-species communication channel, with species appropriate interfaces on both ends.  One of the interfaces is a robot, the other isn’t quite.

The behavior in question is the species-specific flocking, which is detected in group decision making.  As such, the digital connection across the network link effectively creates a single, multi-species “flock”, making a joint decision, visible by the behaviors of the two local single species groups.  So, technically, this is not just communication, but also “social integration” of the two social species!

Did it work?  The study reports a bunch of statistics to show that it really worked, both ways.

This is really cool!

Of course, this is one of the easiest possible cases of communication/social integration:  these species are highly social and the signaling are pretty straightforward, and, for that matter the experimental task is the simplest possible behavior (a binary choice).

But still—they actually did it.  Well done!

I was surprised to read Sensei Evan Ackerman’s rather obtuse take on the experiment.

“This is a neat experiment, but what’s the point? What can the bees and robots do that they couldn’t do before?” (from [1])

Who says that the point is to create something that is “useful” for humans or any species?

The point is to understand animal communication and behavior, and to understand how to design computer systems for unusual tasks.  And, in general, the point is to show what is possible.

The researchers suggest that the same concept might be applied to other living systems, including fungi and microorganisms.  Well, sure, though this isn’t necessarily useful either (though we have really good reasons to want to persuade microorganisms to go here and not go there.)

This makes me think of other ideas. What about communicating with autonomous swarms of robots?  If you make the not at all far-fetched assumption that intelligent swarms will learn and evolve, then I say they will quickly leave their original design parameters behind, and become difficult to command via simulator.  So, we’ll need to communicate with them through various means not unlike bee and fish “language”.  That sounds pretty cool!

  1. Evan Ackerman, Robots Help Bees Talk to Fish, in IEEE Spectrum – Robotics. 2019.
  2. Frank Bonnet, Rob Mills, Martina Szopek, Sarah Schönwetter-Fuchs, José Halloy, Stjepan Bogdan, Luís Correia, Francesco Mondada, and Thomas Schmickl, Robots mediating interactions between animals for interspecies collective behaviors. Science Robotics, 4 (28):eaau7897, 2019.
  3. Robert E. McGrath, Species-appropriate computer mediated interaction, in Proceedings of the 27th international conference extended abstracts on Human factors in computing systems. 2009, ACM: Boston, MA, USA.



Robot Wednesday

US Solar Installations Growing Faster Than Any Others

Solar electric generation is growing fast in the US.  In fact, for the first month on record, solar generating capacity coming on line in the US in January 2019 beat all other forms of energy [1].

Despite a hostile administration in Washington and brisk tariff headwinds, solar continues to come on line.  (And, by the way, aging coal plants are continuing to retire at a fast pace.)

Why?  Costs are stable (despite tariffs), and once on line, the economics are as strong as ever.  Fuel costs?  Zero.

I’m pretty sure that many of these new installations are in jurisdictions that have policies encouraging renewable energy (to a first approximation, US states controlled by Democrats, which is a very unfortunate situation).  Think how fast soar would grow with a consistent national policy.

As far as I can tell, this trend will continue for the next year or more.

For example, in my own area, in addition to large wind farms, significant new PV installations are coming on line, at all scales.

New large scale arrays are coming at the University of Illinois  (55 acres), Parkland College, the Urbana City Landfill (41 acres), and the largest PV array in the state of Illinois (1200 acres!) coming near Sidney. There is also a continuing effort to install home and building PV, which will continue.

For comparison, these resources together likely will generate considerably more electricity than the coal fired Abbot Power Plant at the University of Illinois, and something like a quarter of the capacity of the soon-to-be-retired Clinton nuclear plant.

In a couple of years, our area will soon be home to what is probably the largest concentration of solar power generation in the state of Illinois—although I expect other projects elsewhere may eclipse (get it?) our efforts.

  1. Federal Energy Regulatory Commission (FERC), Office of Energy Projects: Energy Infrastructure Update For January 2019. Federal Energy Regulatory Commission, Washington, 2019.
  2. Kelly Pickerel, New U.S. solar generating capacity beats all other forms of energy brought online in January, in Solar Power World. 2019.