Elizafox's Ramblings

There I said it.

The benefit to anyone is questionable and could cause untold harm to millions.

I say this as an advocate of open source software, but also as someone who has been in FOSS a very long time as well as the professional software industry. I know how the sausage is made and I don't want that on anything critical to safety of life without rigorous standards.

Quality

Most open source takes the attitude that bugs are to be fixed by the end user, or at least reported, and the user is the QA department.

If your medical device firmware goes wrong, there may not be a user to report anything anymore.

Without rigorous testing and QA, a device's firmware would never be allowed to be on a device used for safety of life. For good reason too.

THIS SOFTWARE IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The lack of warranty or accountability is a huge obstacle.

It's fine to have no warranty for say, a calendar app. If it breaks in two, that's your problem.

Now imagine if someone updates the software on their insulin pump with something they downloaded off GitHub, and an integer overflow causes a massive overdose in insulin and results in death of the person.

Who is held to account for this?

Now let's say a well-intentioned 16 year old modifies Grandma's pacemaker to close a vulnerability with its near-field communications, telling their grandmother this could be used to hack it and kill them, but the fix inadvertently introduces another issue meaning some arrythmias aren't detected.

Who is held to account for this, too?

Again, these issues don't come up in non-safety of life applications, but open source is not an accountable medium.

Will anyone actually give a shit?

Open sourcing a project does not imply anyone will step up to the plate.

If a pacemaker only has a few thousand implantations, mostly amongst the retired elderly, who largely don't know how to use computers... the odds are remarkably low that you will find someone who has had it implanted and has the requisite experience to maintain it. And ideally, you would want more than this for reviews.

Willingness to follow industry safety-of-life standards

Standards such as P7774 and the JSF air vehicle coding rules are standards designed to ensure safety-critical systems don't fail.

I can't imagine many FOSS devs willing to follow these standards. We can't even get them all to agree a code of conduct is a good thing whilst sexual assault in the industry happens daily.

But it's worth noting these standards are written in blood. The Therac-25 killed people. I am not convinced FOSS methodologies as commonly practised could have prevented this tragedy, especially at the time. After all, with FOSS, the user is the QA department.

Regulatory hurdles

For reasons stated above, I am not convinced a regulator would allow anything developed with classical FOSS methodologies to be allowed anywhere near a human.

I think it may be possible to develop this software in the open, but “homebrew” software on devices critical to safety-of-life is a terrifying proposition to me.

Dunning-Kruger effect

The Dunning-Kruger effect is the observation that people often perceive themselves to be more capable than they are.

This is especially true of programmers.

I like the idea of hacking implants, but I don't like the reality of someone killing me with bad code and lax standards.

I don't think most programmers have requisite knowledge to maintain these devices. They often require specialist knowledge and working hand-in-hand with medical professionals.

Getting your average programmer to even go to the doctor is a nightmare.

Now imagine getting these people together to volunteer.

So, what's the answer?

I honestly think companies need to be legally held responsible for the maintenance of their product software until the last one is no longer in use.

In addition, we need industry protocols and standards for adjusting these devices so things don't become “obsolete” in short order.

I think in extreme cases, the government should arrange a contractor to maintain the devices if the previous one goes out of business.

We already make oil well drillers pay to plug their wells (though there are problems there too, like stripper well scams and the plugging bonds being laughably too low), we should do the same for device manufacturers.

Conclusion

I am not against FOSS or open-source in any way. I think a FOSS model can work, with company backing, for safety-critical code. But without said backing, someone to hold accountable for the software, I think it's too risky to have these devices open to user modification.

— Elizabeth Ashford (Elizafox) Fedi (elsewhere): @Elizafox@social.treehouse.systems Tip jar: PayPal || CashApp || LiberaPay