How California should watch medical AI

Capitol Weekly welcomes Opinions on California public policy or politics. Please read our guidelines for opinion pieces before submitting an Op-Ed. Submissions that do not adhere to our guidelines will not be considered for publication. 

OPINION — I read imaging most days, and I also build clinical software that California’s new AI law covers.

Studies that get left in the queue are not oddities. They happen with outpatient scans for patients seen in routine clinic visits, for symptoms that were not seen or described in a manner that would denote their urgency. We can sometimes open these studies up and notice things that the patient and ordering clinician did not expect, from appendicitis to hip fractures or intracranial bleeds. These are real cases. Thank God they are not common occurrences, but for the patient they happen to, they are common.

Starting Jan. 1, Senate Bill 503 says clinics and physician offices that use AI decision support tools “shall regularly monitor” them and take “reasonable and proportionate steps” against “biased impacts.” It doesn’t say what regularly monitoring means.

That lack of specificity may lead to overbuilding, when what may be necessary is a flagging system: for when providers agree with the tool, disagree with it, or see a recommendation, suggestion, action or change in clinical workflow that is unsafe.

Software that is biased? We know that health care can be biased, not intentionally but because of systemic issues that unfortunately cannot be legislated away. It’s impossible and unreasonable to ask providers to flag bias. How can a clinician flag bias in a piece of software when we know clinically that bias exists in health care? That would be part of flagging issues, as described above.

I’m not someone who thinks legislation should be telling providers how to do this. But if it is going to get involved, it should facilitate the creation of a simple, straightforward system, low cost and auditable for research in the future by providers and by device companies as they iterate. It should not cost much or be complex, and it should not contain HIPAA-level data. If it begins to become complex, it should be internalized by the practice, not mandatory. The open-ended concerns that are there now are not helpful.

We’ve known of many software pieces for computer-aided detection that aren’t all that useful. Unfortunately, some took a stronghold in things like mammography, where most mammographers (I am not one) have suggested that it is not all that useful. It took hold because it was paid for, and it stayed. This is wasteful.

Dictation software was initially not very good and was glitchy, but it has improved significantly since its inception. If it had been kneecapped early on, with constant messages and workflow stoppages asking for input after each report instead of only when there was a real issue, no one would have been able to withstand it, suggest edits and corrections, and get the software improved.

If I could change one thing before the law goes out, it would be to allow for flexibility, so providers and developers can experiment to find the best method to evaluate and work on these systems. Frankly, there are many ways this could be approached. I suggest one, but the best approach remains to be seen, and we shouldn’t pigeonhole ourselves into a methodology that can make us fall behind. We should try various ideas and iterate. This process should be just like development in software and tech has been for years: We create something imperfect, then take feedback and iterate until it improves.

This doesn’t put patients at risk, because nothing here is making the clinical decision. It’s making suggestions for patient care. And although changes in workflow can affect outcomes, providers are great at noticing when a device, a new technology or even AI does them a disservice when they are trying to get to the right diagnosis and treatment.

Pouyan Golshani, MD, is an interventional radiologist and founder of GigHz, whose clinical software SB 503 covers.