AB 2575: Rage against the machine

As lawmakers race toward the end of the legislative session, AB 2575 is just one of many AI-related bills on the docket for them to consider. But amid the countless discussions currently surrounding AI, AB 2575 addresses one issue that may not immediately come to mind: the rights of healthcare workers using these systems.

Authored by Assemblymember Liz Ortega (D-Hayward), AB 2575 would protect healthcare workers from employer retaliation solely based on their overriding of an AI system within their scope of practice. Additionally, the bill would bar AI developers from asserting a defense that, in the case of patient harm due to an AI recommendation, a healthcare worker’s failure to override the system absolves the company of any liability.

AB 2575 would also enforce transparency and regulation around AI systems used in healthcare. Under the bill, healthcare facilities that integrate AI systems into patient care must readily provide information regarding these systems to its users, including its intended role in patient care, known risks and limitations and its method of generating outputs.

Ortega introduced AB 2575 in February after hearing increasing concerns from healthcare workers about AI in her roles as both the Chair of the Labor and Employment Committee and a member of the Privacy and Consumer Protection Committee.

“What I was hearing from healthcare workers and nurses is that they were being pushed to use AI more and more, and the expectation from their bosses was to follow the output,” she said. “And when AI gets it wrong, healthcare workers are put in a double bind. Follow the machine, and then they get blamed, or override the machine and then they risk retaliation.”

AB 2575 is co-sponsored by the California Nurses Association and the California Federation of Labor Unions, and is supported by several labor and consumer organizations. Carmen Comsti, CNA Government Relations Director, stated that while the bill would protect all healthcare workers, it is especially crucial for nurses in their role as patient advocates.

“Nurses, within their scope of practice, have a duty for patient advocacy. So it’s quite literally in their scope of practice to alert doctors, the healthcare facility, if there’s something that they believe would not be beneficial to the patient,” Comsti said. “Part of our bill is making sure that the relationship between the nurse and the patient is maintained, and that we are protecting that clinical relationship within California law.”

The bill is also opposed by a number of industry and healthcare associations, including the California Hospital Association, the California Medical Association and the California Chamber of Commerce.

David Simon, Senior Vice President of Communications at the California Hospital Association, expressed concern over the bill’s implications for clinician accountability when working with AI systems.

“Our read on this bill [is that] if a clinician overrides an AI support tool, and patient harm results from that override, that clinician is insulated from remedial actions,” he said.

However, Ortega maintained that removing clinician accountability has never been the bill’s intent.

“Healthcare providers [and] nurses must still follow the standard of care,” she said.

AI systems have been used to assist with patient care in a variety of ways in recent years, from clinical notetaking to generating possible diagnoses and treatment plans.

“When AI gets it wrong, healthcare workers are put in a double bind. Follow the machine, and then they get blamed, or override the machine and then they risk retaliation.”

For Sarah Rahman, a primary care internal medicine physician and Chief Medical Information Officer at Highland Hospital in Oakland, the implementation of AI into her practice has proved extremely useful — particularly Ambient Scribe, an AI tool that listens to doctor-patient conversations and generates clinical notes.

“The reduction in cognitive burden actually allows me to give better clinical care to the patient, improves my documentation, allows me to update the problem list [and] allows me to ask the patient a question that maybe I wouldn’t have had time for about their healthcare maintenance,” she said.

An emergency department physician at Kaiser Permanente, who requested anonymity due to Kaiser’s media policies, agreed that the AI listening tools have proved helpful in reducing administrative burden, particularly in a fast-paced emergency room. But they also find themselves editing the tool often, noting that it needs strict physician oversight.

“Occasionally it’s great and I don’t have to make changes. But typically, I have to tweak a few things that it gets wrong,” they said.

As for the recommendation aspect of the tool, which suggests potential diagnoses or treatment plans, the physician considers it to be far less useful.

“I find myself often wanting to delete a lot of it,” they said. “Sometimes it will emphasize symptoms that I don’t think are as major and make them almost on par with the other symptoms.”

This distinction between AI tools that assist with clinical decisionmaking and those that assist with administrative tasks is an integral part of discussions around how to oversee AI usage in healthcare, especially given that there is currently no standard procedure for how to deploy these systems into healthcare facilities.

According to Rahman, Highland utilizes a cautious and multistep approach to implementing new AI systems, which she stated is especially critical given that certain AI tools may have more variability than others.

“The nature of [a generative AI tool like Ambient Scribe] is using unstructured text and speech, and pulling that into a note where there’s opportunity to be creative,” Rahman said. “I might generate that note, and then if I generate it again, it’ll be slightly different. So there’s a higher risk involved with that.”

Comsti pointed to this variability as a key reason for transparency and regulation surrounding AI tools in patient care.

“Oftentimes, we don’t know what the inputs are with an AI tool. We don’t know how these decisions are being made. [Nurses] can’t double-check what’s happening with some of these AI tools, because we don’t have that baseline of regulatory supports to know that these tools are safe and effective,” Comsti said. “We know the FDA approves different types of medical devices that are being used in health care systems, but there isn’t necessarily that type of analysis for all the other types of AI tools being rolled out in health care.”

AB 2575’s introduction is also notable in that it comes at a time of sweeping national healthcare cuts, specifically with last year’s passage of H.R.1 or the One Big Beautiful Bill Act. H.R.1 is set to cut national healthcare spending by over $1 trillion across a decade, impacting widespread avenues to healthcare coverage such as Medicaid and Affordable Care Act marketplaces.

According to a 2023 economic analysis by the National Bureau of Economic Research, broader adoption of AI in the healthcare sphere could reduce total spending by 5-10% annually without compromising patient care standards. Opponents of AB 2575 argue that a bill that could potentially disrupt the deployment of these cost-effective AI systems is not compatible with current fiscal realities.

“The idea of curbing something at a time when there is an opportunity to mitigate the federal cuts without reducing the level of patient care or services [doesn’t make sense] to me,” Simon said.

Rahman shared a similar concern, noting that “The [revenue] cycle is actually probably our number one area of priority [when] we’re looking to use these tools. … We are absolutely concerned with the implications of H.R.1, and the low resources that we already have are going to be stretched even more thinly.”

She added that while she agrees with the intent of AB 2575, she has concerns that its combining of “low-risk” and “high-risk” AI tools in its regulation policies could cause more administrative burden than is manageable.

AB 2575 was moved to the suspense file by the Senate Appropriations Committee on August 3rd, which typically means that a bill’s cost effects are estimated to exceed a certain threshold. Several departments and institutions have estimated significant cost demands from AB 2575, according to a Senate Appropriations Committee report from July 31st.

“The idea of curbing something at a time when there is an opportunity to mitigate the federal cuts without reducing the level of patient care or services [doesn’t make sense] to me.”

This includes the California Department of Public Health’s estimation that annual ongoing costs will range between $1.8-3.6 million for state administration, and the Labor Commissioner’s Office’s estimation that costs would range anywhere from $3-7 million annually. Expenses would be used for various tasks, including hiring staff to verify that health facilities are operating under the bill’s requirements and tracking data from third-party vendors for each AI system used in these facilities.

Looking ahead, Ortega plans to amend the bill in a few key ways: making clear that employers can still discipline employees for reasons outside of overriding an AI system, clarifying that healthcare workers must still follow existing laws and procedures when deciding whether to override an AI system and accepting AI developers’ requested amendments that any reasonable precautions they took to address harm can be considered in the case of a lawsuit.

According to Ortega’s team, the Assemblymember also plans to pass further amendments should the bill make it out of the suspense file, which would allow employers to rely in good faith on information provided by developers regarding the bill’s transparency provisions.

AB 2575 is scheduled to be addressed by the Senate Appropriations Committee on August 13th.