Growing use of AI tools has some medical providers on edge

Although its impact has reached nearly every workforce, AI in healthcare remains an especially pertinent issue as healthcare facilities rapidly implement AI tools into patient care. This year’s legislative docket saw a number of bills addressing AI in healthcare, which raised questions surrounding the rights of patients, healthcare workers and hospitals as AI becomes commonplace in the healthcare sphere.

When it comes to advocating for healthcare workers’ rights amid AI implementation, the California Nurses Association has been particularly active. CNA has sponsored a number of AI-related bills this legislative cycle, including Assemblymember Liz Ortega’s (D-Hayward) AB 2575, which would protect healthcare workers’ right to override an AI system and enforce transparency around these systems in health facilities, and Assemblymember Mia Bonta’s (D-Oakland) AB 1979, which would prohibit health facilities from deploying AI systems to replace clinicians’ judgment. (NOTE: As of this writing both measures are awaiting action from Gov. Gavin Newsom.)

For CNA Government Relations Director Carmen Comsti, the concern is not with the introduction of new technology to the healthcare space, but instead the threat of nurses and other healthcare workers being excluded from these discussions.

“What we’ve been demanding on new technology for decades is that any new technology needs to be validated as safe and effective before they are implemented,” Comsti said. “Healthcare workers who are the ones implementing and using these tools in patient care should have a seat, not just at the table but in the driver’s seat, in terms of that decision on whether a tool is used in the first place.”

Ortega stated that prior to introducing AB 2575 in February she had been hearing increasing concerns from healthcare workers surrounding AI in her roles as both the Chair of the Assembly Labor and Employment Committee and a member of the Privacy and Consumer Protection Committee.

“What I was hearing from healthcare workers and nurses is that they were being pushed to use AI more and more, and the expectation from their bosses was to follow the output,” she said. “And when AI gets it wrong, healthcare workers are put in a double bind. Follow the machine, and then they get blamed, or override the machine, and then they risk retaliation.”

However, the California Medical Association and the California Hospital Association were among a number of healthcare facilities to oppose AB 1979 and AB 2575. CMA and CHA cited primary concerns over the bills increasing the burden of AI implementation and enforcing strict oversight once the systems are in place.

“Healthcare workers who are the ones implementing and using these tools in patient care should have a seat, not just at the table but in the driver’s seat, in terms of that decision on whether a tool is used in the first place.”

“Rather than deploy a tool that is broadly helpful and helps outcomes on the whole, you’re creating a very chilling environment where a single error or a single bias would be a more powerful counterweight to the potential benefits of that tool,” said David Simon, CHA Senior Vice President of Communications.

AI systems have been used to assist with patient care in recent years in a variety of ways, from clinical notetaking to generating possible diagnoses and treatment plans.

For Sarah Rahman, a primary care internal medicine physician and Chief Medical Information Officer at Highland Hospital in Oakland, the implementation of AI into her practice has proved extremely useful — particularly Ambient Scribe, an AI tool that listens to doctor-patient conversations and generates clinical notes.

“The reduction in cognitive burden actually allows me to give better clinical care to the patient, improves my documentation, allows me to update the problem list [and] allows me to ask the patient a question that maybe I wouldn’t have had time for about their healthcare maintenance,” she said.

An emergency department physician at Kaiser Permanente, who requested anonymity due to Kaiser’s media policies, agreed that the AI listening tools have proved helpful in reducing administrative burden, particularly in a fast-paced emergency department. But they also find themselves editing the tool often, noting that it needs strict physician oversight.

“Occasionally it’s great and I don’t have to make changes. But typically, I have to tweak a few things that it gets wrong,” they said.

As for the recommendation aspect of the tool, which suggests potential diagnoses or treatment plans, the physician considers it to be far less useful.

“I find myself often wanting to delete a lot of it,” they said. “Sometimes it will emphasize symptoms that I don’t think are as major and make them almost on par with the other symptoms.”

This distinction between AI tools that assist with clinical decision-making and those that assist with administrative tasks is an integral part of discussions around how to oversee AI usage in healthcare, especially given that there is currently no standard procedure for how to deploy these systems into healthcare facilities.

According to Rahman, Highland utilizes a cautious and multistep approach to implementing new AI systems, which she stated is especially critical given that certain AI tools may have more variability than others.

“The nature of [a generative AI tool like Ambient Scribe] is using unstructured text and speech, and pulling that into a note where there’s opportunity to be creative,” Rahman said. “I might generate that note, and then if I generate it again, it’ll be slightly different. So there’s a higher risk involved with that.”

Comsti pointed to this variability as a key reason for transparency and regulation surrounding AI tools in patient care.

“Oftentimes, we don’t know what the inputs are with an AI tool. We don’t know how these decisions are being made. [Nurses] can’t double-check what’s happening with some of these AI tools, because we don’t have that baseline of regulatory supports to know that these tools are safe and effective,” Comsti said. “We know the FDA approves different types of medical devices that are being used in healthcare systems, but there isn’t necessarily that type of analysis for all the other types of AI tools being rolled out in healthcare.”

The recent legislative focus on AI regulation in healthcare is also notable in that it comes at a time of sweeping national healthcare cuts, specifically with last year’s passage of H.R.1 or the One Big Beautiful Bill Act. H.R.1 is set to cut national healthcare spending by over $1 trillion across a decade, particularly impacting healthcare coverage avenues such Medicaid and Affordable Care Act marketplaces.

“The idea of curbing something at a time when there is an opportunity to mitigate the federal cuts without reducing the level of patient care or services [doesn’t make sense] to me.”

According to a 2023 economic analysis by the National Bureau of Economic Research, broader adoption of AI in the healthcare sphere could reduce total spending by 5-10% annually without compromising patient care standards.

Simon argued that enforcing AI regulations, which could potentially disrupt the deployment of these cost-effective systems, is not compatible with current fiscal realities.

“The idea of curbing something at a time when there is an opportunity to mitigate the federal cuts without reducing the level of patient care or services [doesn’t make sense] to me,” Simon said.

Rahman shared a similar concern, noting that “The [revenue] cycle is actually probably our number one area of priority that we’re looking to use these tools. … We are absolutely concerned with the implications of H.R.1, and the low resources that we already have are going to be stretched even more thinly.”

But according to the Kaiser physician, human oversight is still crucial when it comes to responsibly administering patient care, especially with many AI tools still in early stages.

“[The technology is] very early, I would say,” they said. “The doctor still needs to come into the room. The doctor needs to be the one eliciting the history. You can’t have a robot doing that and prompting, even if you’ve got some language model. Patients aren’t computers. … They have emotions that are affecting the whole encounter, and that requires time, and it requires being there in person.”

For Rahman, this intersection between prioritizing clinician judgment and patient safety while still supporting AI innovation is one that she is constantly considering.

“We want [AI implementation] to be done well in a responsible way,” she said. “So, how do you target it in a way that promotes safe implementation?”

Olivia Bye is a junior at Tufts University. She was previously an intern in the Capitol Weekly internship program.

The next round of Capitol Weekly internships will begin in January 2027. To apply, please submit two writing samples, a letter of recommendation from a supervisor or professor, your resume and a cover letter to [email protected] by October 16, 2026.