Business, financial groups fear unintended consequences of proposed AI rules

Despite a successful push to pass a consensus artificial-intelligence regulation law earlier this year, financial lenders, healthcare providers and business groups are concerned that some specific rules Colorado is developing could burden companies in inordinate ways.

Those fears have come to the surface through letters sent to by those groups to the Colorado Attorney General’s office, which is writing the AI rules, and through a Monday rulemaking hearing in which both business and consumer advocates offered concerns. The AG’s office must get the rules by Jan. 1 — a looming deadline that also is causing concern among all groups that will be affected by the regulations.

Among the biggest fears are that the law’s allowance for human review of cases in which people were adversely affected by AI-influenced decisions could require unnecessary reconsiderations even when AI uses the same standards human lenders or doctors apply. And if national companies are subject to that review standard — and the record-keeping requirements that go with it — for every Colorado-based job applicant they consider, some may simply decide they won’t accept Colorado applicants, several groups said.

“This rule comes at a critical time for Colorado’s economic competitiveness,” said Brittany Morris Saunders, president/CEO of the Colorado Technology Association, alluding to reports that regulations and costs are hurting Colorado’s position versus other states. “Colorado should not have to choose between protecting consumers and leading in innovation.”

What the new AI law governs

That same sentiment was what led legislators to spend two years working to replace a 2024 law that represented the country’s most comprehensive AI regulations but was so tough that tech groups warned it would drive AI development out of state. What Gov. Jared Polis signed in May was a law that requires AI developers to provide documentation to deployers about what their systems do, requires deployers to inform consumers when AI has a meaningful role in making consequential decisions and permits consumers to appeal adverse decisions for human review when commercially reasonable.

Those consequential decisions could involve employment applications, financial-aid requests, healthcare decisions, housing determinations and other choices, and AG’s office leaders have been working to pin down exactly what is covered and how appeals will work. While numerous business leaders expressed at Monday’s hearing that they are pleased with changes that have been made to rules throughout the process, they believe some proposed wording leaves open the possibility of unintended consequences.

No group of professionals emphasized that more than a series of representatives for banks, credit unions and other financial-service groups who cautioned that the rules’ definition of “material influence” that AI may have on consequential decisions is too broad.

That definition presumes that automated decision-making technology had a material influence on a decision if it reaches the same outcome as a human making the final decision, said Michael Blanco, tech policy director for the U.S. Chamber of Commerce. That’s one of several provisions of the rules that appear to exceed the statutory text of the law Gov. Jared Polis signed, he said.

Will the law trigger too many human reviews?

Such material influences should not trigger consumer warnings and human-led reviews if they are simply following policies set out by the company, added Ben Metzger, vice president of regulatory affairs for Wings Credit Union. For example, if a potential customer must have a minimum credit score to qualify for a loan, it should not trigger a review if the AI simply acknowledges they don’t have that floor-level credit score.

Danielle Arlowe, senior vice president for the American Financial Services Association, noted that her sector has worked for decades to come up with formulas that take human bias out of the decision-making process for financial applications. Requiring lenders to have to review the claims of anyone who is upset that they didn’t get a loan threatens to add the risk of bias back into the process, she said.

It also could require an untenable number of reviews from financial-service companies, warned John Pitts, a vice president for Affirm, a national underwriting firm. His company makes 37.8 million credit decisions annually and declines about 15% of credit applicants, meaning that it could have to do 11 reviews per minute under Colorado’s law.

The law requires human review only when it is “commercially reasonable,” but that vague standard concerns both companies worrying it will require a lot of reassessments and consumer groups fearing it will allow businesses to avoid reviews in most circumstances. Kara Williams, an attorney for the Electronic Privacy Information Center, said the repeated use of that term and of “technically feasible” in the rules will minimize transparency and the number of human reviews that take place.

Could companies limit Colorado job applicants?

But it’s not just reviews of loan applications that will be subject to this potentially vague standard. Job applications are considered consequential decisions as well, and there are questions about how many applicants who are vetted in part by AI but who are not granted an interview or position will be able to seek human review under the new laws.

The law applies specifically to employers doing business in Colorado, but that wording leaves open the door to whether the consideration of a Colorado-based applicant for a remote job or position in another state falls under that language, said William Martinez, a counselor to the multi-industry State Privacy and Safety Coalition. Considering that the law also requires yearslong retention of records, some out-of-state employers could choose not to consider Colorado candidates in order to avoid being subject to its regulations, he said — just what happened after Colorado passed a 2019 law requiring employers advertising to state residents to include pay ranges in their notices.

Regardless of how out-of-state companies are considered in regard to the regulations, the law could be cumbersome for any company that is hiring if it must send notices of AI decision-making and grant review to all applicants who did not get the job, said Ben Picotte, Colorado Chamber communications and advocacy coordinator.

“Notices to every applicant for high-volume hiring would create significant burdens for employers,” Picotte said.

Healthcare also impacted by AI law

Healthcare providers also expressed concern Monday that the new rules on notifications could clash with federal Health Insurance Portability and Accountability Act privacy guidelines around medical decision-making. And the state needs to provide clarity on whether routine clinical support tools considered by AI systems are considered material influence, since those are the same tools doctors would use without the help of AI, said Michelle Gelroth of Aspen Valley Health, speaking for the Colorado Hospital Association.

With all these concerns piling up, numerous individuals asked the AG’s office either to delay enforcement of the new rule for a year or to consider good-faith compliance as compliance during the first year. Businesses need time to update systems, train their staff and develop disclosure materials, argued Kouri Marshall, senior director of state and local policy for the tech-focused national Chamber of Progress.

The biggest pushback to all those requests came from consumer-advocacy groups, which pleaded for the AG’s office to enforce the new law in a way that protects Coloradans from potential discriminatory practices of AI. Some expressed concern about changes in the proposed rules that already were made between the August and October drafts of those regulations, including the removal of mandatory stays of decisions while reviews are undertaken and revisions to the standard of commercial reasonableness.

Consumer advocates seek greater protections from AI

Williams, of the Election Privacy Information Center, said too many of the draft rules continue to allow deployers to defer responsibility for their decisions to developers, which are harder for consumers to contact and question regarding their adverse outcomes. And Chris Bryant, a senior counsel with the National Student Legal Defense Network, said more transparency is needed when higher-education institutions choose to offer students financial aid that those learners consider to be insufficient.

“I think the department is under real pressure to weaken these rules, and I’m asking you to hold the line,” said Vincent Le, vice president of policy for the group Tech Equity. “The most recent draft should be the floor.”

Officials with Attorney General Phil Weiser’s office will consider all the concerns as they work to come back with a final draft of the new AI rules in order to get the law implemented in the next two months.