What AI Should Never Decide in Care: Human Judgement in an AI Powered Care Quality Platform

Eleven principles, and one of them does most of the work. In its statement on AI, CQC sets out principles for providers who use AI, and the first is short enough to memorise: “AI can enhance, but not replace human decision making”.

Most readers nod at that sentence and move on. The harder part comes later, in a real week, with forty findings on screen, a visit approaching and a tool offering to sort the lot by risk. A well-built AI powered care quality platform has real work to do in that moment. It also makes the sentence easy to break without anybody noticing.

The usual line, that AI should never replace people, needs no defence here. Nobody sensible disagrees. This article asks something narrower. What has to be true for the review a person carries out to count as judgement? And which parts of the work of a quality platform can go to AI without the decision going with them?

Who holds the pen

Two jobs are easy to confuse. The pen is the drafting job: spotting a pattern, sorting a pile, summarising a hundred comments, suggesting the wording of an action. The signature is the deciding job: choosing what a pattern means, ranking what matters, closing the action, answering an inspector. AI can hold the pen. A named person holds the signature.

The government guidance on using AI in adult social care describes the pen side well, in its section on auditing tools. Some services use AI to watch audits across several sites. A member of staff completes a checklist, and the system writes a summary of what was done and flags what is still outstanding. Nothing in that decides anything. It prepares the ground for somebody who will.

Where the pen was enough, and where it was not

The same guidance tells a short story that shows the gap. In a residential care home, a digitised incident reporting system pointed to several residents falling. AI analysis of the data found the falls clustered in one place at one time of day. Staff went to look and found strong sunlight through a window, affecting vision and leaving people unsteady. Blinds went up. Falls and hospital admissions dropped.

The AI saw where and when. People found why, and chose what to do. A pattern alone would never have justified buying blinds, and blinds bought without the pattern would have been a guess.

A division of labour

In an AI powered care quality platform, the split might look like the table below. It is illustrative, and each service decides its own. The thing to notice is that every row ends with a person.

TaskWhat AI can doWhat a named person does
Repeat findings across auditsFlag the pattern across servicesDecide whether it is real and what causes it
Survey commentsGroup comments into themes and sentimentRead a sample behind each theme and decide what it means
Rating a finding by riskSuggest a rating from past patternsConfirm or change it, and record why
Drafting an actionPropose wording and a likely priority with deadlineAgree the action and deadline, choose the owner
Closing an actionShow whether the previous actions heldA mini RCA before closing
Anything about same repeated mistakesHighlight them at question level quicklyFollow the procedures of the service and decide
Reporting to CQC or answering an inspectorAssemble evidence and a narrativeCheck every line and answer for it

Time, information and permission

Clicking a button does not turn a review into judgement. The CQC list of risks in its statement on AI suggests what else has to be in place.

Start with time. Among its risks, CQC lists that human oversight can add to workload. A review squeezed into a gap between jobs turns into a glance, so reviews belong in the schedule with a named owner, and they count as work.

Information comes next. CQC also lists staff who lack the trust, confidence or training to check outputs, and AI that states untrue things as fact with complete authority. A flag with nothing behind it can only be believed or ignored. The reviewer needs the underlying audit, comment or record within a click or two. On InvictIQ the live dashboard lets a user move from a compliance score into the specific report and evidence behind it, which is the kind of access that makes a review real.

Permission is the awkward one. CQC lists de-skilling as a risk, and a likely route to it is a workplace where overriding the tool feels like an error. People stop overriding. Record overrides as a normal and welcome result, since each one shows somebody thinking. A reasoned view, not a regulatory position: if the suggestions from a tool go months without a single override, the question is whether the tool is flawless or the review has turned ceremonial.

What the regulator and government say

The CQC statement is worth reading in full. Counting from the page, it sets out eleven principles for providers and twelve risks. CQC says it does not assess or approve specific technologies, and that the absence or presence of AI does not predict a specific rating. It ties the principles to existing regulations, Regulation 17 on good governance among them, which in this context means adhering to regulation, managing risk effectively and monitoring outputs. A data protection impact assessment appears among the principles as the way to assess and record risks to privacy.

The Department of Health and Social Care guidance gets more practical. Providers should always include human review of AI-generated outputs, checking for accuracy, fairness and fit with organisational standards, and where AI is used in care planning a member of care staff must review the plan. The guidance suggests an AI policy even if the policy is that staff cannot use AI for work, and it gives five questions for the policy to answer. Who is responsible for what the AI does? How is its use overseen? Who is accountable? How will adoption go? Is a framework or decision tool used to judge whether AI is right for the task? On data, it advises against putting personally identifiable information into free or online tools, because without a contract the provider cannot guarantee how the data is handled.

Who answers for it

Legal responsibility for how a service is run sits with the registered persons, under the Health and Social Care Act 2008 and its Regulations. Regulation 7 sets fitness requirements for registered managers, including the qualifications, competence, skills and experience to manage the regulated activity. A tool does not alter any of it.

A tool cannot be registered with CQC. It cannot explain itself to an inspector, and it cannot be held responsible for a missed risk. A person can. That is part of why CQC lists unclear accountability and liability among its risks, and why its principles call for clear mechanisms for dealing with harm caused by AI.

How InvictIQ draws the line

InvictIQ describes its approach as technology that enhances human expertise and does not replace it. According to this approach, the AI brings forward what matters, automates repetitive tasks and turns data into analysis, and AI analysis of survey comments surfaces sentiment and themes. Findings can then become assigned actions that can be escalated when they stall, and the reports and evidence sit behind the dashboard, so a person can look behind any score.

A responsible AI powered care quality platform keeps that evidence one click away, leaves the decision to people and makes the review visible. Any provider adopting an AI tool, InvictIQ included, should still complete a data protection impact assessment and decide who reviews what. The features page and the modules page show what each part does, and an earlier article on how governance evidence is judged shows where these reviews fit in an inspection.

Frequently asked questions

Can AI make decisions about care in a care home?

The first CQC principle says AI can enhance but not replace human decision making. The government guidance adds that where AI helps with care planning, a member of care staff must review the plan for accuracy.

Does using AI change the responsibilities of the Registered Manager?

No. Legal responsibility for how the service is run sits with the registered persons under the Health and Social Care Act 2008 and its Regulations. A tool changes how work is prepared, not who answers for it.

Will CQC rate a service differently for using AI?

CQC states that the absence or presence of AI does not predict a specific rating, and that it does not assess or approve specific technologies. It looks at whether the service meets the regulations, however the work is done.

Does a service need an AI policy?

The government guidance says all organisations should consider having one, even if the policy is that staff cannot use AI for work. It suggests the policy sets out how different roles interact with AI and includes a review checklist.

What does human review look like in practice?

It means a named person with time to review, access to the evidence behind each output, and permission to disagree. The government guidance recommends defining the human role and setting review guidelines so that human processes keep pace with the AI.

Keeping the signature human

The sentence from the regulator is easy to agree with and easy to wear away, one rushed click at a time. The test of any AI powered care quality platform is whether a busy manager can see the evidence, has time to weigh it and feels free to say the tool is wrong. Where all three hold, AI holds the pen and a person signs.

To see how InvictIQ keeps evidence behind every score and decisions with people, book a demo or read about who InvictIQ helps. AI tools are designed to support clinical and managerial judgment, not replace the accountability of the Registered Manager unde

More Articles

"A tool cannot be registered with CQC. It cannot explain itself to an inspector, and it cannot be held responsible for a missed risk. A person can."

Ready to Join Next Generation of Care Quality?

Smarter tools. Safer care. Stronger outcomes powered by AI.