AI feature resistance: users don't trust what they can't explain

I spent three weeks at Schneider Electric watching users fight a thermostat. Not literally, although it sometimes looked that way. The product was an IoT thermostat with an automated adjustment feature. It learned the building's patterns, factored in outdoor temperature, occupancy data, and energy pricing to set the temperature at the optimal level. On paper, it was the smartest thing in the room. In practice, every facility manager we observed was overriding it.

My first instinct was that they did not understand the value. That was me being arrogant, which I am moderately good at.

The facility managers understood the value perfectly well. But when the thermostat dropped the temperature by two degrees at 2pm on a Tuesday, they had no idea why. Was it reacting to the outdoor temperature? The occupancy sensor? An energy pricing spike? A bug? They could not tell. And when you cannot tell why a system did what it did, you do not trust it. You override it.

Users do not distrust AI because it is wrong. They distrust it because they cannot tell when it is right.

Why: the psychology underneath

This is not a technology problem. It is a human pattern that existed long before anyone put a language model inside a product.

Think about the difference between a pilot and an autopilot. The plane flies the same route regardless. But passengers feel safe when a human is at the controls and nervous when the autopilot is engaged, even though the autopilot's error rate is lower. The reason is not competence. It is legibility. You can see the pilot. You can infer intent from a human face in a way you cannot from a system running silently behind a panel.

The same dynamic plays out in every product shipping AI features right now. The AI might be accurate. But accuracy is not the currency of trust. Legibility is. If the user cannot form a mental model of why the system did what it did, the feature fails a test that has nothing to do with accuracy. It fails the trust test.

I call this the explainability threshold. Every AI feature has one. Below that threshold, the user cannot predict what the system will do next. And unpredictability, in a tool someone depends on for their work, is not exciting. It is threatening. A system the user cannot predict is a system the user will avoid, work around, or override. Not because it is wrong. Because it might be, and they have no way to check.

This is not the same as transparency in the way most product teams think about it. Showing a user that an AI model used fourteen data points and three neural network layers is not transparency. It is a spec sheet. The explainability threshold is simpler: can the user form a one-sentence explanation of why the system did what it did? If they cannot, you have not crossed the threshold.

How: the black box penalty

The penalty for failing to cross the explainability threshold is not complaints. It is silence. Users do not write angry support tickets about AI features they cannot understand. They quietly stop using them.

I have been mentoring a founder who learned this the slow, expensive way. She built an AI recommendation engine for a B2B procurement tool. The engine was genuinely good. It analysed purchasing patterns, supplier reliability data, and pricing trends to suggest optimal vendors for each order. The suggestions were right more often than the manual process. By every technical metric, the feature was a success.

But the usage data told a different story. Users were actively avoiding the AI suggestions. They would scroll past the recommendation panel, ignore the suggested vendors, and manually search for suppliers they already knew. Not because the recommendations were wrong. Because the interface offered no indication of why a particular vendor was being recommended. Was it price? Reliability? Delivery speed? Previous order history? The user had no idea. And in procurement, where a bad vendor decision can cost your company real money, "the AI said so" is not a reason anyone is willing to stake their reputation on.

She called it a trust problem. But I told her it was a design problem wearing a trust costume. The AI was doing its job. The interface was not doing its job, which was to make the AI's reasoning visible enough for a human to evaluate it.

That is the black box penalty. When users cannot see inside the system's reasoning, they apply a discount to everything it produces. Good recommendations get treated as suspect. Accurate predictions get double-checked manually. The AI feature becomes the colleague nobody trusts with real work, the one whose output everyone quietly redoes before passing it along.

What: making the invisible visible

At Schneider, we eventually did something embarrassingly simple. We added a one-line explanation to every automated adjustment. "Adjusting because outdoor temp dropped 5 degrees." "Reducing by 1 degree because occupancy is below 40%." "Shifting to off-peak energy rate."

That was it. One sentence. No machine learning explainability dashboards. No detailed model breakdowns. But it worked. Just a plain-language reason the user could read in two seconds and either agree with or override.

Override rates fell by more than half.

The facility managers did not need to understand the model. They needed to understand the decision. Those are very different things, and most product teams building AI features are confusing one for the other. They invest in model interpretability (a research problem) when what users actually need is decision legibility (a design problem).

But the founder I was mentoring had a harder version of the same challenge. Her recommendation engine could not always reduce its reasoning to a single factor. Sometimes the suggestion reflected a weighted combination of price, reliability, and delivery speed. The temptation was to show all the weights, which would have been accurate and completely useless. Nobody in procurement wants to interpret a radar chart before placing an order.

What worked was highlighting the primary factor. "Recommended: lowest cost with 98% on-time delivery." Not the full picture. But enough of the picture for the user to form a judgment. Enough to cross the explainability threshold.

The best AI feature is the one the user understands well enough to disagree with. That is the bar. Not accuracy. Not sophistication. The ability to look at what the system produced and say, "I see why you did that, but I think you are wrong." That disagreement is not a failure of the AI. It is proof the user trusts the system enough to engage with it honestly rather than ignore it entirely.

The design problem nobody assigned

Most product teams shipping AI features have a machine learning team and an interface design team. The machine learning team optimises for accuracy. The design team optimises for usability. But nobody is optimising for the gap between what the system knows and what the user can see. That gap is where trust lives and dies.

The teams getting this right treat explainability as a design constraint, not a feature to be added later. They ask, before shipping any AI-driven output: can a user form a one-sentence explanation of why they are seeing this? If the answer is no, the feature is not ready. Not because the model is wrong. Because the interface has not done its job.

Every AI feature your users ignore is a feature that failed the trust test before it ever got a chance to prove its accuracy. You cannot earn trust by being right. You earn trust by being understood.

The explainability threshold is not a nice-to-have. It is the price of admission.