Practice / Safety and trust

Trust is operational

Trust gets filed with the soft subjects, and then it turns up as a stalled rollout, a shipment nobody will release, and a team with no room for the thing you just asked them to absorb.

The claim#

Anything worth doing starts with somebody trying a thing before they know how it turns out. The research on when people will do that, across several fields that don’t talk to each other, keeps landing in the same place: only when they are not under threat.

The same shape shows up everywhere once you know to look. A team nods in the meeting and changes nothing. Somebody has had the new system for six months and still keeps their own spreadsheet. I don’t read either one as an attitude problem.

What it costs when it breaks#

Counterparty risk in global trade used to be assessed privately, inconsistently, and mostly on relationships. Two firms could look at the same supplier, reach opposite conclusions, and neither could show their work. While that gets resolved, goods sit.

I spent five years on that problem, and the thing that moved it was a way to score a counterparty consistently enough that two parties who had never met could compare one and agree on what they were looking at.

Where it was worked out first#

Donald Winnicott’s account of play is that a child tests reality by throwing things, trying things, and getting them wrong, and that this only happens when somebody is present in a way that lets it happen without compliance or anxiety. Take the holding away and play turns into performance.

Attachment research put a mechanism under it. A child who is confident somebody is available explores, and a child who isn’t stays close. In institutional settings Bowlby and colleagues described frozen watchfulness in children who had no secure base, who often did not play at all, and whose development was delayed as a result.

The same thing at work#

Amy Edmondson named this for organizations in 1999: psychological safety, the belief that a team is safe for interpersonal risk. Asking the question, admitting the mistake, saying the idea is wrong.

Her hospital finding is the one I think about most. Units with higher psychological safety reported more errors, not fewer, because people felt able to say so, which is the only condition under which anybody learns anything. And psychologically safe teams picked up new procedures faster.

It also settles an argument that comes up every time this is raised. Safety and high standards aren’t in tension. Both are required, and a team that has one without the other underperforms in a way that’s easy to misdiagnose.

What distrust actually spends#

Distrust inside a team draws on the same account the new work needs. People check, re-read, quietly redo something a colleague already did, wonder whether a thing landed. That’s capacity, and no plan has a line for it.

It runs the other way too. When something arrives past the edge of what somebody knows how to hold, it lands as a threat rather than an opportunity, and people drop to the layer where they protect themselves. Nobody creates from there.

Which is why the most common reason people give for holding back on a new tool is worry about their own job, and why no amount of training touches it. What gets filed as resistance is very often somebody deciding, correctly, that this week is not the week to be visibly bad at something.

What it asks of the thing you’re building#

Most of this turns into ordinary design decisions, and they’re the ones I check for.

It works on the first screen, populated, with nobody setting it up
A first attempt cannot break anything, and the way back is obvious
It shows what will happen before it happens
Nothing puts somebody in front of a colleague looking foolish
The old path keeps working while the new one is learned
Nobody has to ask for help to get started

And the harder half of safety#

The other half is that you can’t assume good intent. In anything built for children I learned to design for misuse directly, which means treating the person doing harm as somebody with goals and constraints of their own rather than as a moral category. You can’t remove a pathway you haven’t understood.

Eva PenzeyMoog does the clearest work I know on this for product teams, with an abuser archetype and a survivor archetype carried through design the way a persona is. She came into it from domestic violence education, which is the same route into product a lot of the best safety thinking has taken.

And Safety by Design, the framework from Australia’s eSafety Commissioner, puts the load in the right place in its first principle: the burden of safety should never fall solely upon the user.

Where this lands now#

Ben Shneiderman’s framing of human-centered AI is reliable, safe and trustworthy, and his argument is that high automation and high human control are not opposite ends of one dial. You can have both, and the designs that have both are the ones that raise what a person can do rather than routing around them.

Which is the same sentence as the rest of this page. Somebody who trusts the system will try things with it. Somebody who doesn’t will keep their spreadsheet, and they will be right to.