You got as far as the screen that asks for your mailbox, and you stopped.
The headline about a chatbot inventing a court case is somebody else’s story. You’ve watched software do things in the last two years you didn’t expect to see in your career, so you know this is real. You stopped because of what the mailbox holds. Eleven years of other people’s affairs. Bank correspondence. A supplier dispute that’s going to end badly. The message from a client’s spouse that arrived at the wrong moment and told you something you now have to be careful with. Whatever this product turns out to be, it’s going to be that first.
Two answers come back at this point, and both of them are useless. One says it’s fine, everybody’s doing it, have you seen how thorough the security page is. The other says you’re the last person in the country still printing emails. Neither of them is an argument. You had a feeling rather than an argument yourself, which is why you closed the tab.
This series is about the distance between that closed tab and using the thing every day without thinking about it. It starts by agreeing with you.
What the pause is actually made of
Look at what the pause is made of.
Software itself is old ground for you. You file returns through a portal. You move client money on the strength of a browser session. You keep an entire practice inside a system you’ve never seen the inside of. You took all of that on plainly, and you were right to. It was ordinary, and it worked.
Every one of those does exactly what it’s told. You type a figure, it stores the figure. When something comes out wrong, the wrongness came from you, or from a defect somebody can be shown and made to repair. That kind of software is a very fast set of instructions. It holds no view about anything.
What you’re being asked to connect works differently, in one specific way, and naming it matters, because the rest of this series follows from it. This one exercises discretion. It reads the mail and forms a view. This matters more than that. This can wait until Thursday. This is what thirty replies amount to. This is roughly what you’d say back. Discretion is the part of the job clients are actually paying for. It’s also the part testing can’t settle, because there’s no correct answer sitting somewhere to compare it against.
You’ve already lived with a small version of this for twenty years, and built exactly the right habit around it without ever putting the habit into words. The junk folder. A spam filter is software with an opinion about your mail. It’s wrong in both directions, you know it’s wrong in both directions, and so you glance through the folder before you empty it. You worked that out on your own. Software that makes judgments needs watching, in a way the instruction-following kind never did.
That instinct is right. It’s about to be asked to cover a great deal more ground.
Nobody else is on the hook
When you hand a piece of work to a person, you get something you’ve probably never itemized. A second mind on it. That mind has its own stake in how it turns out.
A bookkeeper who’s been with you four years takes the signed accounts to send across to the joinery client. Sometimes they stop. They come and find you instead, because something about the request sat wrong, and half the time they can’t tell you what. That hesitation catches more than most of the checks you’ve written down anywhere. You never paid for it. It arrived free, as a side effect of the work being done by somebody who’d hate to be the person who got it wrong.
Software has no stake. It takes the request and does the thing. That’s true of software in general, and a better model keeps it true, because a thing that can’t be embarrassed can’t be reluctant. So one of the two protections you were quietly relying on is missing here. Noticing that before you lean on the arrangement is reading the arrangement correctly.
The responsibility stays exactly where it’s always sat. That holds for delegating to people too, and in a practice it comes with a rule book and a literature behind it, which whether client financial data is safe near AI works through properly. The plain version carries this far. Whatever goes out goes out under your name. The client affected by it will be talking to you.
The errors do not look like errors
You’ve spent a working life getting good at spotting when something’s off, and most of that skill is pattern. The reply that came back too fast. The number that’s round when everything about it should be uneven. The particular flatness of a sentence written by a colleague who’s guessing and hoping. Human error leaks, faintly, most of the time. Tiredness has a texture. A guess reads differently from a fact.
A model’s mistakes arrive smooth. A summary that missed the only sentence in the thread that mattered reads exactly like a summary that caught everything. The same even, unbothered prose. Level at eleven at night, level at nine in the morning. The signal your eye has been trained on for thirty years is absent here, rather than faint.
That’s why the confident-nonsense stories land the way they do. You expect anything to be wrong sometimes. What gets people is that being wrong looked identical to being right, and the person leaning on it had no way in.
So your check has to come from somewhere other than the surface of the output. Reading it and feeling satisfied is precisely the move that fails. You want to see what was done, rather than what was produced. That’s an unusual thing to ask a product for, and it’s worth asking early, while you’re still deciding.
It is not your risk to spend
Here’s what separates this from every other tool decision you’ve made. It’s why the hesitation is sharper over client mail than over your own.
Try something new on your own affairs and a bad outcome lands on you. Irritating, instructive, finished. The mailbox you’re being asked to connect is yours in one sense only. It sits on your server. What’s inside it belongs to other people. The software was your choice rather than theirs. Nobody asked them about it, and in most cases they’ll never learn it was anywhere near their file.
Spending your own risk is a preference. Spending somebody else’s is a judgment made on their behalf, and wanting to make it slowly is sound. The reflex that says wait is doing something quite precise. It wants a look at the arrangement before a stranger’s exposure turns into your convenience.
That’s right. It’s also the exact spot where careful people get stuck for two years.
What a feeling cannot do
Being right to be careful and being careful well are two different things. Almost all the cost of this decision lives in the gap between them.
A doubt that stays a feeling has three problems.
It answers a question with no shape to it. Do I trust AI settles about as well as do I trust staff does. Which staff, doing what, with how much of it unwatched. You’d take somebody on by looking at the person and the job, rather than by first settling how you feel about people in general. The same move seems reasonable here only because the subject is still unfamiliar enough to arrive as one lump.
It hides its own price. Doing nothing reads as the safe end, because the days look the same. The arrangement you’re preserving carries risk too. It’s the risk you’ve grown used to. One person reads everything. They rank it in a quarter of a second on a name and a subject line. On a Tuesday they get one of them wrong. That failure has been running quietly for years, it gives no sign either, and nobody writes an article about it. Weighing a new risk against zero is the most common error in this whole subject. The honest comparison is against the risk already in the building.
And it stays where you left it. A verdict that came out of a feeling in a single evening has no parts in it. So nothing inside it can later turn out to be wrong, and nothing in it asks to be looked at again. Two years on, you’re still declining the product as you last understood it. By then the product has moved.
The same defect runs through all three. Vague caution stays exactly where it is, whatever anyone shows you, because it floats free of anything showable. That’s what makes it feel permanent and wise. It’s also what makes it useless. It protects you from the decision rather than from the risk.
Six questions instead of one
Caution turns useful the moment it attaches to something a person can put in front of you. So take the lump apart. Underneath I don’t trust it sit six ordinary questions. Each one has an answer somebody can demonstrate rather than assert.
What can it see? The instant a mailbox is connected, some quantity of your clients’ correspondence is readable by something other than you. That’s true before a single setting is touched, and before anything at all has been done with it. How much, and what becomes of it afterwards, is the ground floor of the whole subject.
What can leave? Reading stays inside. Anything that goes out keeps going, and once a message has been delivered it lives on hardware belonging to the recipient, past anybody’s reach. The distance between those two is the largest single fact here.
Can I find out what happened? The output carries its own tells poorly, so your check has to be a record. What was done, in what order, written in plain language for an owner to read.
How far can it go without asking? That has a set of answers rather than one. The setting belongs to you, and filing a receipt can sit at a different place on it than answering a client.
Does one thing I run stay clear of another? The practice and the other venture share a person. Their mail can still stay apart.
Who else can get in? A mailbox is as private as the route into it. Everything else rests on that, and it’s reliably the last thing anyone asks.
Six questions, and the useful property of the list is that a vendor can stay vague about is it safe forever and has to get specific about these. How good the judgment is in accuracy terms sits alongside them as its own separate matter, taken up in AI email triage and whether you can trust it. The six are what the rest of this series works through, one at a time, in the order a practice actually meets them. Point is the worked example throughout, with the full inventory of what it does as the reference behind it.
Where it has to start
The order is deliberate, and the first question is one most people would put later.
You’d expect to start with how much the thing is allowed to do on its own, because that’s where the fear lives and that’s what the arguments are about. The first question sits earlier than that. Every setting there is comes downstream of something that’s already happened. The connection itself. Before anything has been ranked or drafted or sent, before you’ve chosen one thing, the mailbox has been opened to something. That’s true on the first morning of any product of this kind. It holds with every dial at the top, and it holds with every dial at the bottom.
It’s also the question people are most confident they already know the answer to, which is usually the sign that the real answer is more specific than the assumption. Everything, is the assumption. Everything, forever, and probably fed into a model somewhere. Parts of that are true of some products. Each product answers that its own way, and the difference between the versions is the difference between a decision you can make and a mood you have to live inside.
So that’s where this goes next. What it sees, on the first morning, before anything else.