Skip to content

Why you're right to be careful with AI

Part 01 of 7

On this page

You got as far as the screen that asks for your mailbox, and you stopped.

Not because of a headline about a chatbot inventing a court case, and not because you think any of this is a fad. You have watched software do things in the last two years that you did not expect to see in your career. You stopped because the mailbox in question holds eleven years of other people’s affairs. Bank correspondence. A supplier dispute that is going to end badly. The message from a client’s spouse that arrived at the wrong moment and told you something you now have to be careful with. Whatever this product turns out to be, it is going to be that first.

The two things people say next are both useless. One is that it is fine, everybody is doing it, have you seen how thorough the security page is. The other is that you are the last person in the country still printing emails. Neither is an argument, and you did not have an argument either, which is why you closed the tab.

This series is about the distance between that closed tab and using the thing every day without thinking about it. It starts here because the first move is not a reassurance. It is agreeing with you.

What the pause is actually made of

Begin by clearing away what it is not.

It is not fear of software. You file returns through a portal, move client money on the strength of a browser session, and keep an entire practice inside a system you have never seen the inside of. You took all of that on without much ceremony, and you were right to. It was ordinary, and it worked.

Every one of those does exactly what it is told. You type a figure, it stores the figure. When it produces something wrong, the wrongness came from you, or from a defect somebody can be shown and made to repair. The software is a very fast set of instructions. It holds no view about anything.

What you are being asked to connect is different in one specific way, and naming it precisely matters, because the rest of this series follows from it. This one exercises discretion. It reads the mail and forms a view: this matters more than that, this can wait until Thursday, this is what thirty replies amount to, this is roughly what you would say back. Discretion is the part of the job clients are actually paying for. It is also the part that cannot be verified by testing, because there is no correct answer sitting somewhere to compare it against.

You have already lived with one small version of this for twenty years, and built exactly the right habit around it without ever putting the habit into words. The junk folder. A spam filter is software with an opinion about your mail. It is wrong in both directions, you know it is wrong in both directions, and so you glance through the folder before you empty it. Nobody taught you that. You worked out on your own that software making judgments needs watching in a way that software following instructions does not.

That instinct is correct. It is simply about to be asked to cover a great deal more ground.

Nobody else is on the hook

When you hand a piece of work to a person, something happens that you have probably never itemised. You get a second mind on it, and that mind has its own stake in how it turns out.

A bookkeeper who has been with you four years, asked to send the signed accounts across to the joinery client, will occasionally not send them. They will come and find you instead, because something about the request did not sit right, and half the time they cannot tell you what. That hesitation catches more than most of the checks you have written down anywhere. You never paid for it. It arrived free, as a side effect of the work being done by somebody who would rather not be the person who got it wrong.

Software has no stake. It does not get a bad feeling about a request and it will not come and find you. That is not a complaint about any particular product, and it is not something a better model repairs, because a thing that cannot be embarrassed cannot be reluctant. It means one of the two protections you were quietly relying on is absent, and noticing its absence before you rely on the arrangement is not timidity. It is reading the arrangement correctly.

The responsibility, meanwhile, stays exactly where it has always sat. That is true of delegating to people as well, and in a practice it comes with a rule book and a literature behind it, which whether client financial data is safe near AI works through properly. The plain version carries this far: whatever goes out goes out under your name, and the client affected by it will be talking to you.

The errors do not look like errors

You have spent a working life getting good at spotting when something is off, and most of that skill is not analysis. It is pattern. The reply that came back too fast. The number that is round when nothing about it should be round. The particular flatness of a sentence written by a colleague who is guessing and hoping. Human error leaks, faintly, most of the time. Tiredness has a texture. A guess reads differently from a fact.

A model’s mistakes have no texture at all. A summary that missed the only sentence in the thread that mattered arrives in the same even, unbothered prose as a summary that caught everything. No hedge, no smudge, no eleven at night in the phrasing. The signal your eye has been trained on for thirty years is not faint here. It is not there.

This is the actual reason the confident-nonsense stories land the way they do. Not that the machine was wrong, since you expect anything to be wrong sometimes, but that being wrong looked identical to being right, and the person leaning on it had no way in.

What follows is not refusal. It is that your check has to come from somewhere other than the surface of the output, because reading it and feeling satisfied is precisely the move that fails. You need to see what was done rather than what was produced. That is an unusual thing to ask a product for, and it is worth asking early, while you are still deciding.

It is not your risk to spend

Here is what separates this from every other tool decision you have made, and it is why the hesitation is sharper over client mail than it would be over your own.

Try something new on your own affairs and a bad outcome lands on you. Irritating, instructive, finished. The mailbox you are being asked to connect is yours only in the sense that it sits on your server. What is inside it belongs to people who did not choose this software, were not asked about it, and in most cases will never learn it was anywhere near their file.

Spending your own risk is a preference. Spending somebody else’s is a judgment made on their behalf, and there is nothing superstitious about wanting to make it slowly. The reflex that says wait is doing something quite precise. It is declining to convert a stranger’s exposure into your own convenience without looking first.

Which is right, and is also the exact spot where careful people get stuck for two years.

What a feeling cannot do

Being right to be careful and being careful well are different things, and almost all the cost of this decision lives in the gap between them.

A doubt that stays a feeling has three problems.

It answers a question that has no shape. Do I trust AI is not a question anyone can settle, any more than do I trust staff is. Which staff, doing what, with how much of it unwatched. You would never take somebody on by first deciding how you feel about people in general, and the only reason the same move seems reasonable here is that the subject is still unfamiliar enough to arrive as one lump.

It conceals its own price. Doing nothing reads as the safe end, because nothing happens and nothing changes. But the arrangement you are preserving is not free of risk. It is merely full of the risk you have grown used to: one person reading everything, ranking it in a quarter of a second on a name and a subject line, and being wrong about one of them on a Tuesday. That failure has been running quietly for years, it gives no sign either, and nobody writes an article about it. Weighing a new risk against zero is the most common error in this whole subject. The honest comparison is against the one already in the building.

And it never gets looked at again. A verdict arrived at in a single evening, out of a feeling, has no parts in it, so there is nothing inside it that can later turn out to be wrong. You will not revisit it, because there is nothing there to revisit. Two years on you are still declining a product as you last understood it, which by then is not the product.

Running through all three is the same defect. Vague caution cannot be satisfied. Nothing anyone shows you will move it, because it is not attached to anything showable. That is what makes it feel permanent and wise, and it is also what makes it useless, because it is protecting you from the decision rather than from the risk.

Six questions instead of one

Caution turns useful the moment it attaches to something a person could put in front of you. So take the lump apart. Underneath I don’t trust it sit six ordinary questions, and each one has an answer that can be demonstrated rather than asserted.

What can it see? The instant a mailbox is connected, before a single setting is touched and before anything at all has been done with it, some quantity of your clients’ correspondence is readable by something that is not you. How much, and what becomes of it afterwards, is the ground floor of the whole subject.

What can leave? Reading is contained. Anything that goes out is not, and once a message has been delivered it lives on hardware belonging to the recipient, past anybody’s reach. The distance between those two is the largest single fact here.

Can I find out what happened? Since the output carries no tells of its own, your check has to be a record: what was done, in what order, written so you can read it without a technical background.

How far can it go without asking? Not one answer but a set of them, and the part that matters is that the setting belongs to you and does not have to be the same for filing a receipt as for answering a client.

Does one thing I run stay clear of another? If the practice and the other venture share a person, they should not thereby share a view of each other’s mail.

Who else can get in? A mailbox is only as private as the route into it, which is what everything else rests on and is reliably the last thing anyone asks.

Six questions, and the useful property of the list is that a vendor can stay vague about is it safe indefinitely and cannot stay vague about these. How good the judgment is in accuracy terms sits alongside them as its own separate matter, taken up in AI email triage and whether you can trust it. The six are what the rest of this series works through, one at a time, in the order a practice actually meets them, using Point as the worked example throughout and the full inventory of what it does as the reference behind it.

Where it has to start

The order is not arbitrary, and the first question is not the one most people would pick.

You would expect to begin with how much the thing is allowed to do on its own, because that is where the fear lives and that is what the arguments are about. It is the wrong place. Every setting there is comes downstream of something that has already happened: the connection itself. Before anything has been ranked or drafted or sent, before you have chosen one thing, the mailbox has been opened to something. That is true on the first morning of any product of this kind, and it is true whether every dial is at the top or every dial is at the bottom.

It is also the question people are most confident they already know the answer to, which is usually the sign that the real answer is more specific than the assumption. Everything, is the assumption. Everything, forever, and probably fed into a model somewhere. Parts of that are true of some products. None of it is a single fact about all of them, and the difference between the versions is the difference between a decision you can make and a mood you have to live inside.

So that is where this goes next. Not how much it does. What it sees.

Join the private beta

We're onboarding a few teams at a time. Leave your email, confirm it once, and we'll send an invitation the moment a place opens.