How To Decide What To Automate With AI For Your Team & Customers | In The Loop Episode 41

Every week, new AI features and automations appear, but the real question isn't whether you can automate something—it's whether you should.

Your CEO, the board, sales & marketing teams, and your customers all want change. Organizations right now are under pressure to transform but most advice is either abstract or pure hype.

Today, I wanted to do something different. I'll walk through a framework—from the Department of Product Substack—that helps you decide where to start with AI automation in your product. We'll cover the concept of verifiability, how to score opportunities across safety, volume, and ease of validation, and why most AI features should be assistants, not full autopilots.

This is In The Loop with Jack Houghton. I hope you enjoy the show.

‍

‍

‍

Why automation is AI's biggest superpower right now

The conversations we have with prospects and customers every week are staggering. For them, deciding how or where to automate first isn't easy.

We're in a unique position—our product development process is AI-first from product through to engineering and product marketing. Our entire product enables technology companies to have a conversational interface with their customers and users. We sit at the intersection of understanding this opportunity and taking action.

A recent post from the Department of Product opened with a reference to Benedict Evans, a tech analyst. In his latest presentation, Evans said that automation is AI's biggest superpower right now. While it's not as exciting as AGI, automation alone is a big deal—it's the boring cousin of artificial general intelligence. He makes a good point: barcodes and databases weren't sexy, but they transformed retail. We're seeing the same thing happen with AI and AI features in products right now.

The wins aren't necessarily coming through big model breakthroughs every six months. They're coming from the application layer—the technology people interact with that takes away frustrating and tedious tasks from their day-to-day lives.

But with that comes a key tension: just because you can automate something doesn't mean you should, because the outputs from these automation efforts are often imperfect. That judgment call—what to automate for your users or your team—is what this episode covers today.

At Mindset AI, we’re building an AI agent (to be released in 2026) that helps people with this, as these are new ways of looking at technology and processes. People have to understand how AI and agents work, what they're good at and what they're not, and how to rip apart an entire process or product to improve it, with those strengths and weaknesses in mind.

‍

‍

The verifiability principle

Ilya Sutskever, OpenAI’s co-founder, said: a good candidate for AI automation is a task that is easily verifiable. The more verifiable a task or job is, the more amenable it will be to automation. It's a simple principle, but powerful because the core question is: can you tell if it's worked?

There are many tasks where an individual can make that judgment call themselves. But big enterprise organizations will be obsessed with the verifiability principle.

If you think about it from a user's perspective, if AI auto-categorizes all their expenses, they can verify them instantly. They can say whether it was right or wrong. But if AI writes personalized recommendations, it's hard to know whether that was right or wrong. They're in subjective territory.

If the AI isn't good at doing things that are easy to classify as right or wrong, then it might not be a good candidate for automation. The nuance here is how good the process is around that AI. However, this creates tension because most of the things that are hard to verify are also where you're going to get the most significant gains from automation.

What we learned at Mindset AI, is that you have to build processes around these things because judgment becomes the key success criterion of a human in a modern-day process. We need people who can make judgment calls on the quality of an AI output, and that's going to give you a good indication of how much human-in-the-loop is needed here. Most of the time, an AI agent with a bunch of tools—things it can do in the world—and the ability to reason well can carry out most tasks to a high standard.

If you want to automate it fully and “forget it,” and are happy to say "Go do that thing and never speak to me," then the verifiability principle becomes essential. Otherwise, you need a human in the loop. It's simpler for your team because you are a member of both the team and the process.

‍

‍

Deciding what to automate for users vs. teams

This gets even harder when you're trying to think about what type of automation experience you should offer to your customers. The article goes deeper into whether this should be a product feature or something just for internal process automation.

To help decide whether you want to create a product for your users or your customers inside your existing product, you should be thinking about:

• What triggers the automation? Is it a user action? Is it an event that happens on the platform? Someone hovers over or clicks on something—is it a particular signal?
• What relevant data will be collected, and what processing will the AI do? Is it going to be classifying, generating, writing, creating, or predicting something?
• What is the logic or instructions on how it decides what happens next? Should it auto-execute that task? Should it come back and ask the user?
• Finally, what is the end action or outcome?

At Mindset AI, we run sessions with companies to create job descriptions for their AI ideas. What are the inputs? What are the user questions? What situations will trigger it? When will it appear and say, "I'm here to do something"? What skills does this AI need? Does it need access to user memory or data? Does it need access to some other system in a third-party application?

A good example here is predictive shopping carts. You could autofill a shopping cart based on the user's behavior or actions in the application. Or you could show them suggestions and let them approve each one. Or a middle ground: auto-add high-confidence items, and then suggest medium to lower confidence items to the user.

All of that could happen based on a trigger in the application, the AI processing data to generate or suggest things, and then either bringing in a human or doing it itself. Those design decisions will determine whether the user feels helped or bulldozed.

‍

‍

The three-dimensional framework: safety, volume, and verifiability

How do we understand whether something is a candidate for automating for your users or team? The framework suggested in the Department of Product article rates opportunities across three dimensions and scores them from 1 to 3.

‍

1. Safety

Safety is about user impact. Take the shopping app example: if the AI suggests the wrong product substitution and the user catches it before checkout, it's annoying, but it's fine. That gets a simple safety score. If it auto-approves items in the cart, and costs people time or money—or potential risk to their health or safety—, then you've damaged trust, and it's not the safest of ideas.

Score three for anything simple, easy to catch, low-impact, low-stakes error. Score one for anything that could hurt a user before they can intervene.

‍

2. Volume

Volume is about whether the automation matters enough to build. If users do this task constantly—every session, every day—and it takes a lot of time, it's an excellent opportunity to automate. If it's a once-a-month thing, it's not worth the investment.

Score three for constant daily actions. Score one for rare ones.

‍

3. Verifiability

Verifiability is about feedback loops. Can users easily tell if the AI got it right? Math, for example, is a simple yes-or-no and objective; score it a 3. However, personalized recommendations are subjective so I might score it lower, a one or a two.

The maximum score is nine. You may not want to add all the scores together, but instead assess each one within its own dimension. Because some ideas are such a great opportunity, even if they're not that safe or verifiable, but the volume makes up for it.

‍

The output of this for you and your team is going to be a classification:

• Automate it now
• Make it just an assistant—AI assistance versus full automation
• Not ready yet or unsure.

For user-facing features, most will land in the middle. It'll be an AI assistant zone, and this is where design matters. AI assistant means AI doing the work, but the user has control before anything is finally committed.

The article gives examples of things that scored six out of nine, like AI meal planning based on health objectives or AI-generated meal plans based on shopping cart items. These require taste and a preference judgment, and the AI could do a good job of recommending it, but not just doing it all for the user.

‍

‍

Three reasons AI automation fails

We now know why AI automations succeed, so why might they fail? If you create an AI automation, it might fail because of three things:

‍

1. Lack of trust

Early mistakes are often visible. Users will start ignoring AI suggestions or actions it can perform for them, and adoption will flatline.

‍

2. Frustration with edge cases

It might work great 80% of the time, but that 20% where it's wrong is infuriating. Users feel like they have to double-check everything all the time.

‍

3. Creepiness

It knows too much, or it's too aggressive, and users feel like it's in their face or creepy.

These are just things that you have to design for and think through. Make sure the user is careful and aware of what this AI can and can't do, or just label something as not ready yet.

‍

‍

Why doing nothing is no longer an option

Right now, the only thing you can't afford is doing nothing at all. The world is moving quickly, and every leader must be ripping apart their processes and thinking about every opportunity to automate inside their product for their users 24/7. This should be the only thing you're focusing on.

Start reflecting on what you will automate in 2026.

That's it for me this week. I hope you enjoyed this episode, and I'll see you next week.

‍