The Human in the Lead: AI and the Rules of Engagement

By Nathan Marsh

I recently used AI to help redesign my kitchen. I could describe what I wanted, test different ideas, refine the design, and get remarkably close to something I could actually build. But there was still a line I wasn’t quite ready to cross: I didn’t hand the AI my credit card and tell it to order the fittings, hire the plumber, and get on with it.

That line between AI that advises and AI that acts—and the related critical trust and governance issues—were the topics explored by my recent panel at the Financial Times Future of AI Summit in London.

Moderated by the FT’s Andrew Hill, I joined Philipp Heltewig of NiCE Cogni, Ken Moore of Mastercard, and Sara Hooker of Adaption for a conversation spanning customer experience, financial services, AI development and, from my perspective at Bentley Systems, infrastructure.

Across our various sectors, AI is moving from answering questions to taking action. NiCE is using AI agents to resolve customer issues. Mastercard is working with agents that have permission to transact, based on consumer intent, within defined controls. Bentley is helping our users automate routine work, interact with applications in new ways, and explore and optimize more alternatives. But there is a difference between advising and taking action.

These use cases aren’t built around autonomy for its own sake. They're helping us to define the relationship between what AI can do and when humans need to step in. As AI takes on more consequential work, organizations need to be explicit about where AI’s role ends, where human judgement is essential, and who ultimately remains accountable. For us, the human engineer is always accountable.

As Ken Moore said—and I wholeheartedly agree—it’s less about the human in the loop and more about the human in the lead. Nowhere is this truer than in engineering, where the ultimate accountability for public safety and structural integrity belongs to the chartered engineer who signs off on the work.

Start with context

We also discussed industry context, as the foundation for trustworthy AI. The more consequential the work, the more important it is that AI understands the specific standards, constraints, history, and real-world conditions surrounding the decision it is helping to make.

For NiCE, that means agents working with the customer's history and the workflows behind each journey. For Mastercard, it means richer context behind every payment, starting with the consumer's intent. For Adaption, it means systems that adapt to the specific task and data in front of them.

In infrastructure, that context includes site realities, engineering standards, and the institutional experience that organizations gain over years of delivering projects.

The acceptable level of uncertainty is very different when AI is recommending the destination for your summer getaway and informing a decision about the structural integrity of a bridge.

Why was a particular design decision made regarding an asset? What changed during construction? What maintenance has already happened? Without that history, AI can recommend something that looks sensible but ignores decisions made for good reasons.

This valuable information is easily lost when it's scattered across files and systems or leaves with engineers who retire. Organizations need to deliberately capture and connect that information by wrapping AI capabilities in technologies built on shared schemas that bring together, contextualize, and preserve the infrastructure.

Deep industry context helps to make sure we are making the right decisions, not just the fastest decisions.

AI-Literate Teams

AI literacy, we agreed, is another key area for scrutiny. Simply putting an AI tool into the hands of colleagues is not enough. We must also equip them to use it effectively, question its outputs, and understand its limitations. As my colleague Bentley CTO Julien Moutte has pointed out, that puts an even greater premium on systems thinking and critical judgement.

For infrastructure, we are only beginning to see what becomes possible when engineers match machine speed with engineering rigor. Our recently held Year in Infrastructure Awards showed how quickly that potential is becoming real, with nearly 70% of finalists and more than a quarter of all submissions using AI in their work.

But we all must learn much more, and our panel discussion underlined that. As the AI story moves on, the perspectives from one industry can help to shape and sharpen the questions another needs to ask.

Thank you to Andrew Hill and the Financial Times for creating the space to explore these contrasts, and to my fellow panelists for bringing such different perspectives to an important conversation.