Ever since agents made it into the mainstream, I’ve been really interested in how they work. I’ve always wanted my own personal Jarvis, and now I feel like I have the chance to build one myself. At its score, it’s mechanics: a loop that observes input, decides what to do, acts, and repeats. At scale, things get much more complex, especially for user-facing products like ChatGPT or other AI coding assistants. What lies in between is a lot of lessons, techniques, and mechanics.

Over the past few weeks, I’ve been tasked with developing a strategy to track and measure AI usage within my engineering team. As an AI-curious leader, I expected to focus on tools and dashboards—but what I quickly realized is that successful measurement starts with asking the right questions.

Along the way, I leaned on a few resources that helped shape my thinking:

• Pete Hodgson blog post: Leading Your Engineers to an AI-Assisted Future: A practical framework for rolling out AI in phases - starting with experimentation, then adoption, then impact.
• The Pragmatic Engineer blog post: A New Way to Measure Developer Productivity: An interview with the creators of a new framework, DevEx.
• The Pragmatic Engineer YouTube Interview: Measuring the Impact of AI on Software Engineering with DX CTO Laura Tacho: Insight into measuring developer productivity and how to add AI to the mix.

These perspectives, combined with what I’ve observed within my own team, have led me to a few early takeaways about measuring and leading AI adoption in engineering.