I run a fantasy football league on things get pretty heated during the season. Debates and trash talks go on daily in our GroupMe chat and the guys like to remind each other about their resume. How many wins they have, how many championships, how many times someone beat someone else, you name it. Ideally these status would be easy to check on our ESPN fantasy app, but ESPN’s platform has some shortcomings here.

Ever since agents made it into the mainstream, I’ve been really interested in how they work. I’ve always wanted my own personal Jarvis, and now I feel like I have the chance to build one myself. At its score, it’s mechanics: a loop that observes input, decides what to do, acts, and repeats. At scale, things get much more complex, especially for user-facing products like ChatGPT or other AI coding assistants. What lies in between is a lot of lessons, techniques, and mechanics.

Over the past few weeks, I’ve been tasked with developing a strategy to track and measure AI usage within my engineering team. As an AI-curious leader, I expected to focus on tools and dashboards—but what I quickly realized is that successful measurement starts with asking the right questions.

Along the way, I leaned on a few resources that helped shape my thinking:

• Pete Hodgson blog post: Leading Your Engineers to an AI-Assisted Future: A practical framework for rolling out AI in phases - starting with experimentation, then adoption, then impact.
• The Pragmatic Engineer blog post: A New Way to Measure Developer Productivity: An interview with the creators of a new framework, DevEx.
• The Pragmatic Engineer YouTube Interview: Measuring the Impact of AI on Software Engineering with DX CTO Laura Tacho: Insight into measuring developer productivity and how to add AI to the mix.

These perspectives, combined with what I’ve observed within my own team, have led me to a few early takeaways about measuring and leading AI adoption in engineering.