A leader that I cherish was there for me this week to converse about AI. She shared the Stanford studies on AI that are helping guide her — quantiatively — decisions as a top-level engineering leader.

https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf

Specifically, she shared the following video

And what the video found feels extremely accurate; AI definitely improves productivity in a majority of domains, but not equally and not as much as the market makes it seem. It appears that you should, in most cases, expect between a 5-20% boost (which is good still). AI-generated code is mostly “reworked” code i.e., refactoring, but on recent code. Finally, surveys are a very poor indicator of productivity and tend to significantly over-estimate; there’s little correlation to true productivty. Surveys are good for the “feels”.

Then I spewed some thoughts in public channels at work to try to encourage more formal policies around ai adoption (because the lack of policy is exhausting); what follows is that spew of thoughts.

I’m a believer in Horowitz’s “define policies, not exceptions and constantly be evaluating and updating the policy”

Spew of thought

i’m constantly asking myself should i let ai rip this and explain or should i do the thinking

and frankly flipping back and forth is hard. my assumption is hopefully we get better at using these really sharp tools and don’t continue to feel this way, but i don’t think we can just avoid them and hope.

i think we use them (in domains where they increasingly make the most sense, aligning with the capabilities of the models and tooling at that point in time) and reflect constantly

i don’t think they are silver bullets, i think we apply them in applications that make the most sense

they are really effective at solving many classes of problems. maybe more as they mature, but definitely not all problems everywhere.

i think any policy needs to be multi-faceted (like many systems solutions); i think we have to be careful to not make sweeping policies, treating them like silver bullets

in my opinion i think they are the closest tool that we’ve experienced that “feels” like a silver bullet (i dont know that there’s a good point of reference, maybe “moving to the cloud”?) which makes them difficult to handle

and i think we (as humans) need to continue to “label” or define these classes of problems as we gain experience with their limitations

like it’s way easier for me to give ai a bit of context (instead of trying to tell a whole motivational story, and how it helps a junior, etc. etc.). and this makes me really sad.

do we expect juniors to just “skip” key career experiences? do we deliberately not use ai sometimes to allow for junior growth?

oooh one last point. i think working with ai often is fighting with sunk cost falacy. it’s really hard to stop trying to prompt once you’ve begun trying to get ai to do it. and be like, wait maybe i should do this.

this is a skill im still strengthening

one other concern that i have that i hope improves as the industry matures is concentration of power and leverage. i’d like to see open models continue to improve

AI for tangential work or PoCs within domain, but not what you first reach for for solutions (sure use it for linting, etc.) within domain

sadly, i think what complicates all of this is that they are really good at work you’d give to a person who is more junior. and i dont know what to do about that. what do you think?

if you’ve solved a problem a handful of times within domain, maybe it’s fine, but then why don’t you actually put in place a deterministic, performant solution for that problem instead?

i think i will continue learning and thinking in my domain and not try to improve efficiency in my domain.

i’ve always felt that i spent a majority (yes, sad) doing tangential work e.g., fix ci, update release process, write some script to do some random thing, etc.. it’s for these things that i am hopeful i can free myself up to think on the things that i should be.

so really this boils down to minimizing ai use in-domain (there’s a proper name for this).

i used to pride myself on learning all of tools tangential to my domain that helped me day-to-day be more efficient, but feels like i no longer need to spend thinking / learning effort doing this anymore

and then the maintainers of those tools / systems do the same. they (humans) think within domain and use ai on tangential stuff