I’ve been trying to use AI tools and different models to see how useful they are in helping with Product Management work, testing each one in some detail, and I feel like I’m in a different reality from people telling me that to be effective I need to integrate AI tools into every part of my workflow.
I’m going to take a walk through ProdPad’s “The Product Manager’s Guide to Using AI to Work Better and Faster” ebook. I chose this as an example because I like them, they’re generally pragmatic and their work is aligned with my actual experience on the ground (they make an enormous amount of great content available, I recommend it to people all the time).
I’m going to walk through their sections, as Section Header, and then talk about my experience with that. Quotes in quotes or italic blocks.
(And caveats: in the experiments I’ve been doing, I’ve been deliberately not doing a lot of prompt engineering in order to compare the engines. I hear that can make a huge difference, but I have not heard specifics on how, or how I’d do it in a way that allowed me to still compare the models.)
Product Strategy
Writing or improving your product vision statement: PP offers that you can use AI to generate a “motivating, clear vision statement” or to use it to revise and improve existing ones.
My experience: I find the vision statements generated by all the models generic and bland, but potentialy indistinguishable from product vision statements you see in the wild.
Generally, I’ve found AI tools far worse than talking things through with a peer, and that they tend to water down actual inspiration.
Setting product goals and OKRs
I agree with PP, you can build these by throwing random thoughts and different material at them, and you’ll get something decent, if generic. The use case of “help me turn this set of things into this kind of other thing” works, and it’s particularly good for “wait, no, rewrite that to be in active tense” or whatnot.
You’re not going to get an AI tool to help you pair objectives to health metrics, or how to make leaps (if you want to build a culture of test and learn, you’ll get advice like “ship 100 A/B tests” which seems decent but an experienced product person’s going to realize how that can go wrong, and think bigger)
Ideation
“AI’s perfect for that early-stage brainstorming” the guide says, which I absolutely disagree with. Today’s LLMs are more consensus predictors than creative minds. It’s why they’re so much worse at writing a good joke, which requires ingredients like inspiration, a delighted leap in concepts, empathy, and a sense of the absurd, compared to “explain what this javascript function does and when would I use it compared to this other one?”
Is it useful to use a tool to guide the conversation, make sure you think about things, offer prompts to creative thinking? Sure.
To get you to the kind of idea that’s going to delight your customers and advance your business, that’s bringing in diverse viewpoints and approaches, it’s time to contemplate while you take a hike or sit staring at the ocean for a day.
Discovery
Here, market and competitor research. PP’s guide is careful and correct that “good insights come from real research” and to caution you to double-check what it comes up with, and I agree. I also agree it can be a great starting point, if you understand its limitations and its incompleteness.
User research
“The same goes for user research – use AI with caution. You need to understand where it can help and where it can’t. After all, nothing should replace your efforts to speak to real or potential customers.”
Yup. Absolutely. This is why I’m working through this guide. There are probably a dozen people in my LinkedIn feed (and spamming my email) telling me I can just AI the whole user research thing, and they’re wrong.
PP offers some things where AI tools can help, and I agree with these.
AI could help by:
- Suggesting research methodologies
- Generating research questions for user interviews or focus groups
- Writing test scripts for user testing
- Helping to prepare research reports and presentations
- Analyzing data from your research efforts to help you draw conclusions
Cases where the tool’s drawing from the existing body of knowledge do work well: there are many documented ways to go about the thing, it’s all been ingested, and can be regurgitated for you.
A good comparison point is “I’m about to interview for a position as a Lead Product Marketing Manager, what questions might I be asked?” — all the tools will give you great questions to rehearse with.
Analyzing data, though?
My own experiments here have been underwhelming — even when using clean, formatted data, the summaries of data neglect to spot interesting outliers, and generally when I compare the summaries to the actual feedback, the AI-tool summary seems like someone wrote up a very convincing report after not really paying full attention in class.
Prototyping
Yup, totally agree, if you want to whip up something that demonstrates how a product might work, some of the AI coding tools can get you there very fast. It can also be pretty frustrating, where after arguing about “why are you calling this function that doesn’t exist” you wish you were back using a sketch tool with basic “if this is clicked, show this other sketch…”
But this is a place where the tools have come a long way very fast, and where experience using the tools can help a lot. There’s definately utility here.
Capturing feedback
ProdPad’s guide suggests using AI transcription tools. I suggest caution here. Your experience may vary, and they’re getting better, but the problem of confabulation comes up here, too — if you use an old-school transcription tool, and something’s unintelligible because of crosstalk or a passing garbage truck, they won’t try to make sense of it.
I’ve seen LLMs screw this up, where there’s a section that seems plausible but wrong. You go back and listen to the recording and have to piece it back together yourself.
But if I wasn’t paying attention, and I thought the transcription was the source of truth, I’d be in trouble. And we have the same problem here as in other “it’s pretty good but sometimes makes stuff up” — the errors are delivered confidently, and it’s reliable enough that I see it encouraging default acceptance.
I also don’t trust — see my experiences on summarizing feedback above — that AI tools summarizing customer conversations are going to surface the novel and strange outliers (poison!).
I would recommend at the very least taking your own notes of particularly interesting or novel things in conversations you lead, and making sure they’re included. It’s worth doing just for your own brain.
Feedback analysis
My own experiments here, again, they’re 80% good but the 20% can be incredibly important, and there’s no substitute for reading and getting a sense of your customers as reading it yourself.
And I haven’t tried the same experiments using ProdPad’s “Signal” tool, they might have this all dialed in with advanced prompting magic.
Prioritzation and backlog management
The guide says “To get the most time-saving benefits out of AI assistance with your backlog, it’s all going to be about the AI capabilities of your idea management tool.”
Which… sure, and ProdPad talks about how their tool can do things like take care of duplicate ideas. If your tool does these things, it’s great.
Prioritization
Well, when it comes to prioritization, AI won’t replace your
judgment, but it can give you a strong head start. You can ask it to score or stack-rank ideas based on impact, effort, strategic fit, whatever criteria matter to you.
I’m going to write a much longer piece on this. For now, I’d summarize my experience here as “the conversation can be valuable, but the tools don’t really understand impact/effort/complexity or your criteria, and you end up doing the prioritization yourself.”
The magic comes from combining structured data with AI’s ability to spot trends and surface hidden opportunities.
Spotting trends and patterns, I’ll agree here, but surface hidden opportunities… opportunities as things that are there and you’re overlooking, perhaps. Hidden opportunities that require you to think creatively about problems, I just have not seen. Just cross-apply what I said about Ideation, you’re just not going to get good ideas for an entirely novel new thing.
Product documentation and in-product copy
If there’s one AI use case that really hits home for Product Managers, it’s writing. From product documentation to tooltips, there’s a lot of copy to craft, and AI can seriously speed things up.
Yep. This is a great case for AI tools — you have an API spec and you need to write documentation for it, or you want to dump a ton of thoughts and turn that into a set of bullet points you can use to structure a discussion. Totally works.
Stakeholder management
Same thing as documentation, there’s a lot you can do in building your communication, updates, all that good stuff.
I question something, though. In the ebook, reinforcing the “fielding questions”
… CoPilot can answer almost any question
about your product work. This is a complete game changer when
it comes to fielding those day-to-day, impromptu questions from
stakeholders across your organization.
Yes, but is it in a good way?
One of my favorite things about working in Product is when someone comes to me and says “help me understand why…” and we talk through a customer need or an implementation detail, not because it’s randomizing, but because those conversations let me ask what brought up the question, and discover ideas, sources of confusion, better ways to communicate. That’s the business of good product management.
For example, let’s say your boss wants to know everything on the roadmap that relates to a certain strategic objective. Sure they could look at your roadmap (and even group it by Objective in ProdPad), but the chances are they’re just going to fire the question over to you.
Great! That’s what I’m here for. Who turns down the chance to, in real-time, talk to your boss about everything you’re doing related to a strategic objective, being able to expand on points as they’re interested, potentially making adjustments, opening long-term lines of inquiry, building that relationship?
This is the kind of conversation we should hope tools free us for: that we’re not in the weeds writing SDK documentation or something, firing off terse off-putting answers to things.
Coaching and best practices
Totally agree, having ingested and stolen the knowledge of all product managers and related works, AI tools are great for talking through how to structure a post-mortem, or what three different approaches are to prioritizing a sales-led product backlog.
I have another longer post on this, but we should be a little uncomfortable with this one. When you can ask ChatGPT to give you Marty Cagan’s viewpoint on something, rather than buy Marty Cagan’s book, why is he going to write another book? Some of my best learning experiences with a PM are talking through stakeholder management with someone I know who is amazingly great at it, and having them ask me insightful and sometimes uncomfortable questions. If I can get 80% of that from an AI tool and never be truly challenged to improve, that seems like a loss.
Wrap it all up, Derek
I liked the guide as a walk-through of what people are thinking when they say “product managers should be using AI for everything” but what are we left with?
- For documentation, there are things AI tools can help a lot with, with caveats
- For brainstorming and talking things through, then turning that into a form that’s structured and useful: great
- For data analysis and feedback management, it can be useful but you have to double-check, and there’s no substitute for reading it yourself
- For prototyping, yup
- For prioritization and backlog management, I don’t see it being much help
- For advice and coaching, there’s use here
I would also encourage anyone thinking of adopting AI to consider more generally where they, as curious and empathetic humans, can build connections and find insight, and whether tool use helps or hinders that, and whether it long-term might inhibit their ability to grow their experience and skills.