Notes
Chapter 6 - The Root Causes of Failed Product Efforts
The top 10 biggest problems with how most product teams work (idea-biz case-roadmap-requirements-design-build-test-deploy):
- The source of ideas: in this model there are lots of sales-driven and stakeholder-driven products. Teams also lose empowerment here.
- Business cases: at this stage, we have no idea about the two key components of business cases: how much money we’ll make and how much it will cost.
- Product roadmaps: the vast majority are prioritized lists of features and products, but they fail to acknowledge two truths: half our ideas aren’t going to work, and those that do will need several iterations.
- The role of product management in this model is more project management.
- The role of design is similar; they just aren’t involved early enough to create great products, and end up doing what they can within constraints.
- Engineers get brought in way too late. They’re often the single best source of innovation.
- Agile gets brought in way too late; it becomes Agile only for delivery.
- The entire process is project-centric, when product should really be about outcome.
- The biggest flaw of waterfall-type processes is that customer validation happens way too late.
- The biggest loss is usually the opportunity cost of what the organization could have been doing instead.
Chapter 7 - Beyond Lean and Agile
The best product teams leverage the core principles of Lean and Agile, and have three overarching principles:
- Risks are tackled up front, including value risk (whether customers will buy it), usability risk (whether users can figure out how to use it), feasibility risk (whether engineers can build it with the time, skills, and technology available), and business viability risk (whether the solution works with all the aspects of business).
- Products are defined and designed collaboratively, rather than sequentially. Design and engineering in particular are involved throughout the whole process.
- It’s about solving problems, not implementing features. Strong teams focus on business results, not output.
Chapter 8 - Key Concepts
When we talk about product, we are including: functionality (the features), technology (that enables this functionality), user experience (that presents this functionality), how we monetize this functionality, and how we acquire users and customers.
There are two essential activities in all product teams:
- Discovering the product to be built
- Delivering that product to market
Product Discovery
The purpose of product discovery is to separate the good ideas from the bad.
The output of product discovery is a validated product backlog.
This means getting answers to four critical questions:
- Will the user buy this (or choose to use it)?
- Can the user figure out how to use this?
- Can our engineers build this?
- Can our stakeholders support this?
Prototypes: Product discovery involves running a series of very quick experiments, and to do this quickly and cheaply, we use prototypes rather than products. Great product teams test 10-20 product ideas per week.
Product Delivery
The purpose of product delivery is to build and deliver production-quality products, something that can be sold and create a business.
Product/Market Fit
We are striving to build the smallest possible product that meets the needs of a specific market of customers.
Product Vision
This is the long-term objective of the product, 2-10 years out, and is how the product organization intends to deliver on the company’s mission.
Chapter 9 - Principles of Strong Product Teams
Product teams need to be built of missionaries: people that believe in the vision and are committed to solving problems for their customers.
A product team is composed of a product manager, a product designer, and 2-12 engineers.
All other things being equal, a co-located team will significantly outperform a dispersed team.
Product teams should stay together in general. It takes time to learn how to work together, and it can take time to gain enough expertise to innovate. They also need to feel ownership.
Three reasons to build product teams this way:
- Collaboration is built on relationships, and product teams—especially co-located ones—are designed to nurture these relationships.
- To innovate you need expertise, and durable product teams are required to build this expertise.
- The product team needs to understand the business objectives and context, and feel full ownership and responsibility for the outcome.
Chapter 13 - The Product Manager
The product manager needs to be among the strongest talent in the company.
There are four key responsibilities of the product manager:
- Deep knowledge of the customer: their issues, pains, desires, how they think, how they work, and how they decide to buy. They need to be experts on the product.
- Deep knowledge of the data: they need to be comfortable with data & analytics.
- Deep knowledge of the business: they need to understand the role the product plays in the business, and how the business overall works.
- Deep knowledge of the market & industry: they need to understand competitors, market trends, technology changes, customer behaviors & expectations, and the role of social media.
Chapter 11 - The Product Designer
Good product designers think about the customer’s whole journey over time. The list of touchpoints could be long, including questions like:
- How will customers first learn about the product?
- How will we onboard a first‐time user and (perhaps gradually) reveal new functionality?
- How might users interact at different times during their day?
- What other things are competing for the user's attention?
- How might things be different for a one‐month‐old customer versus a one‐year‐old customer?
- How will we motivate a user to a higher level of commitment to the product?
- How will we create moments of gratification?
- How will a user share his experience with others?
- How will customers receive an offline service?
- What is the perceived responsiveness of the product?
There are three common problems related to product design:
- You as product manager try to do the design yourself
- You as product manager don’t provide the designs but, rather, provide very high-level user stories to the engineers
- You as product manager provide the interaction design—especially the wireframes—and then you use a visual or graphic designer to provide the visual design.
All three situations are bad because they rarely provide good results. We need design to not just make our product beautiful but to help us discover the right product.
The five keys to a successful and healthy relationship with your designer:
- Do whatever you need to do to have your designer sit next to you.
- Include your designer from the very inception of every idea.
- Include your designer in as many customer and user interactions as possible. Learn about the users and customers together.
- Fight your temptation to provide your designer with your own design ideas.
- Encourage your designer to iterate early and often. The best way you can encourage this is to not get all nitpicky about design details with the very early iterations.
Chapter 12 - The Engineers
There are two types of discussions you should be engaging with your engineers on each day:
- Soliciting their ideas and input for the items you’re working on in discovery.
- Clarifying questions on items they’re working on delivering to production.
Chapter 20 - Principles of Structuring Product Teams
Here are some core principles to consider when structuring your product teams:
- Alignment with investment strategy: you need to have an investment strategy and your team structure should be a reflection of that.
- Minimize dependencies: we want to minimize dependencies so teams can move faster and feel much more autonomous.
- Ownership and autonomy: this is what is going to help build teams of missionaries, not mercenaries.
- Maximize leverage: as orgs grow, there are common needs and the importance of shared services grows. This is done for speed and reliability. But it also creates dependencies and can impinge on autonomy.
- Product vision and strategy: you want an up-to-date vision and strategy, and you want product teams structured to deliver on them.
- Team size: This is a very practical principle. The minimum size for a product team is usually two engineers and a product manager, and if the team is responsible for user‐facing technology, then a product designer is needed, too.
- Alignment with architecture: Architectures drive technologies, which drive skill sets. While we'd love for every team to be a full stack team that can work on any layer of the architecture, in practice that's often not an option.
- When a company has not paid attention to the architecture when they assemble their teams it shows up a few different ways. First, the teams feel like they are constantly fighting the architecture. Second, interdependencies between teams seem disproportionate. Third, and really because of the first two, things move slowly, and teams don't feel very empowered.
- Alignment with user or customer: aligning with user and customer has big benefits in situations like a marketplace, where the buyer and the seller could be quite different.
- Alignment with business: in larger companies there are often multiple lines of business with a common foundation. This is usually lower in priority because often different business units are selling to the same customers.
- Structure is a moving target: Realize that the optimal structure of the product organization is a moving target. The organization’s needs should and will change over time. It’s not like you’ll need to reorganize every few months, but reviewing your team structure every year or so makes sense.
Chapter 23 - The Alternative to Roadmaps
Roadmaps serve two purposes we still need to fulfill:
- Management wants to make sure that teams are working on the highest-business-value items first.
- There are cases where others need to make date-based commitments, and the roadmap is where they see and track those commitments.
To figure out the best ways to solve business problems, teams need the business context, which is provided by two main things:
- The product vision and strategy
- The business objectives
The grief from commitments and specific dates all stems from one problem: the commitments are made too early, before teams know if they can deliver and whether what is delivered will solve the customer problem.
The solution? The product team asks for time to do product discovery before commitments are made, and then afterwards commits to dates and deliverables.
Chapter 24 - Product Vision and Product Strategy
Product vision: the future we are trying to create, typically 2-5 years away.
Mission statement: more general statements that don’t go into detail about how we’ll accomplish it.
Product strategy: the sequence of products or releases we plan to deliver on the path to realizing the product vision.
Product vision should be inspiring, the product strategy should be focused.
Prioritizing markets: there are three critical inputs to your decision:
- The first is market sizing, usually referred to as total addressable market (TAM).
- The second factor concerns distribution, usually referred to as go to market (GTM)
- The third factor is a (very rough) estimation of how long it will take, referred to as time to market (TTM).
Chapter 25 - Principles of Product Vision
These are the 10 key principles for coming up with an effective product vision.
- Start with why. Use the product vision to articulate your purpose. Everything follows from that.
- Fall in love with the problem, not with the solution
- Don't be afraid to think big with vision. Too often I see product visions that are not nearly ambitious enough, the kind of thing we can pull off in six months to a year or so, and not substantial enough to inspire anyone.
- Don't be afraid to disrupt yourselves because, if you don't, someone else will. So many companies focus their efforts on protecting what they have rather than constantly creating new value for their customers.
- The product vision needs to inspire. Remember that we need product teams of missionaries, not mercenaries. More than anything else, it is the product vision that will inspire missionary‐like passion in the organization.
- Determine and embrace relevant and meaningful trends. Too many companies ignore important trends for far too long. It is not very hard to identify the important trends. What's hard is to help the organization understand how those trends can be leveraged by your products to solve customer problems in new and better ways.
- Skate to where the puck is heading, not to where it was. An important element to product vision is identifying the things that are changing—as well as the things that likely won't be changing—in the time frame of the product vision. Some product visions are wildly optimistic and unrealistic about how fast things will change, and others are far too conservative. This is usually the most difficult aspect of a good product vision.
- Be stubborn on vision but flexible on the details. So many teams give up on their product vision far too soon. This is usually called a vision pivot, but mostly it's a sign of a weak product organization. It is never easy, so prepare yourself for that. But, also be careful you don't get attached to details. It is very possible that you may have to adjust course to reach your desired destination. That's called a discovery pivot, and there's nothing wrong with that.
- Realize that any product vision is a leap of faith. If you could truly validate a vision, then your vision probably isn't ambitious enough. It will take several years to know. So, make sure what you're working on is meaningful, and recruit people to the product teams who also feel passionate about this problem and then be willing to work for several years to realize the vision.
- Evangelize continuously and relentlessly. There is no such thing as over‐communicating when it comes to explaining and selling the vision. Especially in larger organizations, there is simply no escaping the need for near‐constant evangelization.
Chapter 26 - Principles of Product Strategy
Good product strategies have these five principles in common:
- Focus on one target market or persona at a time. Don't try to please everyone in a single release. Focus on one new target market, or one new target persona, for each release.
- Product strategy needs to be aligned with business strategy. The vision is meant to inspire the organization, but the organization ultimately is there to come up with solutions that deliver on the business strategy. So, for example, if that business strategy involves a change in monetization strategy or business model, then the product strategy needs to be aligned with this.
- Product strategy needs to be aligned with sales and go‐to‐market strategy. Similarly, if we have a new sales and marketing channel, we need to ensure that our product strategy is aligned with that new channel.
- Obsess over customers, not over competitors. Too many companies completely forget about their product strategy once they encounter a serious competitor. They panic and then find themselves chasing their competitor's actions and no longer focusing on their customers. We can't ignore the market, but remember that customers rarely leave us for our competitors. They leave us because we stop taking care of them.
- Communicate the strategy across the organization. This is part of evangelizing the vision. It's important that all key business partners in the company know the customers we're focused on now and which are planned for later. Stay especially closely synced with sales, marketing, finance, and service.
Chapter 27 - Product Principles
A good complement to the product vision and product strategy is a set of product principles.
Product principles: principles that speak to the nature of the product you want to create.
Chapter 28 - The OKR Technique
The Objectives and Key Results (OKR) technique is a tool for management, focus, and alignment. Here are the critical points for you to keep in mind when using the tool for product teams in product organizations.
- Objectives should be qualitative; key results need to be quantitative/measurable.
- Key results should be a measure of business results, not output or tasks.
- The rest of the company will use OKRs a bit differently, but for the product management, design, and technology organization, focus on the organization's objectives and the objectives for each product team, which are designed to roll up and achieve the organization's objectives. Don't let personal objectives or functional team objectives dilute or confuse the focus.
- Find a good cadence for your organization (typically, annually for an organization's objectives and quarterly for a team's objectives).
- Keep the number of objectives and key results for the organization and for each team small (one to three objectives, with one to three key results each is typical).
- It's critical that every product team track their active progress against their objectives (which is typically weekly).
- The objectives do not need to cover every little thing the team does, but they should cover what the team needs to accomplish.
- It's important that, one way or another, teams feel accountable to achieving their objectives. If they fail substantially, it's worth having a post‐mortem/retrospective with some of their peers or management.
- Agree as an organization on how you will be evaluating or scoring your key results. There are different approaches to this, and it's in large part a reflection of your particular company culture. What's important here is consistency across the organization, so that teams know when they can depend on one another. It's common to define a score of 0 (on a scale from 0 to 1.0) if you essentially make no progress, 0.3 if you just did the bare minimum—what you know you can achieve, 0.7 if you've accomplished more than the minimum and have really done what you'd hoped you would achieve, and 1.0 if you've really surprised yourselves and others with a truly exceptional result, beyond what people were even hoping for.
- Establish very clear and consistent ways to indicate when a key result is in reality a high‐integrity commitment (described earlier) rather than a normal objective. In other words, for most key results, you may be shooting for that 0.7 score. But for a high‐integrity commitment, these are special, and it's more binary. You either delivered what you promised or you didn't.
- Be very transparent (across the product and technology organization) on what objectives each product team is working on and their current progress.
- Senior management (CEO and executive team) is responsible for the organization's objectives and key results. The heads of product and technology are responsible for the product team objectives (and ensuring they deliver on the organization's objectives). The individual product teams are responsible for proposing the key results for each objective they've been assigned. It is normal to have a give‐and‐take process each quarter as the OKRs are finalized for each team and for the organization.
Chapter 31 - Product Evangelism
Product evangelism is, as Guy Kawasaki put it years ago, “selling the dream.” It's helping people imagine the future and inspiring them to help create that future.
There are several techniques to help communicate the value of what you're proposing to your team, colleagues, stakeholders, executives, and investors. Here are 10 pieces of advice for product managers to sell the dream:
- Use a prototype. For many people, it's way too hard to see the forest through the trees. When all you have is a bunch of user stories, it can be difficult to see the big picture and how things hang together (or even if they hang together). A prototype lets them clearly see the forest and the trees.
- Share the pain. Show the team the customer pain you are addressing. This is why I love to bring engineers along for customer visits and meetings. For many people, they have to see (or experience) the pain themselves to get it.
- Share the vision. Make sure you have a very clear understanding of your product vision, product strategy, and product principles. Show how your work contributes to this vision and is true to the principles.
- Share learnings generously. After every user test or customer visit, share your learnings—not just the things that went well, but share the problems, too. Give your team the information they need to help come up with the solution.
- Share credit generously. Make sure the team views it as their product, not just your product. However, when things don't go well, step forward and take responsibility for the miss and show the team you're learning from the mistakes as well. They'll respect you for it.
- Learn how to give a great demo. This is an especially important skill to use with customers and key execs. We're not trying to teach them how to operate the product, and we're not trying to do a user test on them. We're trying to show them the value of what we're building. A demo is not training, and it's not a test. It's a persuasive tool. Get really, really good at it.
- Do your homework. Your team and your stakeholders will all be much more likely to follow you if they believe you know what you're talking about. Be the undisputed expert on your users and customers. And be the undisputed expert on your market, including your competitors and the relevant trends.
- Be genuinely excited. If you're not excited about your product, you should probably fix that—either by changing what you work on or by changing your role.
- Learn to show some enthusiasm. Assuming you're genuinely excited, it's amazing to me how many product managers are so bad or so uncomfortable at showing enthusiasm. This matters—a lot. Absolutely be sincere, but let people see you're genuinely excited. Enthusiasm really is contagious.
- Spend time with your team. If you're not spending significant face time with your designer and every engineer on your team, then they can't see the enthusiasm in your eyes. If your team is not co‐located, you'll need to make a special effort to travel there and do this at least every couple months. Spending some personal time with every last person on the team pays off big in their level of motivation and, as a result, in the velocity of the team. It's worth your time.
Chapter 33 - Principles of Product Discovery
There are a set of core principles on how to do customer discovery:
- We know we can't count on our customers (or our executives or stakeholders) to tell us what to build. Customers don't know what's possible, and with technology products, none of us know what we really want until we actually see it.
- The most important thing is to establish compelling value. It's all hard, but the hardest part of all is creating the necessary value so that customers ultimately choose to buy or to use. We can survive for a while with usability issues or performance issues, but without the core value, we really have nothing. As a result, this is generally where we'll need to spend most of our discovery time.
- As hard and important as the engineering is, coming up with a good user experience is usually even harder, and more critical to success. While every product team has engineers, not every team has the necessary product design skills, and even when they do, are they being used the way we need to use them?
- Functionality, design, and technology are inherently intertwined.
- We expect that many of our ideas won't work out, and the ones that do will require several iterations. We approach discovery with the mindset that many, if not most, of our ideas won't work out. The most common reason for this is value, but sometimes the design is too complicated, and sometimes it would take far too long to build, and sometimes there turn out to be legal or privacy issues. The point is we need to be open to solving the underlying problem in different ways if necessary.
- We must validate our ideas on real users and customers. One of the most common traps in product is to believe that we can anticipate our customer's actual response to our products. We might be basing that on actual customer research or on our own experiences, but in any case, we know today that we must validate our actual ideas on real users and customers. We need to do this before we spend the time and expense to build an actual product, and not after.
- Our goal in discovery is to validate our ideas the fastest, cheapest way possible. Discovery is about the need for speed. This lets us try out many ideas, and for the promising ideas, try out multiple approaches. There are many different types of ideas, many different types of products, and a variety of different risks that we need to address (value risk, usability risk, feasibility risk, and business risk). So, we have a wide range of techniques, each suitable to different situations.
- We need to validate the feasibility of our ideas during discovery, not after. If the first time your developers see an idea is at sprint planning, you have failed. We need to ensure the feasibility before we decide to build, not after. Not only does this end up saving a lot of wasted time, but it turns out that getting the engineer's perspective earlier also tends to improve the solution itself, and it's critical for shared learning.
- We need to validate the business viability of our ideas during discovery, not after. Similarly, it is absolutely critical to ensure that the solution we build will meet the needs of our business—before we take the time and expense to build out that product. Business viability includes financial considerations, marketing (both brand and go‐to‐market considerations), sales, legal, business development, and senior executives. Few things destroy morale or confidence in the product manager more than finding out after a product has been built that the product manager did not understand some essential aspect of the business.
- It's about shared learning. One of the keys to having a team of missionaries rather than a team of mercenaries is that the team has learned together. They have seen the customer's pain together, they have watched together as some ideas failed and others worked, and they all understand the context for why this is important and what needs to be done.
Chapter 34 - Discovery Techniques Overview
Discovery Framing Techniques
There are two goals here:
- Ensure the team is aligned and clear on purpose: the business objective, the specific problem we’re solving for customers, which customers, and how you’ll know you’ve succeeded.
- Identify the big risks to be tackled during discovery.
Some risks to be considered:
- Technology risk—will our technology perform and scale with this solution?
- Usability risk—will this change make our product harder to use?
- Value risk—do customers want this solved and is our solution good enough to get people to switch from what they have now?
- Financial risk—can we afford this solution?
- Business development risk—does this solution work for our partners?
- Marketing risk—is this solution consistent with our brand?
- Sales risk—is this solution something our sales staff is equipped to sell?
- Legal risk—is this something we can do from a legal or compliance perspective?
- Ethical risk—is this solution something we should do?
There are many different ways to assess an opportunity. Here are three:
- An opportunity assessment is designed for the vast majority of product work, which ranges from a simple optimization to a feature to a medium‐sized project.
- A customer letter is designed for larger projects or initiatives that often have multiple goals and a more complicated desired outcome.
- A startup canvas for those times you're creating an entirely new product line or a new business.
Chapter 35 - Opportunity Assessment Technique
The idea is to answer four key questions about the discovery work you are about to undertake:
- What business objective is this work intended to address? (Objective)
- How will you know if you've succeeded? (Key results)
- What problem will this solve for our customers? (Customer problem)
- What type of customer are we focused on? (Target market)
Chapter 36 - Customer Letter Technique
Amazon has the product manager write an imagined press release for when the product launches. How does it improve the life of our customers? What are the real benefits to them?
In the customer letter technique, rather than communicate the benefits in a press release format, you describe them in the format of a customer letter written to the CEO. The customer describes how it has changed or improved his or her life. It also includes an imagined congratulatory response from the CEO to the product team explaining how this has helped the business.
Chapter 38 - Story Map Technique
These are two‐dimensional maps, in which major user activities are arrayed along the horizontal dimension, loosely ordered by time from left to right. So, if there are a dozen major user activities, they would be along the top from left to right, generally in the order you would do them.
Along the vertical dimension, we have a progressive level of detail. As we flesh out each major activity into sets of user tasks, we add stories for each of those tasks. The critical tasks are higher vertically than the optional tasks.
If you lay out your system this way, you can, at a glance, get the holistic view and consider where to draw the line in terms of different releases and their associated objectives.
We can use this story map to frame our prototypes, and then as we get feedback on our prototypes and learn how people interact with our product ideas, we can easily update the story map to serve as a living reflection of the prototypes. As we finalize our discovery work and progress into delivery, the stories from the map move right into the product backlog.
Chapter 39 - Customer Discovery Program Technique
You want to develop a group of at least six reference customers in your specific target market segment, who are willing to work with you on developing a product: testing out prototypes, giving feedback, etc. Ideally they are well-recognized, marquee names, so you can leverage them later for sales and marketing.
Chapter 41 - Customer Interviews
In every user or customer interaction, we always have the opportunity to learn some valuable insights. Here's what I'm always trying to understand:
- Are your customers who you think they are?
- Do they really have the problems you think they have?
- How does the customer solve this problem today?
- What would be required for them to switch?
Here are some tips for getting the most out of customer interviews:
- Frequency. Establish a regular cadence of customer interviews. This should not be a once‐in‐a‐while thing. A bare minimum would be two to three hours of customer interviews per week, every week.
- Purpose. You are not trying to prove anything during these interviews, one way or the other. You're just trying to understand and learn quickly. This mindset is critical and needs to be sincere.
- Recruiting users and customers. I talk much more about this when we discuss the usability testing technique, but for now, be sure to talk primarily to people in your intended target market. You're looking for about an hour of their time.
- Location. It's always amazing to see customers in their native habitat. There's so much to learn just by observing their environment. But it's also fine to meet them somewhere convenient or have them come to your office. If you need to do this over a video call, that's not as good, but much better than not doing at all.
- Preparation. Be clear beforehand what problem it is you think they have, and think about how you'll either confirm or contradict that.
- Who should attend. My favorite is to bring three people to these interviews: the product manager, the product designer, and one of the engineers from the team (we normally rotate among those that want to attend). Usually, the designer drives (because they've usually been trained how to do this well), the product manager takes notes, and the developer observes.
- Interview. Work to keep things natural and informal, ask open‐ended questions, and try to learn what they're doing today (not so much what they wish they were doing, although that's also interesting).
- Afterward. Debrief with your colleagues to see if you've all heard the same things and had the same learnings. If you made any promises to the customer during that session, be sure you keep them.
Chapter 42 - Concierge Test Technique
The idea behind the concierge test is that we do the customer’s job for them. It requires going out to the actual users and customers and asking them to show you how they work so that you can learn how to do their job, and provide them a better solution.
Chapter 43 - The Power of Customer Misbehavior
There are two main ways to come up with product opportunities:
- Assess the market opportunities and pick potentially lucrative areas where significant pain exists.
- Look at what technology or data enables—what’s now possible—and match that up with a significant pain.
Either can get to a winning product.
But there’s a third approach too:
- Allow, or even encourage, customers to use your product to solve problems other than what was planned for or officially supported.
Creating a public API is one of the ways to encourage this behavior.
Chapter 45 - Principles of Prototypes
Here are 5 key principles behind the use of prototypes:
- The overarching purpose of any form of prototype is to learn something at a much lower cost in terms of time and effort than building out a product. All forms of prototype should require at least an order of magnitude less time and effort as the eventual product.
- Realize that one of the key benefits of any form of prototype is to force you to think through a problem at a substantially deeper level than if we just talk about it or write something down. This is why the very act of creating a prototype so often exposes major issues otherwise left uncovered until much later.
- Similarly, a prototype is also a powerful tool for team collaboration. Members of the product team and business partners can all experience the prototype to develop shared understanding.
- There are many different possible levels of fidelity for a prototype. The fidelity primarily refers to how realistic the prototype looks. There is no such thing as one appropriate level of fidelity. Sometimes we don't need the prototype to look realistic at all, and other times it needs to be very realistic. The principle is that we create the right level of fidelity for its intended purpose, and we acknowledge that lower fidelity is faster and cheaper than higher fidelity, so we only do higher fidelity when we need to.
- The primary purpose of a prototype is to tackle one or more product risks (value, usability, feasibility, or viability) in discovery; however, in many cases, the prototype goes on to provide a second benefit, which is to communicate to the engineers and the broader organization what needs to be built. This is often referred to as prototype as spec. In many cases, the prototype is sufficient for this, but in other cases—especially when the engineers are not co‐located or when the product is especially complex—the prototype will likely need to be supplemented with additional details (usually, use cases, business rules, and acceptance criteria).
Chapter 46 - Feasibility Prototype Technique
There are often times when engineers identify a significant feasibility risk involved in solving a particular problem. Some examples:
- Algorithm concerns
- Performance concerns
- Scalability concerns
- Use of a technology the team has not used before
- Use of a third-party component or service the team has not used before
The main technique used for tackling these types of risks is for one or more of the engineers to build a feasibility prototype. This is often referred to as a proof-of-concept (POC).
These are meant to be quick and dirty, often throwaway code, and should only take a day or two.
Chapter 47 - User Prototype Technique
User prototypes range from low-fidelity—useful to represent things like information and workflow—to high-fidelity, where you can test things like the visual design.
The biggest limitation of a user prototype is that you can’t prove anything—like whether your product will sell.
Chapter 48 - Live-Data Prototype Technique
Live data prototypes are very limited implementations of what you eventually want to build.
The key is to be able to send a limited amount of traffic, and collect analytics on how this live-data prototype is being used.
A live-data prototype is a small fraction of the productization effort, but you can get big value on it. But keep in mind building a shippable product based on it will likely require 90-95% more effort, and you need to plan for that.
Chapter 49 - Hybrid Prototype Technique
Sometimes called a Wizard of Oz prototype, this is a combination of a front-end user experience with someone in the backend handling what would be performed by automation in the future.
It’s not scalable, but it can appear to the user to perform just like the product will in future.
Chapter 50 - Testing Usability
Modern usability testing is done during discovery, using prototypes, before the product is built.
Recruiting Users to Test
- If you have a customer-discovery program, you’re all set.
- You can advertise for test subjects on something like Craigslist, or set up an SEM campaign.
- If you have a list of emails of your users, you can select from here.
- You can solicit volunteers on your company website (make sure to still call and screen).
- You can always go where your users congregate: trade shows for biz software, shopping centres for ecommerce, etc.
- If you’re asking users to come to your location, you’ll need to compensate them for their time. You can often arrange to meet them at a mutually convenient location like a Starbucks.
Preparing the Test
- Do usability testing with a high-fidelity user prototype.
- Have the product manager, product designer, and one engineer (rotate in your team) at the test.
- Define in advance the set of tasks that you want to test.
- One person should administer the test and one should take notes.
- The other testing environment that works well is your customer’s office. You can learn all kinds of additional things here too, like their setup, computer and internet speed, etc.
Testing Your Prototype
- When you start the test, make sure to tell them it’s a prototype, an early idea, and not real. Remind them that you’re testing the idea, not them, and they won’t hurt your feelings with candid feedback.
- See if they can tell from the landing page of the prototype what you do. It’s a good learning opportunity.
- During testing, keep the users in use mode, and out of critique mode. What matters is whether they can do the tasks they need to do, not whether something should be changed.
- The main skill you need during testing is to keep quiet.
- There are three important cases you’re looking for: (1) the user got through the task with no problem at all and no help; (2) the user struggled and moaned a bit, but he eventually got through it; or (3) he got so frustrated he gave up.
- Avoid giving help or leading them in any way. You can ask them what they’re looking for, or to clarify something, but try not to give them help.
- If they’re stuck, act like a parrot and narrate what you see them doing. It will prompt them to tell you what they’re trying to do, or what they’re looking for.
- You’re trying to understand how your target users think about the problem and to identify places in your prototype where the model the software presents is inconsistent or incompatible with how the user is thinking about the problem.
- Pay attention to body language and tone. It’s obvious when they don’t like the ideas, and clear when they genuinely do.
After each test, have the product manager or designer write up a short summary of learnings and share with the rest of the product team.
Chapter 51 - Testing Value
There are three parts to testing value:
- Testing demand: is there actually demand for what we want to build?
- Testing value qualitatively: do customers love this? Will they pay for it?
- Testing value quantitatively: how well does our solution solve the problem?
Chapter 52 - Demand Testing Techniques
The demand-testing technique is called a fake door test.
We put a button or menu in the user experience where they expect it to be, but when the user clicks that button, rather than taking them to the experience (it could be a series of buttons/pages), we take them to a page explaining that we’re working on it and looking for feedback.
In practice, there is very often demand for a product, and users will sign up for a trial, but getting them to switch is very difficult.
Chapter 53 - Qualitative Value Testing Techniques
Product teams should be doing at least two or three qualitative value tests every week. Here’s how:
- Interview first: start the user test with a short user interview to make sure the user has the problems we think they have, find out how they currently solve those problems, and what it would take for them to switch.
- Usability test: to test value, the user first has to understand the product and how it works. So we do a usability test first, usually using a high-fidelity user prototype.
- Using money to demonstrate value: ask the customer if they would get their credit card out and buy the product. If it’s expensive, ask them to sign a non-binding letter of intent to buy.
- Using reputation to demonstrate value: ask them how likely they would be to recommend the product to their friends or co-workers or boss (typically on a scale of 1-10). Ask them to share on social media. Or ask for the email of their boss or friends for a recommendation.
- Using time to demonstrate value: especially for business products, ask the person if they’d be willing to schedule some significant time to work on the product with you.
- Using access to demonstrate value: you can ask people to provide the login credentials for whatever product they would be switching from (because you tell them there’s a migration utility or something).
Remember, the aim is rapid learning. If you get a substantially different response to the prototype for some reason, your job is to figure out why.
If you are a PM, make sure you are at every single qualitative value test. Do not delegate or hire for this.
Chapter 54 - Quantitative Value Testing Techniques
Qualitative testing is about learning and insights; quantitative testing is about collecting evidence.
- A/B testing: in discovery A/B testing we typically show the current product to 99% of our users, and the live-data prototype to 1% of our users. We monitor it closely.
- Invite-only testing: if you’re more conservative, or don’t have enough users to show 1% (or even 10%), you can do invite-only tests. This data isn’t as predictive as a true A/B test.
- Analytics: you can use various types of analytics (user behavior, business, financial, performance, operational costs, go-to-market costs, sentiment) to generate insights about your product.
Chapter 55 - Testing Feasibility
When we talk about validating feasibility, the engineers are really trying to answer several related questions:
- Do we know how to build this?
- Do we have the skills on the team to build this?
- Do we have enough time to build this?
- Do we need any architectural changes to build this?
- Do we have on hand all the components we need to build this?
- Do we understand the dependencies involved in building this?
- Will the performance be acceptable?
- Will it scale to the levels we need?
- Do we have the infrastructure necessary to test and run this?
- Can we afford the cost to provision this?
If the engineers have been following along as the team has been developing and testing new ideas, they can give a much better answer to “What’s the best way to do this and how long would it take?” They might need to create a feasibility prototype (POC) to answer, but they should have more context.
Chapter 56 - Testing Business Viability
There are three different ways to show a prototype, and you must use the right technique for the situation:
- User test: when we test our product ideas on real users and customers. The purpose is to test usability and value of the prototype or product.
- Product demo: when you sell your product to prospective customers or users (or evangelize it internally). This is a sales or persuasion tool and product marketing often leads here. The purpose is to show off the value of the prototype or product.
- Walkthrough: when you show your prototype to a stakeholder and want to make sure they see and note everything that might be a concern. The purpose is to give them the opportunity to spot problems. The product manager usually drives.
Chapter 58 - Discovery Sprint Technique
A discovery sprint is a one‐week time box of product discovery work, designed to tackle a substantial problem or risk your product team is facing.
It starts with framing the problem by mapping the problem space, picking the problem to be solved and the target customer, and then progresses into pursuing several different approaches to the solution. The team next narrows down and fleshes out the different potential solutions, then creates a high‐fidelity user prototype—finally, putting that prototype in front of actual target users and observing their reactions.
There are three situations where a discovery sprint is recommended:
- When the team has something big and critically important and/or difficult to tackle.
- When the team is just learning how to do product discovery.
- When things are moving too slow, and the team needs to recalibrate on how fast they can and should be moving.
Good book on this: Spring: How to Solve Big Problems and Test New Ideas in Just Five Days.
Chapter 59 - Pilot Team Technique
One of the simplest ways to get product teams to shift to a new way of working is to introduce the changes to a pilot team, who tries things out for a quarter or two.
If they’re successful, you can learn what worked and what didn’t, and then roll out the changes to other teams.
Chapter 60 - Weaning an Organization Off Roadmaps
To get organizations to shift off of roadmaps:
- Plan to continue with your existing roadmap process for 6-12 months
- Every time you reference a product roadmap item, include a reminder of the business outcome that feature is intended to help.
The goal is that over time, the organization moves its focus from specific features launching on specific dates to business results.
Keep in mind why stakeholders like roadmaps:
- They want some visibility into what you’re working on, and reassurance that you’re working on the most important items.
- They want to be able to plan the business and need to know when critical things will happen.
Chapter 61 - Managing Stakeholders
How to determine a stakeholder: if they have veto power or can prevent your work from launching.
Responsibilities of the product manager re: stakeholders:
- Understand the considerations and constraints of various stakeholders, and bring this knowledge into the product team
- Convince stakeholders that they understand their issues and is committed to coming up with a solution that works for them and the customer.
To succeed with stakeholders:
- Be a competent product manager: have a deep understanding of your customers, the analytics, the technology, your industry, and in particular, your business.
- Spend one-on-one time with key stakeholders: sit down with them and listen. Explain that the better you understand their constraints, the better your solutions will be. Ask lots of questions and be open and transparent.
- Show the solution as you build it: a common mistake is to show your solution too late. If there are issues that weren’t considered, the stakeholder will be frustrated, but so will your engineering team.
- Avoid a battle of opinions: try and de-risk opinion-based choices with data, and build a collaborative, mutually respectful relationship with your stakeholders.
How to avoid slowing down innovation as you scale:
- Have a very strong and intentional product culture. Share this clearly with new hires.
- Make the expectation of how they will work explicit in interviewing and onboarding. Encourage new hires to shed the habits of previous companies if they weren’t good at it.
Chapter 64 - Good Product Team/Bad Product Team
Good teams have a compelling product vision that they pursue with a missionary‐like passion. Bad teams are mercenaries.
With a grateful nod to Ben Horowitz's classic post “Good Product Manager/Bad Product Manager,” here are some of the important differences between strong product teams and weak teams:
- Good teams have a compelling product vision that they pursue with a missionary‐like passion. Bad teams are mercenaries.
- Good teams get their inspiration and product ideas from their vision and objectives, from observing customers' struggle, from analyzing the data customers generate from using their product, and from constantly seeking to apply new technology to solve real problems. Bad teams gather requirements from sales and customers.
- Good teams understand who each of their key stakeholders are, they understand the constraints that these stakeholders operate in, and they are committed to inventing solutions that work not just for users and customers, but also work within the constraints of the business. Bad teams gather requirements from stakeholders.
- Good teams are skilled in the many techniques to rapidly try out product ideas to determine which ones are truly worth building. Bad teams hold meetings to generate prioritized roadmaps.
- Good teams love to have brainstorming discussions with smart thought leaders from across the company. Bad teams get offended when someone outside their team dares to suggest they do something.
- Good teams have product, design, and engineering sit side by side, and they embrace the give and take between the functionality, the user experience, and the enabling technology. Bad teams sit in their respective silos, and ask that others make requests for their services in the form of documents and scheduling meetings.
- Good teams are constantly trying out new ideas to innovate, but doing so in ways that protect the revenue and protect the brand. Bad teams are still waiting for permission to run a test.
- Good teams insist they have the skill sets on their team, such as strong product design, necessary to create winning products. Bad teams don't even know what product designers are.
- Good teams ensure that their engineers have time to try out the prototypes in discovery every day so that they can contribute their thoughts on how to make the product better. Bad teams show the prototypes to the engineers during sprint planning so they can estimate.
- Good teams engage directly with end users and customers every week, to better understand their customers, and to see the customer's response to their latest ideas. Bad teams think they are the customer.
- Good teams know that many of their favorite ideas won't end up working for customers, and even the ones that could will need several iterations to get to the point where they provide the desired outcome. Bad teams just build what's on the roadmap, and are satisfied with meeting dates and ensuring quality.
- Good teams understand the need for speed and how rapid iteration is the key to innovation, and they understand this speed comes from the right techniques and not forced labor. Bad teams complain they are slow because their colleagues are not working hard enough.
- Good teams make high‐integrity commitments after they've evaluated the request and ensured they have a viable solution that will work for the customer and the business. Bad teams complain about being a sales‐driven company.
- Good teams instrument their work so they can immediately understand how their product is being used and make adjustments based on the data. Bad teams consider analytics and reporting a nice to have.
- Good teams integrate and release continuously, knowing that a constant stream of smaller releases provides a much more stable solution for their customers. Bad teams test manually at the end of a painful integration phase and then release everything at once.
- Good teams obsess over their reference customers. Bad teams obsess over their competitors.
- Good teams celebrate when they achieve a significant impact to the business results. Bad teams celebrate when they finally release something.
Chapter 65 - Top Reasons for Loss of Innovation
Organizations that lose the ability to innovate at scale are inevitably missing one or more of the following attributes:
- Customer‐centric culture.
- Compelling product vision.
- Focused product strategy.
- Strong product managers.
- Stable product teams.
- Engineers in discovery.
- Corporate courage.
- Empowered product teams.
- Product mindset.
- Time to innovate.
Chapter 66 - Top Reasons for Loss of Velocity
- Technical debt
- Lack of strong product managers
- Lack of delivery management
- Infrequent release cycles. Your team should release no less frequently than every two weeks (very good teams release multiple times per day). Correcting this typically means getting serious about test automation and release automation so the team can move quickly and release with confidence.
- Lack of product vision and strategy
- Lack of co‐located, durable product teams
- Not including engineers early enough during product discovery
- Not utilizing product design in discovery and instead having them try to do their work at the same time the engineers are trying to build
- Changing priorities
- A consensus culture
Chapter 67 - Establishing a Strong Product Culture
I think of product culture along two dimensions:
- Whether the company can consistently innovate to come up with valuable solutions for their customers. This is what product discovery is all about.
- Execution. It doesn't matter how great the ideas are if you can't get a productized, shippable version delivered to your customers. This is what product delivery is all about.
What does it really mean to have a strong innovation culture?
- Culture of experimentation—teams know they can run tests; some will succeed and many will fail, and this is acceptable and understood.
- Culture of open minds—teams know that good ideas can come from anywhere and aren't always obvious at the outset.
- Culture of empowerment—individuals and teams feel empowered to be able to try out an idea.
- Culture of technology—teams realize that true innovation can be inspired by new technology and analysis of data, as well as by customers.
- Culture of business‐ and customer‐savvy teams—teams, including developers, have a deep understanding of the business needs and constraints, and understanding of (and access to) the users and customers.
- Culture of skill‐set and staff diversity—teams appreciate that different skills and backgrounds contribute to innovative solutions—especially engineering, design, and product.
- Culture of discovery techniques—the mechanisms are in place for ideas to be tested out quickly and safely (protecting brand, revenue, customers, and colleagues).
What does it really mean to have a strong execution culture?
- Culture of urgency—people feel like they are in wartime, and that if they don't find a way to move fast, then bad things could happen.
- Culture of high‐integrity commitments—teams understand the need for (and power of) commitments, but they also insist on high‐integrity commitments.
- Culture of empowerment—teams feel as though they have the tools, resources, and permission to do whatever is necessary to meet their commitments.
- Culture of accountability—people and teams feel a deep responsibility to meet their commitments. Accountability also implies consequences—not necessarily being terminated, except in extreme and repeated situations, but more likely consequences to their reputations among their peers.
- Culture of collaboration—while team autonomy and empowerment is important, teams understand their even higher need to work together to accomplish many of the biggest and most meaningful objectives.
- Culture of results—is the focus on output or is the focus on results?
- Culture of recognition—teams often take their cues from what is rewarded and what is accepted. Is it just the team that comes up with the great new idea that gets rewarded, or the team that delivered on a brutally tough commitment? And what is the message if missing a commitment is seen as easily excusable?
So, if these characteristics help define each culture, this begs some pretty tough questions:
- Is an innovation culture in any way inherently at odds with an execution culture?
- Does a strong execution culture lead to a stressful (or worse) work environment?
- What types of people, including leaders, are attracted to, and needed, for each type of culture?
I can tell you that there do exist companies that are very strong at both consistent innovation and execution. Amazon is one of the best examples. However, it's also well known that the Amazon work environment is not for the faint of heart. I've found that most companies that are exceptionally strong at execution are pretty tough places to work.
In my experience working with companies, only a few companies are strong at both innovation and execution. Many are good at execution but weak at innovation; some are strong at innovation and just okay at execution; and a depressing number of companies are poor at both innovation and execution (usually older companies that lost their product mojo a long time ago, but still have a strong brand and customer base to lean on).
In any case, what I hope you and your team will consider doing is assess yourself along these dimensions of innovation and execution, and then ask yourselves where you would like to be, or think you need to be, as a team or company.
