Ready to learn Data Science? Browse courses like Data Science Training and Certification developed by industry thought leaders and Experfy in Harvard Innovation Lab.
Hello fellow PMs,
I’m guessing you reading this article because you are
a) New to building Machine Learning Product initiatives and struggles to find where to start
b) Looking for tips and guides on how you can iterate and learn quickly from your customers when it comes to ML products
c) Struggling to find kick-starter materials online which you can use and follow
Well, I had the same set of struggles and here is what I’ve learned and helped us over the past two months. Its insane how much you can learn from qualitative feedbacks and I hope you find this simple kit helpful and remember these are just my opinions and what worked for my team and I 🙂
The Unicorn Idea
This is when you suddenly wake up in the middle of the time with a crazy idea that which will solve a problem for a specific market
Problem/Product Discovery (At least 5 customers)
Now its execution time, this is when you need to dive into the problem space and really try to understand what your customers are struggling with today and whether your idea is valid or not. There is no point developing a product that doesn’t solve a problem, no matter how cool the idea is.
In this step, you are actively working with your UX researcher, Designer, and Data Scientists to conduct customer sessions and understand Customer Challenges. It is crucial to involve your leading Data Scientist in this process.
Unlike building regular features, there are a lot more uncertainties with building data products. It is import to involve the DS early, it will allow them to assess whether we can address the problem with the data and technology we have available today.
The below are three questions I like to ask during these sessions:
- Tell me a bit about yourself and your role in the organisation
- Would you be able to describe to me what a typical day looks like for you?
- What would you say the biggest challenge is about your day to day and why is it?
We try to talk to at least 5 customers to identify a pattern and gather insights. When capturing feedback, I find Trello the easiest and below is a simple template I like to use
Basic Prototype
After gathering the information, give your designer time to apply design thinking to the problem which you have identified and let their creativity go wild.
Do time box this process as you don’t need the perfect design. It is more important to come up with the right journey which you can use with your customer for further qualitative feedbacks. You are also going to be working closely with your customer, you can always hand hold them through the process.
Technology Feasibility
Unlike building a rules based product where it is always feasible.Work with your DS to define the feasibility of the proposed design and figure out if you can solve the problem with data science and technology available today.
Problem Discovery V2 (Few iterations)
Now assuming you have a vanilla ML model that ‘solves’ the problem, our natural instinct as PMs tells us we should build something in the UI for user interaction
STOP right there! This is where ML differs!!! ML products have lot uncertainties with many assumptions made. As a PM, it is crucial for you understand how your customers will be using the presented data.
This is where Problem Discovery V2 comes in. Invision recently released a feature with Craft, where you can insert customer data into Design Prototypes for a more realistic experience. Work with your designer to update the design and go out to the customer again and start running experiments.
Come up with a hypothesis and a set of goals you want to achieve during the sessions. At this stage, qualitative feedbacks is what you are interested in. You want to hear the customers as to why they are or not engaging and that’s why I don’t recommend sending out surveys as you will never find out why your customers are not taking certain actions.
I like to divide these sessions into four sections
- Understand the customer’s organisation, workflow and challenges (This will help you to find market fit and catch on anything you’ve missed)
- Walk them through the technology you have on a high level (You may get new ideas from customers )
- Take them through the basic prototype populated with their data and have them interact with it (This is where you are going to get qualitative feedback on both the experience and the information)
- Gather their feedback on the solution and how they would like to pay for it
In each iteration of this exercise, I will talk to at least 4 customers to establish a pattern and populate them onto Trello. This sounds insane and every time I conduct a customer interview using the prototype, I’m mind blown with amount qualitative feedback I get.
Because you don’t have a product UI available, this approach offers much more insights for the proposed solution and the model output than sending out surveys . It is going to massively reduce the amount of uncertainties when you go out to collect quantitive feedback and its a cheap way of doing it than spending months to build.
At the end of each iteration, I will summarise my learnings into the below sections and work with the respective parties for the next round of updates and insights.
- Product Learnings
- Design Learnings
- Science Learnings
I would go through this learning iteration a few times until I am comfortable. Ideally, you should see more positive every time you go out but failure is ok, the learnings are more important
Potential Market Fit
Now you have found a potential market for you product based on qualitative feedback. The next stage will be starting to gather quantitative feedback, this is something I am still learning and finding the right solution 🙂
These are what I’ve learned so far as a Noob PM and I hope you enjoyed reading. Remember these are just my experience and I’d love to hear your experience and any tips which you have!
Originally published at Towards Data Science
