Machine Learning for Product Managers — The magic of qualitative feedbacks

Shiyu Zhu Shiyu Zhu
February 15, 2019 Big Data, Cloud & DevOps

Ready to learn Data Science? Browse courses like Data Science Training and Certification developed by industry thought leaders and Experfy in Harvard Innovation Lab.

Hello fellow PMs,

I’m guessing you reading this article because you are

a) New to building Machine Learning Product initiatives and struggles to find where to start

b) Looking for tips and guides on how you can iterate and learn quickly from your customers when it comes to ML products

c) Struggling to find kick-starter materials online which you can use and follow

Well, I had the same set of struggles and here is what I’ve learned and helped us over the past two months. Its insane how much you can learn from qualitative feedbacks and I hope you find this simple kit helpful and remember these are just my opinions and what worked for my team and I 🙂

experfy-blog

The Unicorn Idea

This is when you suddenly wake up in the middle of the time with a crazy idea that which will solve a problem for a specific market

Problem/Product Discovery (At least 5 customers)

Now its execution time, this is when you need to dive into the problem space and really try to understand what your customers are struggling with today and whether your idea is valid or not. There is no point developing a product that doesn’t solve a problem, no matter how cool the idea is.

In this step, you are actively working with your UX researcher, Designer, and Data Scientists to conduct customer sessions and understand Customer Challenges. It is crucial to involve your leading Data Scientist in this process.

Unlike building regular features, there are a lot more uncertainties with building data products. It is import to involve the DS early, it will allow them to assess whether we can address the problem with the data and technology we have available today.

The below are three questions I like to ask during these sessions:

  1. Tell me a bit about yourself and your role in the organisation
  2. Would you be able to describe to me what a typical day looks like for you?
  3. What would you say the biggest challenge is about your day to day and why is it?

We try to talk to at least 5 customers to identify a pattern and gather insights. When capturing feedback, I find Trello the easiest and below is a simple template I like to use

experfy-blog

Basic Prototype

After gathering the information, give your designer time to apply design thinking to the problem which you have identified and let their creativity go wild.

Do time box this process as you don’t need the perfect design. It is more important to come up with the right journey which you can use with your customer for further qualitative feedbacks. You are also going to be working closely with your customer, you can always hand hold them through the process.

Technology Feasibility

Unlike building a rules based product where it is always feasible.Work with your DS to define the feasibility of the proposed design and figure out if you can solve the problem with data science and technology available today.

Problem Discovery V2 (Few iterations)

Now assuming you have a vanilla ML model that ‘solves’ the problem, our natural instinct as PMs tells us we should build something in the UI for user interaction

STOP right there! This is where ML differs!!! ML products have lot uncertainties with many assumptions made. As a PM, it is crucial for you understand how your customers will be using the presented data.

This is where Problem Discovery V2 comes in. Invision recently released a feature with Craft, where you can insert customer data into Design Prototypes for a more realistic experience. Work with your designer to update the design and go out to the customer again and start running experiments.

Come up with a hypothesis and a set of goals you want to achieve during the sessions. At this stage, qualitative feedbacks is what you are interested in. You want to hear the customers as to why they are or not engaging and that’s why I don’t recommend sending out surveys as you will never find out why your customers are not taking certain actions.

I like to divide these sessions into four sections

  1. Understand the customer’s organisation, workflow and challenges (This will help you to find market fit and catch on anything you’ve missed)
  2. Walk them through the technology you have on a high level (You may get new ideas from customers )
  3. Take them through the basic prototype populated with their data and have them interact with it (This is where you are going to get qualitative feedback on both the experience and the information)
  4. Gather their feedback on the solution and how they would like to pay for it

In each iteration of this exercise, I will talk to at least 4 customers to establish a pattern and populate them onto Trello. This sounds insane and every time I conduct a customer interview using the prototype, I’m mind blown with amount qualitative feedback I get.

Because you don’t have a product UI available, this approach offers much more insights for the proposed solution and the model output than sending out surveys . It is going to massively reduce the amount of uncertainties when you go out to collect quantitive feedback and its a cheap way of doing it than spending months to build.

At the end of each iteration, I will summarise my learnings into the below sections and work with the respective parties for the next round of updates and insights.

  1. Product Learnings
  2. Design Learnings
  3. Science Learnings

I would go through this learning iteration a few times until I am comfortable. Ideally, you should see more positive every time you go out but failure is ok, the learnings are more important

Potential Market Fit

Now you have found a potential market for you product based on qualitative feedback. The next stage will be starting to gather quantitative feedback, this is something I am still learning and finding the right solution 🙂

These are what I’ve learned so far as a Noob PM and I hope you enjoyed reading. Remember these are just my experience and I’d love to hear your experience and any tips which you have!

Originally published at Towards Data Science

  • Experfy Insights

    Top articles, research, podcasts, webinars and more delivered to you monthly.

  • Shiyu Zhu

    Tags
    Data Science
    © 2021, Experfy Inc. All rights reserved.
    Leave a Comment
    Next Post
    How Will IIoT Create More Shareholder Value?

    How Will IIoT Create More Shareholder Value?

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    More in Big Data, Cloud & DevOps
    Big Data, Cloud & DevOps
    Cognitive Load Of Being On Call: 6 Tips To Address It

    If you’ve ever been on call, you’ve probably experienced the pain of being woken up at 4 a.m., unactionable alerts, alerts going to the wrong team, and other unfortunate events. But, there’s an aspect of being on call that is less talked about, but even more ubiquitous – the cognitive load. “Cognitive load” has perhaps

    5 MINUTES READ Continue Reading »
    Big Data, Cloud & DevOps
    How To Refine 360 Customer View With Next Generation Data Matching

    Knowing your customer in the digital age Want to know more about your customers? About their demographics, personal choices, and preferable buying journey? Who do you think is the best source for such insights? You’re right. The customer. But, in a fast-paced world, it is almost impossible to extract all relevant information about a customer

    4 MINUTES READ Continue Reading »
    Big Data, Cloud & DevOps
    3 Ways Businesses Can Use Cloud Computing To The Fullest

    Cloud computing is the anytime, anywhere delivery of IT services like compute, storage, networking, and application software over the internet to end-users. The underlying physical resources, as well as processes, are masked to the end-user, who accesses only the files and apps they want. Companies (usually) pay for only the cloud computing services they use,

    7 MINUTES READ Continue Reading »

    About Us

    Incubated in Harvard Innovation Lab, Experfy specializes in pipelining and deploying the world's best AI and engineering talent at breakneck speed, with exceptional focus on quality and compliance. Enterprises and governments also leverage our award-winning SaaS platform to build their own customized future of work solutions such as talent clouds.

    Join Us At

    Contact Us

    1700 West Park Drive, Suite 190
    Westborough, MA 01581

    Email: support@experfy.com

    Toll Free: (844) EXPERFY or
    (844) 397-3739

    © 2025, Experfy Inc. All rights reserved.