AI-ML Product Management: Navigating the Intersection of Artificial Intelligence and Machine Learning

 

Introduction:

In this article, I will guide you through the complete lifecycle of a Machine Learning product and explain the role of a Product Manager within it. While I won't go into extensive detail about Product Management, I will focus on the specific responsibilities related to AI/ML products, which are outlined in this article.

Rather than discussing various applications of ML, this article assumes that the Product Manager or a business leader has already identified an application and intends to apply ML to it. Instead, I will emphasize the importance of identifying a genuine problem, as it is one of the most crucial tasks for a Product Manager in a Machine Learning project.

Lastly, I've aimed to make this article comprehensive yet easy to understand, so readers won't just save it and forget about it. Therefore, certain fundamental concepts will not be explicitly defined here. You can easily find definitions through a quick Google search or using the GPT search function.

Table of Contents:

1.Understanding Product Management

2.Strategic Approach: Product Management in ML/AI

3.Product Manager's Responsibilities in the Complete ML Lifecycle

a. Scoping

b. Data

c. Modeling

d. Validation

e. Deployment

4.Additional Considerations

5.Tools and Techniques

 

Will companies now rush to AI, just as they did with cloud computing?

In the race towards cloud computing, many companies, regardless of their size, hastily embraced the trend or were persuaded by tech consulting firms. However, more than 50% of these companies failed to fully capitalize on the promised benefits. One of the contributing factors to this outcome, as stated by David Linthicum, the chief cloud strategy officer at Deloitte Consulting LLP, was the adoption of overly complex infrastructure without adequately considering the operational impact.

With the advent of the GPT wave, there is an impending frenzy to implement AI. A competent Product Manager will play a crucial role in ensuring that their company avoids repeating these past mistakes and successfully leverages the potential of AI.

Being Strategic: The Role of a Product Manager in AI/ML

In the current AI-driven landscape, both companies and individuals are eager to capitalize on the AI wave. Companies are incorporating "AI" into their domain names and product descriptions, while CEOs mention "AI" or "ML" in investor calls to attract investments. Although this may not pose a problem unless the company has no real association with AI and risks facing regulatory consequences from bodies like the FTC.

However, when it comes to product development, a Product Manager must adopt a strategic mindset and question the genuine need for AI. Investing in AI simply for the sake of it can lead to a loss in value and a compromised customer experience.

For a Product Manager operating in the realm of AI, considering the Return on Investment (ROI) and conducting a thorough Cost-Benefit analysis are crucial steps before embarking on an AI project. The ML product lifecycle is inherently less deterministic than traditional software development, which inherently brings higher risks. Consequently, a higher discount rate should be applied to AI projects. Assessing cost-benefit and calculating ROI in AI projects can be challenging due to the intertwined nature of various components. However, referring to industry case studies and competitor results can be helpful in this regard.

To navigate the complexities, it is advisable to develop a hypothesis, make reasonable assumptions, and construct a model. Defining success metrics and envisioning what success will look like are also essential aspects to consider during this process. By approaching AI projects strategically and setting clear objectives, a Product Manager can maximize the chances of achieving successful outcomes.

As a Product Manager, it is crucial not to be swayed solely by the novelty or technical sophistication of an ML model, experiment, or product. Instead, the primary focus should be on the significant business impact the product can generate. Making a positive business impact should be the ultimate goal.

In various informal conversations and industry conferences, numerous ML Product Managers have shared instances where they achieved greater business impact by strategically declining the implementation of ML. It's important to recognize that integrating ML into a system adds complexity, which comes with associated costs.

By carefully evaluating the potential benefits and drawbacks, a Product Manager can make informed decisions regarding the adoption of ML. This includes considering the overall impact on the business, assessing the costs and resources required for implementation, and weighing the potential return on investment. Sometimes, saying "no" to ML can lead to a more streamlined and efficient solution that aligns better with the business objectives. Ultimately, the focus should be on achieving tangible business value rather than chasing technological trends without a clear business rationale.

The AI/ML Product Lifecycle: Applying Product Management Tasks

The general tasks outlined by McKinsey for product management are valuable but require adaptation when it comes to different stages of an ML project. In the following sections, we will delve into each of the five stages, examine the specific tasks of a Product Manager (PM), and provide additional insights along the way.

Product Validation:

During the product validation stage, the Product Manager plays a crucial role in ensuring the accuracy and effectiveness of different ML models. A/B testing is a commonly used method for this purpose. However, for more comprehensive testing involving multiple models, advanced techniques like Multi-Armed Bandit and Contextual Bandit should be employed. Collaborating with IT managers, the PM should establish a system capable of testing multiple models, considering that performance on test data may significantly differ from real-world live data. Netflix has described an approach they used to accelerate the testing of numerous recommendation models, which can serve as a valuable reference.

It is essential for Product Managers to recognize that models inherit biases present in the data. To mitigate potential negative outcomes, PMs can create negative test cases to assess system performance on specific data segments. ML systems must strive for fairness, accountability, transparency, and explainability. A poorly designed system can result in harmful biases, deny opportunities, disproportionately fail in product recommendations, and cause harm.

Translating model metrics into business metrics is a critical task for Product Managers. PMs should continuously analyze the relationship between improvements in model metrics and corresponding business metrics. For instance, a 10% increase in Click Through Rate (CTR) could lead to a 100% improvement in Return on Investment (ROI) for marketing activities. Similarly, a 10% enhancement in product recommendations might drive more traffic to a particular vendor, potentially causing other vendors to lose prominence within the system. To ensure positive business outcomes, PMs need to adopt a holistic view of the system and make informed decisions.

One important takeaway is that ML metrics and business metrics are not the same. The relationship between these metrics is complex, and PMs must formulate hypotheses and validate results through experimentation. The establishment of a link between ML metrics and business outcomes is best achieved through experimentation. Aspiring ML Product Managers should develop a deep understanding of ML metrics like Precision, Recall, AUC, F1 score, among others. While these metrics won't be explained here, aspiring PMs are encouraged to delve into them, gaining an intuition for which metrics are relevant to specific problems. Attention to detail is paramount, with particular focus on outliers to prevent potential challenges later on.

In scenarios where multiple models exhibit similar performance, the Product Manager may prioritize an inherently explainable model, even if it is slightly inferior to the best-performing model. Both internal and external users are often hesitant to adopt ML models that lack explainability. Managing a black-box ML model with billions of parameters becomes challenging when setting guardrails. While there is no perfect answer, discussing this issue with the team and exploring potential solutions is important.

About the Author: Latha Thamma reddi is Strategic Information Technology Leader with Global project management experience driving change and digital transformation. Latha has made substantial contributions as a member of several Awards Committee, including serving as a Jury Award Committee member for ISEF 2023 in the Engineering Technology: Statics & Dynamics category, as well as being part of the Stevie and Globee Awards Jury Committee in various category expert profile review to rank and provide solid feedback. She actively participates in publishing articles, delivering keynote speeches, and acting as a distinguished guest of honor. She has been honored with Woman Excellence and Global Achievers award 2023 by Indian Achievers Forum. Globee 15th Annual 2023 Golden Bridge Business and Innovation Awards Winner. The Grand Globee winners in the 2023 Golden Bridge Award. Golden Globe Winner in the category Application Delivery Innovation. SILVER GLOBEE® WINNER in the category of Technical IT Professional of the Year..