Introduction

Product management has always been a dynamic and data-driven field. At Trality, we have faced the challenges of customer experience optimization in a shrinking company with fewer resources. In this article, we will discuss how we harnessed the power of ChatGPT, an AI-based language model, to produce data analyses on the level of a professional data scientist, enabling us to make more informed decisions for our platform.

Background

Trality used to employ a data analyst to work closely with our customer experience manager. Their role was to identify bottlenecks in the user experience and test hypotheses based on user feedback and drop-offs in the customer flow. Unfortunately, when the company downsized, we had to let go of our data analyst, and these tasks fell on the product management team.

The Challenge

As a product manager, I needed to identify the correlation between specific user actions and retention to improve our customer experience. However, analyzing data in Mixpanel was challenging due to numerous external factors, such as the Bitcoin price, influencing user behavior.

Collaborative Learning and AI-Assisted Data Analysis

To tackle this challenge, I teamed up with a co-worker experienced in predictive statistical analysis using Python. We identified relevant keywords to guide the data analysis process and leveraged ChatGPT to help transform data tables and run statistical analyses. By combining our domain knowledge with AI capabilities, we successfully conducted a customer experience audit and obtained valuable insights.

Results and Key Findings

With the help of ChatGPT, we were able to create survival analyses that provided insights into the impact of specific user behaviors on our key metrics. This allowed us to validate and invalidate some of our assumptions. However, creating predictive analyses, such as determining the optimal time to send nudges to users, remains a challenge.

Through our collaboration with ChatGPT, we discovered several critical points of information:

1. The optimal number of days after which a user is most likely to become a subscriber
2. The number of days after which a user is unlikely to become a subscriber
3. The ideal time for a user to perform core activities to increase the likelihood of becoming a subscriber
4. The optimal number of core activities a user should perform to have a positive effect on subscription rates

Tools and Techniques

We utilized a range of tools and techniques to conduct our analysis:

1. ChatGPT: AI-based language model for running statistical analyses and obtaining insights
2. Mixpanel and Metabase: Platforms to access our MongoDB database for extracting user data
3. Deepnote: An online platform for creating data pipelines

Challenges Faced and Lessons Learned

When analyzing the entire dataset, the correlations were too weak to draw any conclusions. To overcome this, we split our analysis into three categories:

1. Users who stayed longer than one day
2. Users who deployed a bot
3. Users who purchased a subscription

Moreover, the data size was too large to be downloaded from our database and Mixpanel. To work around this, we focused on a three-month timeframe of users.

Process Overview

Our process for data-driven decision-making comprised four main steps:

1. Extract data from Mixpanel and Metabase
2. Combine the extracted data
3. Transform the data into a usable format
4. Run the survival analysis in Deepnote

My Role in the Process

As the product manager responsible for this project, my primary responsibility was to identify areas of improvement in the customer experience and formulate hypotheses. In collaboration with my co-worker, we defined the scope and goals of our analysis, as well as the key metrics we aimed to optimize. I led the extraction and transformation of the data, utilizing tools like Mixpanel and Metabase, and managed the application of ChatGPT for running our survival analyses. Furthermore, I interpreted the results and applied the insights to our product strategy, ensuring that our data-driven decisions would result in a more engaging and effective user experience on our platform.

Conclusion

In conclusion, ChatGPT has proven to be a valuable tool for Trality, enabling us to conduct in-depth data analyses despite limited resources. By leveraging AI-based models, product managers can uncover insights and drive decision-making processes to improve customer experience and grow their business. Our experience showcases how AI can effectively supplement and empower human expertise in data-driven

‍