top of page
Search

Has new updates to my Natural Language Bot lead to more Customer Retention and Satisfaction?


Keywords - A/B testing, Hypothesis testing, NLP, Sentiment Analysis, Power analysis, Transformers, Linear Discriminant Analysis (LDA), ChatGPT, NLTK, Python, Pandas, Matplotlib


Natural Language Bots are becoming exponentially popular in enhancing customer experiences and improving business revenues, especially when integrated on websites compared to traditional bots .


However, more quantitative studies are needed to understand the satisfaction and pain-points of customer interactions. We analyze conversations with 2 versions of personalized chatbot (ChatGPT taken for demonstration purposes) across the globe in various languages and statistically test if upgraded version is leading to better customer satisfaction or not.


The chatbot market is expected to grow from ~1 billion dollar to ~25 billion dollars in a 10 year span from 2020 - 2030


Business Questions


  1. Identify ways of quantifying customer experience via personalized chat bot interaction.

  2. Is staying on top of technology integration with current platforms actually increasing business revenues?

  3. What are the pain-points in this exchange for future customizations and updates in the bot - drop-off topics, customer retention rate etc?

  4. Are new customizations leading to an expected positive impact (A/B testing)?

  5. Measure long-term impact on business ROI.


In this project -


  1. We perform sentiment analysis on A-data and B-data to get the % of conversations with high-confidence positive user-experience (sentiment) before and after the bot upgrade.

  2. We perform Power Analysis to get minimum number of samples required from both the datasets to have statistically significant results

  3. We find that there is 2.5 % increase (57.8 % to 60.2 %) in % of high-confidence positive sentiment conversations.

  4. This is statistically significant and as expected !

  5. We also create WordCloud of high-confidence negative sentiment conversations and find that majority of the topics were related to programming, code, data and modeling.


Data


Real world use case would be a company's webpage with Natural Language Bot assisting in purchases. This interaction decides if a customer ends up buying a product or not (conversion rate).


We will be using ChatGPT-3.5 and ChatGPT-4 conversation data for demonstration purposes. We have conversation exchanges with ChatGPT from users across the globe collected over a period of -


  1. 6 months (Nov'22 - Apr'23, hereafter called A-data) - 52,000 conversations

  2. 3 months (Apr'23 - Jun'23, hereafter called B-data) - 40,000 conversations

This data is retrieved from the shareGPT platform in json format with 3 columns -


  1. 'id', - unique id of a conversation exchange

  2. 'from' - 'human' or 'gpt'

  3. 'value' - natural language text of the interaction

This study goes one step further on recently published work on this dataset (10th Dec 2023) - Early ChatGPT User Portrait through the Lens of Data.


Authors found GPT to be POSITIVE (>80%), NEUTRAL (8%) and NEGATIVE (12%) even when given with negative prompts, suggesting overall positive tone of this Natural Language Bot.










2 views0 comments

Comments

bottom of page