A Study to Analyse the Inaccuracies in ChatGPT- Powered Responses to User Queries

Abstract

This paper examines the inaccuracies of ChatGPT in responding to user queries and analyses the inaccuracies associated with different contexts. Conversational AI tools such as ChatGPT have transformed human interaction with technology to create a seamless and real-time replying mechanism to diverse queries. However, despite their abilities, such tools are not free of errors. The paper aims to examine the nature, frequency, and causes of ChatGPT generated responses to user queries by using a quantitative approach. A mixed sample of queries from the teachers of the Virtual University of Pakistan was collected. A structured frequency distribution was created to categorize error types. Data was analysed using descriptive methods, and statistical tools were applied to determine overall error rates and distributions across categories. The findings of this research demonstrate significant inaccuracies in ChatGPT responses in particular domains. The results highlight prevalent error patterns and their effects on user experience and confidence in AI technologies. This paper concludes with recommendations on how the accuracy of ChatGPT responses can be improved by emphasizing the need for model-enhanced training methodologies and refinement.