|
 |
| LETTER TO THE EDITOR |
|
|
|
| Year : 2023 | Volume
: 55
| Issue : 4 | Page : 266-267 |
| |
Testing the knowledge and interpretation skills of ChatGPT in pharmacology examination of phase II MBBS
Anitha Elango1, Neevedha Kannan1, Isswariya Anandan1, Krishna Mohan Surapaneni2
1 Department of Pharmacology, Medical Education, Research, Panimalar Medical College Hospital and Research Institute, Varadharajapuram, Poonamallee, Chennai, Tamil Nadu, India
2 Department of Biochemistry, Medical Education, Research, Panimalar Medical College Hospital and Research Institute, Varadharajapuram, Poonamallee, Chennai, Tamil Nadu, India
| Date of Submission | 03-Apr-2023 |
| Date of Decision | 13-Jul-2023 |
| Date of Acceptance | 08-Aug-2023 |
| Date of Web Publication | 11-Sep-2023 |
Correspondence Address:
Krishna Mohan Surapaneni
Vice Principal and Professor, Departments of Biochemistry, Medical Education, Research, Panimalar Medical College Hospital and Research Institute, Varadharajapuram, Poonamallee, Chennai - 600 123, Tamil Nadu
India
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/ijp.ijp_188_23
How to cite this article:
Elango A, Kannan N, Anandan I, Surapaneni KM. Testing the knowledge and interpretation skills of ChatGPT in pharmacology examination of phase II MBBS. Indian J Pharmacol 2023;55:266-7 |
How to cite this URL:
Elango A, Kannan N, Anandan I, Surapaneni KM. Testing the knowledge and interpretation skills of ChatGPT in pharmacology examination of phase II MBBS. Indian J Pharmacol [serial online] 2023 [cited 2023 Oct 3];55:266-7. Available from: https://www.ijp-online.com/text.asp?2023/55/4/266/385494 |
Dear Editor,
| » Introduction | |  |
ChatGPT has attracted millions of users worldwide being the most advanced and rapid information generating artificial intelligence (AI) tool that imitates human conversations. ChatGPT is developed by San Francisco-based open AI with the highly advanced generative pretrained transformer-4, a deep learning tool that was made available to the public on November 30, 2022.[1] ChatGPT is being tested in various domains, and one remarkable achievement is its clearance in the United States Medical Licensing Examination (USMLE), which is considered to be one of the most competitive and challenging medical tests.[2] Ever since this commendable performance, ChatGPT has become a topic of discussion worldwide in regard to its usability in medical education and ability of AI-based tools to give a new shape to teaching and learning methods in medical sciences.[2] Furthermore, there are rising concerns about the misuse of ChatGPT. To prevent such exploitations, we need a framework of regulations for governance.[3] Thus, to extend the available evidence about the performance of ChatGPT in medical sciences, we intended to test the knowledge and interpretation skills of ChatGPT in pharmacology examination of phase II MBBS.
| » Methods | | |
We evaluated the performance of ChatGPT using pharmacology question papers of the final university examination in the second professional year of undergraduate medical education which was conducted in the month of February 2023. The syllabus of pharmacology has been segregated into two parts. Part I comprises general pharmacology, autonomic nervous system, central nervous system, peripheral nervous system, autacoids, cardiovascular system, blood, respiratory system, and kidneys, whereas the gastrointestinal system, hormones, antimicrobials, chemotherapy, and miscellaneous topics come under part II. For this assessment, we used the final university examination question paper of part II. Each part is evaluated for 100 marks with 30 marks being allotted for 2 essays (15 marks each) and 50 marks for 10 short answers (5 marks for each). The remaining 20 marks are awarded for multiple-choice questions (MCQs). We tested both theoretical and MCQ question papers of part II. All the questions were typed exactly on ChatGPT, and answers were generated including the four options provided for MCQs. Regeneration of the responses was not performed. All the answers generated in the first attempt were saved and later evaluated by our faculties in the department of pharmacology. Two evaluators separately assessed the answer script to prevent any bias. The scores were later compared, and the average was taken as the final score.
| » Results | | |
ChatGPT generated appropriate and succinct responses to all the questions. No diagrammatic- or flowchart-based representation was provided for the mechanism of action. Uses and adverse effects mentioned in all questions were provided as points in a nutshell without much elaboration which was appreciated by the evaluators. The essays were given as case-based problems from antimicrobials chapter. The first essay was typhoid and the second was urinary tract infection with amenorrhea.
The diagnoses of both the essays were right, and answers were given in short paragraphs. The response generated by ChatGPT for case 1 is represented in [Figure 1]. For the MCQs, the option of choice along with the explanation for the same was generated for all 20 questions. Out of 20 MCQs, ChatGPT has provided the correct answer for 15 questions. Individual marks were given for each question and summed up at the last. ChatGPT has managed to obtain a distinction in pharmacology with the average total score of 76%. The overall performance of ChatGPT in pharmacology was appreciable and highly suggests the beneficial role that AI could play in transforming medical education.
| » Discussion | | |
Our evaluation has suggested that ChatGPT has the potential to emerge as one of the most useful tools in medical education with its ability to generate appropriate and precise information to standard well-defined inputs. In accordance with the findings of our evaluation, studies have indicated that ChatGPT could play a beneficial role in facilitating “self-directed” method of learning and help students improve their knowledge and reasoning skills. However, the information provided by ChatGPT has to be validated as it did not show the standard references, from which information has been retrieved.[4] Hence, it can be predicted that AI tools such as ChatGPT have enormous potential to significantly transform clinical pharmacology, drug information services, medical writing, and the entire field of medicine when used cautiously and under proper guidance.[5]
| » Conclusion | | |
AI-based large language models such as ChatGPT can be used as an effective learning tool owing to its open accessibility and high speed in generating information. However, educators and students cannot solely depend on ChatGPT for learning as there is always a possibility of misleading or incorrect information being generated by such AI tool. Thus, appropriate testing and validation is essential before successful implementation of AI in advancing medical education and research.
Acknowledgements
Authors would like to extend their gratitude to OpenAI, an American artificial intelligence research laboratory for providing free access to ChatGPT.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| » References | | |
| 1. | |
| 2. | Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2:e0000198.  |
| 3. | Arif TB, Munaf U, Ul-Haque I. The future of medical education and research: Is ChatGPT a blessing or blight in disguise? Med Educ Online 2023;28:2181052. |
| 4. | Nisar S, Aslam MS. Is ChatGPT a Good Tool for T&CM Students in Studying Pharmacology? Available from: https://ssrn.com/abstract=4324310. [Last accessed on 2023Jan 14]. |
| 5. | Krumborg JR, Mikkelsen N, Damkier P, Ennis ZN, Henriksen DP, Lillevang-Johansen M, et al. ChatGPT: First glance from a perspective of clinical pharmacology. Basic Clin Pharmacol Toxicol 2023;133:3-5. doi: 10.1111/bcpt.13879. Epub 2023 May 12. PMID: 37170853. |
[Figure 1]
|
