Can ChatGPT Answer Patient Questions About In Vitro Fertilization Accurately?
As patients increasingly turn to artificial intelligence tools for medical information, researchers are evaluating how well conversational large language models handle complex topics like assisted reproductive technology. A recent comparative analysis presented at the American Society for Reproductive Medicine annual meeting examined how ChatGPT’s responses to questions about in vitro fertilization compare to established professional medical education materials.
Led by Nicole Fischer, MD, MPH, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, the research team sought to determine whether patients utilizing chatbots for fertility guidance receive complete and safe answers. To conduct the analysis, investigators used the free GPT-5.6 Luna platform to generate responses to common questions regarding the IVF process and associated risks. Four physicians subsequently reviewed these responses alongside official documents from the American Society for Reproductive Medicine.
Physician Ratings and Content Comparison
The reviewing physicians evaluated the materials on a 1-5 scale, with five representing the highest score. While the artificial intelligence model scored high marks for readability, it fell short of professional materials across multiple metrics:
- Accuracy: ChatGPT averaged 3.98 compared to 5.00 for ASRM materials.
- Completeness: The AI scored 3.40 while professional documents reached 4.88.
- Safety: Reviewers rated the chatbot at 2.92 compared to 4.88 for society resources.
- Quality: The tool achieved 3.38 versus a perfect 5.00 for professional content.
However, clarity ratings were comparable between the two sources, with the chatbot scoring 4.60 and professional materials scoring 4.63. Researchers also measured checklist adherence by evaluating how many key concepts from the ASRM ReproductiveFacts.org resource appeared in the text. While ASRM materials achieved a 100% adherence rate, the AI model averaged 94.1%.
Despite the high adherence score, investigators identified notable concerns regarding the reliability of the output. Half of the AI-generated answers contained an unsupported claim, and 4.2% included explicit misinformation. None of the ASRM responses contained unsupported claims or misinformation.
Success Estimates and Readability Factors
The analysis also tested the model’s predictions regarding IVF live birth success rates across different maternal ages. While the tool generally aligned with data from the Centers for Disease Control and Prevention and the Society for Assisted Reproductive Technology by showing a decline in birth rates with age, discrepancies emerged. The model overestimated success rates for age 35 while underestimating rates for ages 37 and 40.
Investigators noted that the phrasing of prompts influenced the readability of the responses. Conversational prompts yielded answers written at approximately an eighth-grade reading level, whereas generic prompts produced tenth-grade reading levels. In comparison, professional ASRM documents sat at roughly a twelfth-grade reading level.
Study Limitations and Clinical Implications
Study authors acknowledged several limitations, including a small sample size, the evaluation of only a single version of one large language model, subjective descriptive measures, and a lack of patient-level outcomes. They suggested that newer iterations and specialized medical models warrant further investigation.
Because large language models predict responses based on massive datasets, they remain susceptible to hallucinations—instances where the model produces confident, fluent text that contains fabricated details. Experts emphasize that fluent wording can easily mask omissions and errors, making critical health topics particularly vulnerable. Clinicians are encouraged to discuss artificial intelligence usage with patients, guide them on prompting models to cite sources, and direct them toward verified clinical resources.
