Trust-Aware Clinical Decision Support with LLMs: Integrating Fast Heuristics and Slow Deliberative Reasoning for Healthcare AI
Keywords:
clinical decision support, large language models, dual-process theory, trustworthiness, hybrid reasoning, healthcare AI, system architecture, governanceAbstract
The integration of large language models into clinical decision support systems promises transformative improvements in diagnostic accuracy, treatment planning, and workflow efficiency. However, the inherent opacity of neural representations, coupled with the high stakes of clinical environments, demands a careful reexamination of how such systems earn and maintain trust among clinicians and patients. This paper proposes a trust-aware architecture for clinical decision support that explicitly operationalizes the dual-process theory of cognition, distinguishing between fast, intuitive heuristics generated by large language models and slow, deliberative reasoning executed through structured inference pipelines. We argue that a rigid separation between these modes is neither feasible nor desirable; instead, the two should interact in a controlled feedback loop that balances speed, accuracy, and explainability. The proposed framework addresses critical structural trade-offs in system design, including latency versus thoroughness, scalability versus safety, and model flexibility versus regulatory compliance. We examine governance mechanisms, deployment strategies, and policy implications that arise when such hybrid systems are embedded in existing healthcare infrastructures. Through cross-domain comparisons with autonomous driving and financial risk assessment, we draw lessons for ensuring robustness, fairness, and sustainability. The paper concludes with a set of forward-looking recommendations for researchers, developers, and policymakers aiming to build clinically reliable and ethically grounded artificial intelligence systems.
References
1. Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in health and medicine. Nature Medicine, 28(1), 31–38.
2. Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1), 44–56.
3. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623).
4. Marcus, G. (2022). The next decade in AI: Four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177.
5. Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
6. Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences, 40, e253.
7. Shanahan, M. (2024). Talking about large language models. Communications of the ACM, 67(2), 56–64.
8. Dou, Z., Cui, D., Yan, J., Wang, W., Chen, B., Wang, H., ... & Zhang, S. (2025). Dsadf: Thinking fast and slow for decision making. arXiv preprint arXiv:2505.08189.
9. Fu, L., Chen, X., Gao, K., Huang, X., & Tong, K. (2025, October). Memory-Augmented Knowledge Fusion with Safety-Aware Decoding for Domain-Adaptive Question Answering. In 2025 6th International Conference on Machine Learning and Computer Application (ICMLCA) (pp. 1-6). IEEE.
10. Liu, F., Shi, Y., Liu, Y., & Zhang, Y. (2024). Clinical reasoning with large language models: A survey. Journal of Biomedical Informatics, 150, 104608.
11. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 35, 24824–24837.
12. Seshia, S. A., Sadigh, D., & Sastry, S. S. (2022). Toward verified artificial intelligence. Communications of the ACM, 65(7), 46–55.
13. Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
14. Yin, Z., Chen, Y., & Zhang, L. (2024). Dynamic resource allocation for hybrid AI systems in healthcare. IEEE Transactions on Neural Networks and Learning Systems, 35(8), 10215–10228.
15. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
16. Price, W. N., & Cohen, I. G. (2019). Privacy in the age of medical big data. Nature Medicine, 25(1), 37–43.
17. Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H. R., Albarqouni, S., ... & Bakas, S. (2020). The future of digital health with federated learning. NPJ Digital Medicine, 3(1), 119.
18. Li, M., & Zhou, J. (2023). Balancing speed and accuracy in clinical AI: A multi-objective optimization approach. Artificial Intelligence in Medicine, 142, 102595.
19. Mandl, K. D., & Kohane, I. S. (2016). A health information infrastructure for the 21st century. New England Journal of Medicine, 375(18), 1769–1776.
20. Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., Gonzalez, C. G., & van Moorsel, A. (2020). The relationship between trust in AI and trustworthy AI. Nature Machine Intelligence, 2(10), 570–573.
21. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
22. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
23. Vayena, E., Blasimme, A., & Cohen, I. G. (2018). Machine learning in medicine: Addressing ethical challenges. PLoS Medicine, 15(11), e1002689.
24. Heaven, W. D. (2024). The hidden costs of large language model upgrades. MIT Technology Review, 127(3), 44–51.
25. Van der Waa, J., Schoonderwoerd, T., van Diggelen, J., & Neerincx, M. (2021). Interpretable confidence measures for decision support systems. International Journal of Human-Computer Studies, 148, 102593.
26. Shalev-Shwartz, S., Shammah, S., & Shashua, A. (2017). On a formal model of safe and scalable self-driving cars. arXiv preprint arXiv:1708.06374.
27. Bao, Y., Hilary, G., & Ke, B. (2024). Large language models in financial analysis: Opportunities and risks. Journal of Financial Economics, 152, 103772.
28. Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62(3), 54–60.
29. Seeber, I., Bittner, E., Briggs, R. O., De Vreede, T., De Vreede, G. J., Elkins, A., ... & Schwabe, G. (2020). Machines as teammates: A research agenda on AI in team collaboration. Information Systems Research, 31(3), 675–694.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
