Embodied Intelligence Agents with Large Language Models for Real-Time Environment Interaction
Keywords:
embodied intelligence; large language models; real-time interaction; edge computing; human-robot interaction; AI governance; robustness; fairness; sustainability; system architectureAbstract
Embodied intelligence agents that integrate large language models with physical or simulated environments represent a significant shift in artificial intelligence systems research. Rather than treating language as a detached interface for query answering, these agents use language as a coordinating medium for perception, planning, action, memory, and social interaction. This paper examines the system-level architecture, deployment constraints, and governance implications of embodied language agents operating in real time. It argues that the central challenge is not merely model capability but the alignment of semantic reasoning with the temporal, spatial, and material constraints of embodied action. The analysis covers multimodal grounding, closed-loop control, edge-cloud continuity, latency budgets, failure containment, and long-term operational sustainability. It also considers how safety, fairness, and accountability are shaped by distributional infrastructure, data provenance, and human oversight. Case illustrations from robotics, digital twins, assistive environments, and industrial automation are used to show how architectural choices create trade-offs among responsiveness, robustness, privacy, and cost. The paper concludes that real-time embodied language agents require hybrid architectures that combine fast reactive modules with slower deliberative reasoning, clear escalation pathways, and governance mechanisms that remain effective under partial observability and operational stress. Such systems must be designed as socio-technical infrastructures rather than standalone models, with continuous evaluation, observability, and policy adaptation built into their lifecycle.
References
1. Brooks, R. A. (1991). Intelligence without representation. Artificial Intelligence, 47(1-3), 139-159. https://doi.org/10.1016/0004-3702(91)90053-M
2. Pfeifer, R., & Bongard, J. (2006). How the body shapes the way we think: A new view of intelligence. MIT Press.
3. Duan, J., Yu, S., Tan, H. L., Zhu, H., & Tan, C. (2022). A survey of embodied AI: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2), 230-244. https://doi.org/10.1109/TETCI.2022.3141105
4. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Joshi, N., ... Zeng, A. (2022). Do as I can, not as I say: Grounding language in robotic affordances. arXiv. https://arxiv.org/abs/2204.01691
5. Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022). Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference on Machine Learning (pp. 9118-9147). PMLR.
6. Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023). Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (pp. 9493-9500). IEEE. https://doi.org/10.1109/ICRA48891.2023.10160591
7. Vemprala, S., Bonatti, R., Bucker, A., & Kapoor, A. (2024). ChatGPT for robotics: Design principles and model abilities. IEEE Access, 12, 55682-55696. https://doi.org/10.1109/ACCESS.2024.3387941
8. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, T., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv. https://arxiv.org/abs/2305.16291
9. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X
10. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (Vol. 36). Curran Associates.
11. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2), 99-134. https://doi.org/10.1016/S0004-3702(98)00023-X
12. Kober, J., Bagnell, J. A., & Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11), 1238-1274. https://doi.org/10.1177/0278364913495721
13. Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016). End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39), 1-40.
14. Jiang, H., Xin, G., Huang, Y., Zhu, G., Zhang, J., Lin, X., ... & Cao, Z. (2026). AgentOmnia: Scaling Agentic Models for Full-Scenario Applications. arXiv preprint arXiv:2607.23124.
15. Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74-80. https://doi.org/10.1145/2408776.2408794
16. Satyanarayanan, M. (2017). The emergence of edge computing. Computer, 50(1), 30-39. https://doi.org/10.1109/MC.2017.9
17. Shi, W., Cao, J., Zhang, Q., Li, Y., & Xu, L. (2016). Edge computing: Vision and challenges. IEEE Internet of Things Journal, 3(5), 637-646. https://doi.org/10.1109/JIOT.2016.2579198
18. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv. https://arxiv.org/abs/1606.06565
19. Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control. Viking.
20. Floridi, L., & Cowls, J. (2019). A unified framework of five principles for AI in society. Harvard Data Science Review, 1(1). https://doi.org/10.1162/99608f92.8cd550d1
21. Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., Anderson, H., Roff, H., Allen, G. C., Steinhardt, J., Flynn, C., hÉigeartaigh, S. Ó., Beard, S., Belfield, H., Farquhar, S., ... Amodei, D. (2018). The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv. https://arxiv.org/abs/1802.07228
22. Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501-507. https://doi.org/10.1038/s42256-019-0114-4
23. Winfield, A. F., & Jirotka, M. (2018). Ethical governance is essential to building trust in robotics and AI systems. Philosophical Transactions of the Royal Society A, 376(2133), 20180085. https://doi.org/10.1098/rsta.2018.0085
24. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems (Vol. 28). Curran Associates.
25. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2017). Clipper: A low-latency online prediction serving system. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (pp. 613-627). USENIX Association.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
