Latency-Aware Resource Scheduling for Large Language Model Inference Systems
Keywords:
latency-aware scheduling, large language models, inference serving, resource management, fairness, sustainabilityAbstract
Large language model inference has become a central workload in contemporary cloud and edge infrastructures, yet its resource behavior differs substantially from traditional web services and smaller machine learning models. Autoregressive decoding, variable sequence lengths, memory-intensive key-value caches, and bursty request arrivals create latency distributions that are difficult to manage with conventional scheduling policies. This paper examines latency-aware resource scheduling for large language model inference systems from a system-level perspective. It argues that effective scheduling must be understood as a socio-technical and infrastructural problem rather than a narrow optimization of accelerator utilization. The discussion covers the changing nature of inference latency, architectural foundations for predictable serving, batching and placement strategies, predictive control, observability, fairness, governance, sustainability, and robustness. Particular attention is given to structural trade-offs among throughput, tail latency, cost, energy use, tenant isolation, and service equity. The paper also considers how policy choices inside platforms shape external accountability, including questions of prioritization, access, and environmental impact. It concludes that latency-aware scheduling should combine adaptive admission control, workload-aware batching, disaggregated resource pools, robust prediction, and transparent governance. Rather than treating latency as a single technical metric, the paper frames it as an operational contract that connects infrastructure design, business priorities, and public expectations. The resulting research agenda emphasizes resilience, fairness, and sustainability alongside performance.
References
1. Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74-80.
2. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2017). Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (pp. 613-627). USENIX Association.
3. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 611-626). Association for Computing Machinery.
4. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L. M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021). Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350.
5. Zhang, J., & Gao, H. (2026, March). LLM inference delay prediction based on machine learning algorithms. In 2026 International Conference on Generative Artificial Intelligence and Information Security (GAIIS) (pp. 37-41). IEEE.
6. Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media.
7. Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., & Malkhi, D. (2020). Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (pp. 443-462). USENIX Association.
8. Romero, F., Li, Q., Yadwadkar, N. J., & Kozyrakis, C. (2021). INFaaS: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (pp. 397-411). USENIX Association.
9. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M. (2021). Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1-15). Association for Computing Machinery.
10. Pope, R., Douglas, G., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 606-624.
11. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., & Sheng, Y. (2023). SGLang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104.
12. Miao, X., Shi, C., Duan, J., Xi, N., Lin, Y., Cui, B., & Zhang, C. (2024). SpotServe: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (pp. 1112-1127). Association for Computing Machinery.
13. Agache, A., Brooker, M., Ionescu, A., Liguori, A., Neugebauer, R., Piwonka, P., & Popa, R. A. (2020). Firecracker: Lightweight virtualization for serverless applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (pp. 419-434). USENIX Association.
14. Delimitrou, C., & Kozyrakis, C. (2014). Quasar: Resource-efficient and QoS-aware cluster management. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (pp. 127-144). Association for Computing Machinery.
15. Kaffes, K., Chong, T., Humphries, J. T., Belay, A., Mazieres, D., & Kozyrakis, C. (2019). Shinjuku: Preemptive scheduling for microsecond-scale tail latency. In 16th USENIX Symposium on Networked Systems Design and Implementation (pp. 345-360). USENIX Association.
16. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645-3650). Association for Computational Linguistics.
17. Masanet, E., Shehabi, A., Lei, N., Smith, S., & Koomey, J. (2020). Recalibrating global data center energy-use estimates. Science, 367(6481), 984-986.
18. Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017). Inherent trade-offs in the fair determination of risk scores. In 8th Innovations in Theoretical Computer Science Conference (pp. 43:1-43:23). Schloss Dagstuhl.
19. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J. F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems (Vol. 28, pp. 2503-2511). Curran Associates.
20. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., ... Wen, J. R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
