Load-Balanced GPU Computing for Irregular Graph Neural Network Workloads
Keywords:
graph neural networks, GPU computing, load balancing, irregular workloads, sparse computation, distributed training, systems architecture, sustainability, fairness, infrastructure governanceAbstract
Graph neural networks have become a central computational method for learning over relational data, yet their execution on graphics processing units remains stubbornly difficult to optimize. The difficulty arises less from arithmetic intensity than from irregularity: graph neighborhoods vary widely in size, memory accesses scatter across sparse structures, and the amount of useful work per vertex or edge changes across layers and training phases. This paper examines load-balanced GPU computing for irregular graph neural network workloads from a systems perspective. It argues that load balancing cannot be treated as a narrow kernel-level scheduling problem. Instead, it must be understood as an architectural and infrastructural property that spans graph partitioning, memory hierarchy management, multi-GPU communication, runtime adaptation, and deployment governance. The analysis considers how vertex-centric and edge-centric execution models distribute work, how warp-level divergence and memory latency create hidden imbalance, and how dynamic scheduling, graph reordering, compression, and cooperative kernels can improve utilization. It also discusses the trade-offs between preprocessing cost, runtime flexibility, numerical fidelity, and operational predictability. The paper then connects these technical choices to broader concerns of robustness, fairness, sustainability, and policy. Irregular workloads are not merely performance puzzles; they determine which applications can run at scale, under what latency and energy budgets, and with what degree of equitable access to accelerated computing. The conclusion emphasizes that load-balanced GPU systems for graph neural networks require co-design across algorithms, hardware, runtime systems, and institutional deployment practices.
References
1. Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
2. Hamilton, W. L., Ying, R., & Leskovec, J. (2017). Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (pp. 1024–1034).
3. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks. In International Conference on Learning Representations.
4. Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61–80.
5. Peng, H., Xie, X., Shivdikar, K., Hasan, M. A., Zhao, J., Huang, S., ... & Ding, C. (2024, April). Maxk-gnn: Extremely fast gpu kernel design for accelerating graph neural networks training. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (pp. 683-698).
6. Wang, M., Yu, L., Zheng, D., Gu, Q., Gavrilovska, A., Huang, G., & Xu, C. (2019). Salient: Optimizing graph neural network training on GPUs. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (pp. 820–833).
7. Zhang, Z., Leng, J., Ma, X., Miao, Y., Li, C., & Guo, M. (2020). Architectural implications of graph neural networks. IEEE Computer Architecture Letters, 19(1), 59–62.
8. Yan, M., Chen, Z., Deng, L., Ye, X., Zhang, Z., Yuan, H., & Zhang, Y. (2020). Characterizing and understanding GCNs on GPU. IEEE Computer Architecture Letters, 19(1), 1–4.
9. Hu, Y., Ye, Z., Wang, M., Yu, J., Zheng, D., Xu, C., & others. (2021). Featgraph: A flexible and efficient backend for graph convolutional network systems. IEEE Transactions on Parallel and Distributed Systems, 32(8), 2033–2047.
10. Fey, M., & Lenssen, J. E. (2019). Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
11. Wang, Y., Feng, B., Li, G., Wang, S., Ding, Y., Xie, Y., & Li, H. (2020). GNNAdvisor: An adaptive and efficient runtime system for GNN acceleration on GPUs. In 14th USENIX Symposium on Operating Systems Design and Implementation (pp. 515–531).
12. Jia, Z., Lin, S., Gao, R., Zaharia, M., & Aiken, A. (2019). Improving the accuracy, scalability, and performance of graph neural networks with Roc. Proceedings of Machine Learning and Systems, 1, 220–231.
13. Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., ... & Zaharia, M. (2019). PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (pp. 1–15).
14. Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). ZeRO: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–16).
15. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., ... & Gonzalez, J. E. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 611–626).
16. Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W. L., & Leskovec, J. (2018). Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 974–983).
17. Zheng, D., Song, X., Ma, C., Tan, Z., Ye, Z., Ding, J., ... & Wang, Y. (2020). DGL: Large-scale graph neural networks. In Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing (pp. 1–12).
18. Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74–80.
19. Barroso, L. A., & Hölzle, U. (2009). The datacenter as a computer: An introduction to the design of warehouse-scale machines. Morgan & Claypool.
20. Sze, V., Chen, Y. H., Yang, T. J., & Emer, J. S. (2017). Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12), 2295–2329.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
