Energy-Efficient Edge Intelligence: Dynamic Fast–Slow LLM Decision Routing for Resource-Constrained AI Systems
Keywords:
edge intelligence, large language models, energy efficiency, fast-slow decision routing, resource-constrained systems, dual-process theory, AI governanceAbstract
The proliferation of large language models (LLMs) on resource-constrained edge devices presents a fundamental tension between computational intensity and energy efficiency. This paper proposes a novel architectural paradigm—dynamic fast–slow decision routing—that adaptively allocates inference tasks between lightweight, energy-efficient models and computationally expensive LLMs based on contextual complexity, latency requirements, and energy budgets. Drawing inspiration from cognitive dual-process theory, the framework distinguishes between fast, heuristic-driven pathways suitable for routine or low-stakes queries and slow, analytical pathways reserved for ambiguous, high-uncertainty, or safety-critical scenarios. We examine the system-level implications of such routing across multiple dimensions: energy consumption profiles, deployment infrastructure on heterogeneous edge platforms, governance of model selection policies, and robustness under varying network and environmental conditions. Trade-offs between accuracy, throughput, and energy cost are analyzed through the lens of multi-objective optimization without relying on formal mathematics. Case studies in smart agriculture, autonomous navigation, and real-time health monitoring illustrate how dynamic routing can extend battery life while preserving task performance. The paper further addresses fairness concerns arising from differential treatment of user queries and proposes policy frameworks for transparent routing decisions. By integrating fast–slow decision routing with emerging techniques such as speculative decoding, early exit mechanisms, and knowledge distillation, we outline a sustainable path for deploying LLM-based intelligence at the edge. The work concludes with a forward-looking discussion on the need for standardized benchmarks, interpretability of routing policies, and cross-layer energy accounting to enable trustworthy and efficient edge AI systems.
References
1. Wang, Y., Wang, S., Xu, Z., & Liu, J. (2020). Edge intelligence: A survey. ACM Computing Surveys, 52(6), 1–36. https://doi.org/10.1145/3342100
2. Han, S., Mao, H., & Dally, W. J. (2016). Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In ICLR 2016. https://arxiv.org/abs/1510.00149
3. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
4. Caviglia, L., & Benini, L. (2019). Energy-efficient edge computing for deep learning: A survey. IEEE Internet of Things Journal, 6(5), 8017–8030. https://doi.org/10.1109/JIOT.2019.2915408
5. Teerapittayanon, S., McDanel, B., & Kung, H. T. (2016). BranchyNet: An early-exit framework for deep neural networks. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (pp. 1100–1107). IEEE. https://doi.org/10.1109/ALLERTON.2016.7852376
6. Lee, D. D., & Chai, J. (2022). Adaptive inference for on-device language models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4235–4247. https://doi.org/10.18653/v1/2022.emnlp-main.284
7. Dou, Z., Cui, D., Yan, J., Wang, W., Chen, B., Wang, H., ... & Zhang, S. (2025). Dsadf: Thinking fast and slow for decision making. arXiv preprint arXiv:2505.08189.
8. Shih, K., Deng, Z., Chen, X., Zhang, Y., & Zhang, L. (2025, May). DST-GFN: A Dual-Stage Transformer Network with Gated Fusion for Pairwise User Preference Prediction in Dialogue Systems. In 2025 8th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE) (pp. 715-719). IEEE.
9. Satyanarayanan, M. (2017). The emergence of edge computing. Computer, 50(1), 30–39. https://doi.org/10.1109/MC.2017.9
10. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2704–2713). https://doi.org/10.1109/CVPR.2018.00286
11. Hinton, G. E., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
12. Bolukbasi, T., Wang, M., Dekel, O., & Saligrama, V. (2017). Adaptive neural networks for efficient inference. In ICML 2017. https://arxiv.org/abs/1702.07811
13. Kaya, Y., Hong, S., & Dumitras, T. (2019). Shallow-deep networks: Understanding and mitigating network overthinking. In ICML 2019. https://arxiv.org/abs/1810.07052
14. Xue, M., He, Y., & Liu, J. (2022). Learning to route for energy-efficient LLM inference on edge. Proceedings of the 2022 International Conference on Embedded Software (EMSOFT), 81–92. https://doi.org/10.1145/3551665.3551677
15. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from transformers via speculative decoding. In ICML 2023. https://arxiv.org/abs/2211.17192
16. Bhatt, V., & Bhatt, P. (2021). Energy-efficient deep learning on edge: A survey. IEEE Access, 9, 147210–147234. https://doi.org/10.1109/ACCESS.2021.3124003
17. Neiswanger, W., & Xing, E. P. (2021). Predictive uncertainty quantification for routing in neural networks. In NeurIPS 2021. https://arxiv.org/abs/2106.00033
18. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://doi.org/10.1038/nature14236
19. Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017). Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12), 2295–2329. https://doi.org/10.1109/JPROC.2017.2761740
20. Lane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., Jara, L., Kawsar, F., & Matic, A. (2016). DeepX: A software accelerator for low-power deep learning inference on mobile devices. In Proceedings of the 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) (pp. 1–12). IEEE. https://doi.org/10.1109/IPSN.2016.7460653
21. Horowitz, M. (2014). 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) (pp. 10–14). IEEE. https://doi.org/10.1109/ISSCC.2014.6757322
22. Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., Guestrin, C., & Krishnamurthy, A. (2018). TVM: An automated end-to-end optimizing compiler for deep learning. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI) (pp. 578–594). USENIX Association.
23. Li, Z., Hoiem, D., & Hoi, S. C. H. (2018). Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2935–2947. https://doi.org/10.1109/TPAMI.2017.2773081
24. Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. In ICLR 2015. https://arxiv.org/abs/1412.6572
25. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2
26. Tsourounis, D., Andriopoulou, M., & Lymberopoulos, D. (2023). Efficient pest detection on edge devices using knowledge distillation. IEEE Transactions on AgriFood Electronics, 1(2), 112–123. https://doi.org/10.1109/TAFE.2023.3270247
27. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A. Y. (2009). ROS: An open-source Robot Operating System. In ICRA Workshop on Open Source Software.
28. Chang, H.-Y., & Chen, Y.-C. (2022). TinyTransformer: A compact language model for wearable health assistants. ACM Transactions on Embedded Computing Systems, 21(6), 1–22. https://doi.org/10.1145/3558260
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
