Prompt Compression and Selective Injection for Resource-Constrained Edge Intelligence Applications
Keywords:
prompt compression, selective injection, edge intelligence, large language models, resource-constrained systems, system architecture, fairness, governanceAbstract
The proliferation of large language models has created a significant tension between their immense capabilities and the severe resource constraints of edge computing environments. Deploying such models on smartphones, IoT devices, and embedded systems demands radical efficiency in how prompts are constructed and transmitted. This paper presents a comprehensive system-level analysis of prompt compression and selective injection as architectural strategies for enabling edge intelligence. We examine the structural trade-offs between prompt fidelity, inference latency, energy consumption, and memory footprint, framing compression not as a lossy heuristic but as a managed degradation of semantic context under resource budgets. The discussion spans the full stack from model-agnostic compression pipelines to adaptive gating mechanisms that dynamically determine which tokens, demonstrations, or instructions reach the model, and when. We further investigate the implications for fairness, robustness, and long-term governance, highlighting how compressed prompts can amplify existing biases or erode safety guardrails if not carefully regulated. The paper articulates a design philosophy in which the prompt ceases to be a static artifact and becomes a fluid, resource-aware construct continuously negotiated between local edge agents and cloud-based orchestration layers. Throughout, we ground the analysis in recent empirical findings and propose a roadmap for sustainable, accountable edge intelligence systems that treat prompt engineering as a first-class systems concern.
References
1. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. In Advances in neural information processing systems (pp. 1877–1901).
2. Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023). LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 13358–13376).
3. Mu, J., Li, X., & Goodman, N. (2023). Learning to compress prompts with gist tokens. In Advances in neural information processing systems.
4. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in neural information processing systems (pp. 9459–9474).
5. Satyanarayanan, M. (2017). The emergence of edge computing. Computer, 50(1), 30–39.
6. Min, S., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th annual meeting of the Association for Computational Linguistics (pp. 5316–5330).
7. Zhu, W., & Tan, M. (2023, December). SPT: Learning to selectively insert prompts for better prompt tuning. In Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 11862-11878).
8. Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., ... & Stoica, I. (2018). Ray: A distributed framework for emerging AI applications. In 13th USENIX symposium on operating systems design and implementation (pp. 561–577).
9. Xu, Y., Zhu, C., Wang, S., Sun, S., Cheng, H., Li, X., ... & Liu, Y. (2023). Prompting large language models with dynamic planning for compositional generalization. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 12586–12601).
10. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.
11. Zhao, W., Queralta, J. P., & Westerlund, T. (2022). Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In 2022 IEEE symposium series on computational intelligence (pp. 489–496).
12. Patterson, D. A., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., ... & Dean, J. (2021). Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350.
13. Wu, C.-J., Brooks, D., Chen, K., Chen, D., Choudhury, S., Dukhan, M., ... & Hazelwood, K. (2022). Sustainable AI: Environmental implications, challenges and opportunities. In Proceedings of the 5th conference on machine learning and systems (pp. 795–813).
14. Nielsen, J. (1993). Usability engineering. Academic Press.
15. Kelly, F. P. (1979). Reversibility and stochastic networks. Cambridge University Press.
16. Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and machine learning. fairmlbook.org.
17. Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019). Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 conference on empirical methods in natural language processing (pp. 2153–2162).
18. Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency (pp. 59–68).
19. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., ... & Zhao, S. (2021). Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1–2), 1–210.
20. Sodagar, I. (2011). The MPEG-DASH standard for multimedia streaming over the internet. IEEE MultiMedia, 18(4), 62–67.
21. Chen, L., Lee, K., Srinivas, A., & Abbeel, P. (2021). Decision transformer: Reinforcement learning via sequence modeling. In Advances in neural information processing systems (pp. 15084–15097).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
