GeoWorldNet: Multi-Modal World Model Learning for Autonomous Driving via Hyperspectral and LiDAR Scene Representation
Keywords:
autonomous driving, world models, hyperspectral imaging, LiDAR, multi-modal fusion, scene representation, robustness, sustainability, infrastructure, fairnessAbstract
Autonomous driving systems increasingly rely on learned world models that integrate multiple sensor modalities to predict future states and plan safe trajectories. Existing approaches predominantly fuse RGB camera data with LiDAR point clouds, yet they often fail to capture material composition and atmospheric phenomena that are critical for robust perception under adverse conditions. This paper introduces GeoWorldNet, a multi-modal world model architecture that fuses hyperspectral imagery with LiDAR point clouds to construct a unified scene representation for autonomous driving. We propose a hierarchical fusion framework that preserves spectral signatures from hyperspectral sensors while leveraging geometric precision from LiDAR to generate dense semantic, material, and depth maps. The world model is trained using a self-supervised objective that jointly predicts future hyperspectral cubes, LiDAR sweeps, and ego-motion, enabling long-horizon planning without extensive manual annotation. We analyze structural trade-offs in sensor placement, compute allocation, and latency constraints, and discuss system-level considerations including infrastructure requirements for real-time deployment, governance of training data diversity, sustainability of high-bandwidth processing, and fairness implications of material-based classification biases. Through extensive cross-domain comparisons with RGB-LiDAR baselines, we demonstrate that hyperspectral information significantly improves robustness to fog, rain, and low-light conditions while reducing false positive detections of non-vehicular objects. GeoWorldNet offers a pathway toward more resilient and interpretable autonomous driving systems by grounding world model learning in physically meaningful spectral properties. We conclude with policy recommendations for integrating hyperspectral sensing into next-generation autonomous vehicle certification frameworks.
References
1. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., ... & Zieba, K. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.
2. Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015). DeepDriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2722-2730).
3. Ha, D., & Schmidhuber, J. (2018). World models. arXiv preprint arXiv:1803.10122.
4. Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019). Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.
5. Held, D., Thrun, S., & Savarese, S. (2016). Robust lane detection for autonomous driving using deep learning. In IEEE Intelligent Vehicles Symposium (pp. 331-336).
6. Ge, X., Ding, J., & Liu, T. (2020). Hyperspectral imaging for material identification in autonomous driving. IEEE Transactions on Intelligent Vehicles, 5(4), 612-621.
7. Codevilla, F., Miiller, M., López, A., & Koltun, V. (2018). End-to-end driving via conditional imitation learning. In Proceedings of the IEEE International Conference on Robotics and Automation (pp. 4693-4700).
8. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017). CARLA: An open urban driving simulator. In Conference on Robot Learning (pp. 1-16).
9. Pomerleau, D. A. (1989). ALVINN: An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems (pp. 305-313).
10. Zhexiao Xiong, Xin Ye, Burhan Yaman, Sheng Cheng, Yiren Lu, Jingru Luo, Nathan Jacobs, and Liu Ren. UniDrive-WM: Unified understanding, planning and generation world model for autonomous driving. arXiv preprint arXiv:2601.04453, 2026.
11. Yin, H., & Li, X. (2021). LiDAR-based object detection for autonomous driving: A review. IEEE Transactions on Intelligent Vehicles, 6(2), 282-297.
12. Zhang, L., Zhang, L., & Du, B. (2016). Deep learning for remote sensing data: A technical tutorial on the state of the art. IEEE Geoscience and Remote Sensing Magazine, 4(2), 22-40.
13. Yang, J. X., Wang, J., Li, Z., Sui, C., Long, Z., & Zhou, J. (2025). HSLiNets: Evaluating Band Ordering Strategies in Hyperspectral and LiDAR Fusion. IEEE Geoscience and Remote Sensing Letters.
14. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770-778).
15. Li, S., & Zhang, H. (2022). Robust perception for autonomous driving under adverse weather. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6815-6832.
16. NHTSA. (2020). Automated driving systems: A vision for safety. National Highway Traffic Safety Administration Report.
17. Wachenfeld, W., & Winner, H. (2016). The release of autonomous vehicles. In Autonomous Driving (pp. 425-449). Springer.
18. Zhang, Y., & Wang, L. (2023). Energy-efficient deep learning for autonomous vehicles: A survey. ACM Computing Surveys, 55(10), 1-35.
19. Grigorescu, S., & Trasnea, B. (2020). A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3), 362-386.
20. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (pp. 5574-5584).
21. Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1501-1510).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Data Intelligence and AI Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
