File size: 4,644 Bytes
a1bd3c6 8bdcd7d a1bd3c6 8bdcd7d 2c09573 8bdcd7d 2c09573 b4b70e6 2c09573 f8d9928 2c09573 b4b70e6 2c09573 8bdcd7d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
license: apache-2.0
pipeline_tag: image-feature-extraction
library_name: transformers
tags:
- earth-observation
- remote-sensing
- foundation-model
- multi-sensor
---
<p align="center">
<img src="imgs/logo.png" alt="Spectral Coverage" width="400"/>
</p>
# Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images
[](https://arxiv.org/abs/2506.19585)
[](https://huggingface.co/collections/gsumbul/smarties-685888bb5ecded3f802cc945)
[](https://opensource.org/licenses/Apache-2.0)
[](https://gsumbul.github.io/SMARTIES/)
## 🚀 Introduction
<!-- Spectral coverage figure -->
<p align="center">
<img src="imgs/spectra_fig.png" alt="Spectral Coverage" width="600"/>
</p>
From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet, but recent foundation models (FMs) are often specific to single sensors or to fixed combinations.
SMARTIES is a generic and versatile FM lifting sensor-dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, SMARTIES was trained as a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup, while modulating its feature representations to accept diverse sensors as input.
## ✨ Key Features
- 🛰️ **Multi-Sensor Representations:** SMARTIES enables sensor-agnostic processing of Earth observation data, including optical (e.g., Sentinel-2), radar (e.g., Sentinel-1), and sub-meter resolution RGB (e.g., Maxar) imagery and unseen ones in a zero-shot manner.
- 🌈 **Spectrum-Aware Projections:** SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space: given a specific sensor, each one of its bands is projected by projection layers specific to wavelength ranges.
- ⚡ **Lightweight and Scalable:** SMARTIES is designed to be lightweight and scalable, making it suitable for a wide range of remote sensing applications.
- 🔀 **Flexible Band Combinations:** SMARTIES can handle arbitrary combinations of spectral bands from different sensors, enabling flexible remote sensing applications.
- 🔄 **Downstream Transfer:** SMARTIES enables downstream transfer using a unified model across a diverse set of sensors and tasks, including scene classification, semantic segmentation, and multi-label classification.
<!-- Model architecture figure -->
<p align="center">
<img src="imgs/model_fig.png" alt="SMARTIES Model Architecture" width="700"/>
</p>
This repository contains the model weights of SMARTIES (ViT-L).
## 🧩 Using SMARTIES
SMARTIES is designed to be flexible and can be easily adapted to new datasets and sensors. You can easily use SMARTIES with a single line of code with Hugging Face transformer interface:
```python
model = transformers.AutoModel.from_pretrained(
"gsumbul/SMARTIES-v1-ViT-L",
trust_remote_code=True
)
```
[A Jupyter notebook](https://github.com/gsumbul/SMARTIES/blob/main/SMARTIES_huggingface.ipynb) is provided in [SMARTIES GitHub page](https://github.com/gsumbul/SMARTIES) to show in detail how to use pretrained model weights. The details of SMARTIES are described in our paper, available on [arXiv](https://arxiv.org/abs/2506.19585).
## 📣 Attribution
If you use SMARTIES, please cite the paper:
```
@article{smarties,
title={{SMARTIES}: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images},
author={Gencer Sumbul and Chang Xu and Emanuele Dalsasso and Devis Tuia},
journal={arXiv preprint arXiv:2506.19585},
year={2025}
}
```
## 📄 License
This repository is released under the Apache v2 License.
## 🙏 Acknowledgements
SMARTIES is supported by the European Space Agency (ESA) through the Discovery and Preparation Program, and is part of the project Toward a Foundation Model for Multi-Sensor Earth Observation Data with Language Semantics. |