Xiaokunfeng2022

CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features > Xiaokun Feng, Dailing Zhang, Shiyu Hu, Xuchen Li, Meiqi Wu, Jing Zhang, Xiaotang Chen, Kaiqi Huang This is an official pytorch implementation of the paper CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features. \[5/2024\] CStrack's code is available! \[5/2024\] CStrack is accepted by ICML25! 📣 Overview Our motivation & Core modeling approach Effectively modeling and utilizing spatiotemporal features from RGB and other modalities (e.g., depth, thermal, and event data, denoted as X) is the core of RGB-X tracker design. Existing methods often employ two parallel branches to separately process the RGB and X input streams, requiring the model to simultaneously handle two dispersed feature spaces, which complicates both the model structure and computation process. More critically, intra-modality spatial modeling within each dispersed space incurs substantial computational overhead, limiting resources for inter-modality spatial modeling and temporal modeling. To address this, we propose a novel tracker, CSTrack, which focuses on modeling Compact Spatiotemporal features to achieve simple yet effective tracking. Specifically, we first introduce an innovative Spatial Compact Module that integrates the RGB-X dual input streams into a compact spatial feature, enabling thorough intra- and inter-modality spatial modeling. Additionally, we design an efficient Temporal Compact Module that compactly represents temporal features by constructing the refined target distribution heatmap. Extensive experiments validate the effectiveness of our compact spatiotemporal modeling method, with CSTrack achieving new SOTA results on mainstream RGB-X benchmarks. | Tracker | LasHeR (AUC) | RGBT234 (MSR) | VOT-RGBD22 (EAO) | DepthTrack (F-score) | VisEvent (AUC) | |------------|--------------|---------------|------------------|----------------------|----------------| | CSTrack | 60.8 | 70.9 | 77.4 | 65.8 | 65.2 | | SDSTrack | 53.1 | 62.5 | 72.8 | 61.9 | 59.7 | | OneTracker | 53.8 | 62.5 | 72.7 | 60.9 | 60.8 | | ViPT | 52.5 | 61.7 | 72.1 | 59.6 | 59.2 | | UnTrack | 51.3 | 62.5 | 72.1 | 61.0 | 58.9 | Data Preparation Our CSTrack is jointly trained on RGB and RGB-D/T/E datasets. Put these tracking datasets in ./data. It should look like: Set project paths Run the following command to set paths for this project After running this command, you can also modify paths by editing these two files Train Prepare pretrained backbone The backbone and patch embedding of CSTrack are initialized with pre-trained weights from Fast-iTPN. Please download the fastitpnbasecliple1600.pt checkpoint and place it in ./resource/pretrainedmodels. Stage1: Train CSTrack without Temporal Compact Module Stage2: Train CSTrack with Temporal Compact Module Then, run the following command: First, you need to set the PRETRAINEDPATH in ./experiments/cstracks2/cstracks2.yaml to the path of the model weights obtained from Stage 1 training. Then, run the following command: Test and evaluate on benchmarks First, you need to set settings.checkpointspath in ./lib/test/evaluation/local.py to the dir where the model checkpoint to be evaluated is stored. Then, run the following command to perform evaluation on different benchmarks. 📊 Model Zoo The trained models, and the raw tracking results are provided in the [](https://huggingface.co/Xiaokunfeng2022/CSTrack) ❤️Acknowledgement We would like to express our gratitude to the following open-source repositories that our work is based on: SeqtrackV2, SUTrack, ViPT, AQATrack, iTPN. Their contributions have been invaluable to this project.