RTMPose Spatial Representation Enhancement

Official implementation of A Lightweight Spatial Representation Enhancement Method for RTMPose Toward Real-Time Generalizable Pose Estimation.

This repository provides two complementary enhancements for RTMPose:

• P4/P5 Semantic-Detail Fusion: a lightweight feature-fusion module that introduces higher-resolution spatial details into the prediction pathway.
• Scale-Adaptive Crop Resolution (SACR): an input-side strategy that adjusts crop padding according to target scale.

The method is evaluated on:

• COCO human pose estimation
• AP-10K animal pose estimation
• COCO-WholeBody keypoint estimation

---

Overview

RTMPose mainly relies on the low-resolution P5 feature for coordinate classification. Although P5 contains strong semantic information, its limited spatial resolution may reduce localization accuracy, particularly for small-scale targets and fine-grained keypoints.

We enhance the spatial representation of RTMPose through two complementary components.

P4/P5 Semantic-Detail Fusion

The higher-resolution P4 feature supplements spatial details that may be weakened in P5. P4 and P5 are first projected to the same channel dimension. P5 is then bilinearly upsampled to the P4 resolution, and the two features are fused by element-wise addition followed by lightweight convolutional refinement.

The fused feature remains at stride 16 and is directly passed to the original RTMCCHead.

Scale-Adaptive Crop Resolution

For a target bounding box with width (w) and height (h), the target scale is defined as:

$$ a = \sqrt{w h} $$ The scale thresholds are:

Small / Medium threshold: 96
Medium / Large threshold: 160

For COCO and AP-10K:

Small targets:   padding = 1.15
Medium targets:  padding = 1.15
Large targets:   padding = 1.25

For COCO-WholeBody, a more conservative setting is used:

Small targets:   padding = 1.20
Medium targets:  padding = 1.20
Large targets:   padding = 1.25

SACR introduces no learnable parameters and no additional model-side inference FLOPs.

---

Highlights

• Lightweight P4/P5 semantic-detail fusion
• No modification to the coordinate-classification mechanism of RTMCCHead
• Scale-adaptive input cropping without additional learnable parameters
• Consistent improvements across RTMPose-S, RTMPose-M, and RTMPose-L
• Cross-task evaluation on human, animal, and whole-body pose estimation
• Real-time inference performance is preserved

---

Main Results

COCO val2017

The official RTMPose baselines use 256 × 192 inputs, while our COCO models use 288 × 216 inputs.

Model	Official Input	Official AP	Ours Input	Ours AP	AP Gain
RTMPose-S	256 × 192	72.13	288 × 216	73.66	+1.53
RTMPose-M	256 × 192	74.60	288 × 216	75.87	+1.27
RTMPose-L	256 × 192	75.80	288 × 216	77.04	+1.24

Cross-Task Generalization

Dataset	Model	Official Input	Official AP	Ours Input	Ours AP	AP Gain
AP-10K	RTMPose-M	256 × 256	72.20	288 × 288	77.32	+5.12
COCO-WholeBody	RTMPose-M	256 × 192	58.20	256 × 192	61.15	+2.95

Accuracy–Efficiency Trade-off

Task	Model	RTMPose Latency	Ours Latency	Ours FPS
COCO	RTMPose-S	6.170 ms	6.707 ms	149.11
COCO	RTMPose-M	8.719 ms	8.618 ms	116.04
COCO	RTMPose-L	10.701 ms	10.365 ms	96.48
AP-10K	RTMPose-M	8.579 ms	8.551 ms	116.95

Efficiency benchmarking protocol:

GPU:                 RTX 4070
Precision:           [FP32 / FP16]
Batch size:          1
Flip test:           disabled
Warm-up iterations:  50
Measured iterations: 300

Absolute latency depends on the hardware and software environment. Official and proposed models should therefore be compared under the same benchmarking setup.

---

Repository Structure

RTMPose-SpatialEnhancement/
│
├── assets/
│   └── p4p5_fusion_overview.png
│
├── configs/
│   ├── coco/
│   │   ├── rtmpose_s/
│   │   ├── rtmpose_m/
│   │   └── rtmpose_l/
│   ├── ap10k/
│   └── wholebody/
│
├── rtmpose_spatial_enhancement/
│   ├── models/
│   │   └── necks/
│   │       └── p4p5_add_fusion_neck.py
│   └── datasets/
│       └── transforms/
│           └── sacr.py
│
├── LICENSE
└── README.md

---

Configurations

COCO

Model	Input	Epochs	Configuration
RTMPose-S	288 × 216	420	rtmpose-s_420e_coco-288x216_p4p5_sacr.py
RTMPose-M	288 × 216	420	rtmpose-m_420e_coco-288x216_p4p5_sacr.py
RTMPose-L	288 × 216	420	rtmpose-l_420e_coco-288x216_p4p5_sacr.py

AP-10K

Model	Input	Epochs	Configuration
RTMPose-M	288 × 288	210	rtmpose-m_210e_ap10k-288x288_p4p5_sacr.py

COCO-WholeBody

Model	Input	Epochs	Configuration
RTMPose-M	256 × 192	420	rtmpose-m_420e_coco-wholebody-256x192_p4p5_sacr.py

---

Installation

This repository is designed to work with a clean official MMPose installation.

1. Prepare MMPose

Install MMPose and its dependencies according to the official MMPose installation instructions.

The repository expects an MMPose 1.x-style environment with compatible versions of PyTorch, MMCV, MMEngine, and MMDetection.

2. Clone this repository

git clone https://github.com/ChenJie1029/RTMPose-SpatialEnhancement.git
cd RTMPose-SpatialEnhancement

The custom modules are registered through:

custom_imports = dict(
    imports=['rtmpose_spatial_enhancement'],
    allow_failed_imports=False,
)

Make sure that the repository root is available in PYTHONPATH.

Linux

export PYTHONPATH=/path/to/RTMPose-SpatialEnhancement:$PYTHONPATH

Windows PowerShell

$env:PYTHONPATH="C:\path\to\RTMPose-SpatialEnhancement;$env:PYTHONPATH"

A clean official MMPose installation is strongly recommended. See Troubleshooting if duplicate registry errors occur.

---

Dataset Preparation

The configuration files use the following dataset layouts.

COCO

data/
└── coco/
    ├── annotations/
    │   ├── person_keypoints_train2017.json
    │   └── person_keypoints_val2017.json
    ├── train2017/
    └── val2017/

AP-10K

data/
└── ap10k/
    ├── annotations/
    │   ├── ap10k-train-split1.json
    │   ├── ap10k-val-split1.json
    │   └── ap10k-test-split1.json
    └── data/

COCO-WholeBody

data/
└── coco/
    ├── annotations/
    │   ├── coco_wholebody_train_v1.0.json
    │   └── coco_wholebody_val_v1.0.json
    ├── person_detection_results/
    │   └── COCO_val2017_detections_AP_H_56_person.json
    ├── train2017/
    └── val2017/

---

Training

Run the commands from the MMPose root directory.

COCO RTMPose-S

python tools/train.py \
    /path/to/RTMPose-SpatialEnhancement/configs/coco/rtmpose_s/rtmpose-s_420e_coco-288x216_p4p5_sacr.py

COCO RTMPose-M

python tools/train.py \
    /path/to/RTMPose-SpatialEnhancement/configs/coco/rtmpose_m/rtmpose-m_420e_coco-288x216_p4p5_sacr.py

COCO RTMPose-L

python tools/train.py \
    /path/to/RTMPose-SpatialEnhancement/configs/coco/rtmpose_l/rtmpose-l_420e_coco-288x216_p4p5_sacr.py

AP-10K RTMPose-M

python tools/train.py \
    /path/to/RTMPose-SpatialEnhancement/configs/ap10k/rtmpose-m_210e_ap10k-288x288_p4p5_sacr.py

COCO-WholeBody RTMPose-M

python tools/train.py \
    /path/to/RTMPose-SpatialEnhancement/configs/wholebody/rtmpose-m_420e_coco-wholebody-256x192_p4p5_sacr.py

---

Testing

Use the official MMPose testing script:

python tools/test.py CONFIG_FILE CHECKPOINT_FILE

Example:

python tools/test.py \
    /path/to/RTMPose-SpatialEnhancement/configs/coco/rtmpose_m/rtmpose-m_420e_coco-288x216_p4p5_sacr.py \
    /path/to/checkpoint.pth

Accuracy settings follow the corresponding configuration files.

In particular:

• COCO accuracy configurations use flip_test=False.
• AP-10K and COCO-WholeBody accuracy configurations use flip_test=True.
• All latency benchmarks use flip_test=False.

---

Model Zoo

Dataset	Model	AP	Config	Checkpoint
COCO	RTMPose-S	73.66	Config	Coming soon
COCO	RTMPose-M	75.87	Config	Coming soon
COCO	RTMPose-L	77.04	Config	Coming soon
AP-10K	RTMPose-M	77.32	Config	Coming soon
COCO-WholeBody	RTMPose-M	61.15	Config	Coming soon

Checkpoints are not stored directly in this Git repository.

---

Troubleshooting

Duplicate registry errors

The custom modules in this repository use the registry names:

P4P5AddFusionNeck
ScaleAdaptiveBBoxScale

Do not simultaneously use a locally modified MMPose installation that already contains custom modules with the same registry names.

Otherwise, an error similar to the following may appear:

KeyError: 'ScaleAdaptiveBBoxScale is already registered'

Use a clean official MMPose installation together with this repository.

The custom module cannot be found

Make sure that the repository root is included in PYTHONPATH before launching training or testing.

You can verify the current MMPose import path with:

python -c "import mmpose; print(mmpose.__file__)"

For a clean environment, the path should point to the intended official MMPose installation rather than an older locally modified experiment directory.

---

Release Status

Available now:

• P4/P5 semantic-detail fusion module
• SACR implementation
• COCO RTMPose-S configuration
• COCO RTMPose-M configuration
• COCO RTMPose-L configuration
• AP-10K configuration
• COCO-WholeBody configuration
• Method overview figure

Planned additions:

• Pretrained checkpoints
• Qualitative visualization examples
• Exact environment lockfile

---

Citation

The paper is currently under review.

Citation information will be updated after publication.

---

Acknowledgment

This project is built upon the excellent open-source MMPose and RTMPose projects.

---

Funding

This work was supported by the Jiangsu Provincial Higher Education Association Project of China (Grant No. 2026DGS049) and, in part, by the National Natural Science Foundation of China (Grant No. 62401235).

---

License

This project is released under the Apache License 2.0.