-
SHED Light on Segmentation for Depth Estimation
-
Seung Hyun Lee and Sangwoo Mo and Stella X. Yu
-
Transactions on Machine Learning Research, 2026
-
Paper
|
arXiv
-
Abstract
-
Dense prediction infers per-pixel values from a single image and underlies 3D perception and robotics. Real-world scenes are highly structured, yet standard dense prediction models typically decode spatial outputs without explicitly tying reconstruction to a learned hierarchy of image regions. We propose SHED, an encoder--decoder that constructs an explicit hierarchy of image segments and reverses this hierarchy during decoding. The encoder groups image regions into progressively coarser segment tokens, and the decoder unpools coarse representations back to finer regions through the learned assignments rather than generic feature upsampling. The hierarchy is trained end-to-end from the dense prediction objective alone, without segmentation labels. SHED improves mean monocular-depth accuracy over a matched DPT baseline, with the largest relative gain under HyperSim-to-NYUv2 transfer without target scale or shift alignment. Beyond per-pixel accuracy, boundary, object-wise, and Chamfer evaluations show smaller mean differences in geometric structure. Representations learned under depth supervision also improve alignment with held-out PartNet part annotations. Finally, applying the reverse hierarchy to RAFT reduces end-point error under matched optical-flow training on FlyingChairs.
-
Keywords
-
monocular depth estimation, hierarchical representation learning, segmentation-guided decoding, cross-domain generalization, optical flow
|