NeurIPS 2026

The Mask Is Not the Object: Volumetric Supervision for 3D Gaussian Segmentation

Jeonghwan Cho, Minsu Kim, Junyoung Hong, Seon Joo Kim

Yonsei University

Paper soon arXiv soon Code soon Removal-3D soon
TL;DR

3DGS segmentation renders accurate masks but selects an incomplete set of object Gaussians. VODA is a plug-and-play objective whose depth-aware rasterizer supervises every Gaussian in an object's depth cluster directly, and Removal-3D measures whether removing them actually changes the scene in 3D.

Contributions

  1. 1

    Volumetric segmentation

    We formulate the task, showing how an accurate rendered mask can still correspond to a Gaussian set that fails to account for the object in 3D.

  2. 2

    VODA

    A plug-and-play objective that supervises every Gaussian in the selected depth cluster directly, leaving the baseline's representation and inference untouched.

  3. 3

    Removal-3D

    A benchmark and the Removal Depth Error for depth-based removal evaluation, with consistent gains across four baselines.

Motivation

A mask can be right and the selection still wrong. Ask for the “sheep” and you get Gaussians whose rendered mask looks correct. Delete them and the sheep is still there.

For the query sheep, the selected Gaussians render a plausible 2D mask, yet removing them barely changes the rendered depth in the sheep region.
The selection projects to a convincing silhouette, but removing it leaves the depth in the sheep region almost unchanged. The surfaces behind never appear.
green apple

Hover a panel to magnify, all three follow the same point.

Rendered maskbaseline

After removalbaseline

After removal+ VODA

What has to change

The problem

Supervision is scaled by how much a Gaussian contributes to the render, and that contribution collapses behind the first opaque surface. The interior of an object is never asked to belong to it, so the mask comes out right and the set comes out hollow.

Our take

Choosing the dominant contributor per pixel, or its top-k, still answers which Gaussian explains a pixel. The question worth asking is which Gaussians make up the object, and it belongs along the ray rather than on the image plane.

Key idea

Volumetric supervision. Split each ray where the depth jumps, keep the cluster the object actually occupies, and supervise every Gaussian in it directly, whatever it contributes to the image. Occluded Gaussians inherit the target of the visible ones they sit behind.

VODA turns this into two pieces that attach to an existing baseline: a depth-aware rasterizer and a direct per-Gaussian loss.

Method

Feature gradients are scaled by rendering weight wi = Ti αi, so anything behind the first opaque surface learns almost nothing. VODA splits each ray at depth gaps above an adaptive τ, keeps the cluster with the most contribution, and supervises every Gaussian in it directly.

Selection along a single ray under three strategies: contribution-weighted rendering, top-k contributors, and depth-cluster selection.
Three ways to pick Gaussians on one ray. Contribution-weighted rendering and top-k selection both stop at the visible surface; the depth cluster keeps going. On the right, transmittance collapses while the object's own Gaussians continue past it.

Interactive walkthrough

Depth clustering along a single ray, and the cluster viewer for a real scene.

Coming soon

What one pixel reaches

Hover either render to magnify. The probed view on the left stays as it is.

The rendered scene with a ring marking the probed pixel.

Probed pixelx 447, y 196

55 Gaussiansbaseline

497 Gaussians+ VODA

For that single pixel the baseline supervises 55 Gaussians, the thin magenta shell at the front surface. VODA supervises 497, keeping 54 of the same 55 and adding the purple mass behind them.

VODA pipeline: a depth-aware rasterizer clusters per-pixel contributors, and the rendered feature is applied as a direct per-Gaussian target.
The baseline loss keeps running and anchors the targets; VODA adds the depth-aware rasterizer and the direct per-Gaussian term. Inference is untouched.

Results

VODA is attached to two self-supervised baselines (OpenGaussian, LaGa) and two supervised ones (Gaussian Grouping, ILGS), and measured on two kinds of benchmark.

Standard benchmarks

They score a rendered mask or a visible surface label. A selection can do well here and still be hollow, because nothing behind the first visible surface is ever looked at.

3D object selection from a text query, mean over four scenes
Self-supervised Supervised
MethodmIoU ↑mAcc@0.25 ↑ MethodmIoU ↑mAcc@0.25 ↑
OpenGaussian37.1256.65Gaussian Grouping22.7546.43
+ VODA42.9857.84+ VODA43.7376.27
LaGa56.8076.33ILGS30.0050.00
+ VODA60.2882.56+ VODA49.6281.48

Qualitative

Each panel renders only the Gaussians the method selected. A set that misses the interior renders the object thin and smeared.

“Pikachu”

Hover a panel to magnify, all three follow the same point.

The queried object as it appears in the reconstruction.
Ground truthreference
The object rendered from the Gaussians the baseline selected.
Selected setOpenGaussian
The object rendered from the Gaussians VODA selected.
Selected set+ VODA
3D semantic segmentation from a text query, point-cloud metrics
Method 19 classes 15 classes 10 classes
mIoU ↑mAcc ↑ mIoU ↑mAcc ↑ mIoU ↑mAcc ↑
OpenGaussian26.2039.3528.0042.6734.5748.38
+ VODA29.0042.8029.4344.3836.8950.69
LaGa32.2047.3537.1953.8843.0661.31
+ VODA33.2049.6337.2155.5945.5365.24

Qualitative

Points the method assigned to the queried class, against the ground truth. The failure to look for is leakage into the neighbours.

“Bed”
baseline

Hover a panel to magnify, all four follow the same point.

The reconstructed scene mesh.
Reference meshscene
Ground-truth points for the queried class.
Ground truthlabels
Points the baseline assigned to the queried class.
PredictedOpenGaussian
Points VODA assigned to the queried class.
Predicted+ VODA

Volumetric benchmark

Removal-3D scores the hole, not the silhouette. A synthetic indoor benchmark of three scenes with paired depth maps, before and after the ground-truth object is removed.

The scenes, and what RDE measures

A tour of the three scenes, and the depth-map pair the Removal Depth Error is read from.

Coming soon

mRDE is the removal depth error; mIoU is reported alongside it to catch selection outside the target region. Mean over the three scenes.
Self-supervised Supervised
MethodmIoU ↑mRDE ↓ MethodmIoU ↑mRDE ↓
OpenGaussian52.8739.31Gaussian Grouping26.2350.05
+ VODA60.9027.34+ VODA30.1838.12
LaGa75.9336.93ILGS31.6649.98
+ VODA79.0721.66+ VODA38.7535.65

Qualitative

Each card wipes from the selected mask to the scene once those Gaussians are removed. What a method leaves behind shows up as a smear where the object was.

hamburger
drag to wipe both at once
Segmented Removed

Mask to removalbaseline

Segmented Removed

Mask to removal+ VODA

Analysis

Two questions decide whether the diagnosis holds: which Gaussians VODA actually recovers, and whether a simpler fix would have done the same job.

Two things have to hold at once. VODA has to be picking out genuinely low-contribution Gaussians, the ones rendered supervision never reaches, rather than just selecting more of what was already visible. And it has to do that while accuracy goes up, not by trading precision for recall. The first is testable because the two checkpoints share identical geometry and opacity, so every Gaussian's maximum contribution is fixed before either method runs; the second is why mIoU is reported beside mRDE everywhere above.

Cumulative distribution of maximum rendering contribution for baseline-selected Gaussians versus those recovered only by VODA, on three Removal-3D scenes.
The recovered set sits consistently to the left. At the rasterization cutoff w = 1/255, 17 to 36% of it falls below the threshold, 10 to 16 points more than among the baseline-selected set. Those Gaussians also appear in roughly 6× fewer pixels and 3× fewer views, and 18.7% receive zero gradient in every view. They are reached without giving anything up: on Removal-3D every baseline gains mIoU in the same run that lowers mRDE.

The question underneath is whether this problem is hard at all. The obvious objection is that a low-contribution Gaussian simply needs a bigger gradient. If that were true, VODA would be unnecessary: rescaling the rendered-feature loss that already exists would recover the same Gaussians. So we tried three such fixes, and report every metric each one moves.

Select a variant to read what it changes

OpenGaussian

    Gaussian Grouping

      Applications

      Editing is where an incomplete set shows. Every edit acts on the selected Gaussians directly: paint a partial selection red and the original green survives underneath, lift it and a ghost of the apple stays on the table.

      Editedbaseline

      Edited+ VODA

      BibTeX

      Coming soon