Reconstruction Literature Review
Literature review for neural network based reconstruction approach.
Visual Representation
DINO: Emerging Properties in Self-Supervised Vision Transformers
Self-supervised pretraining of a Vision Transformer (ViT) with no labels, producing general-purpose features whose attention maps spontaneously highlight object boundaries and semantics. Used by many later foundation models as a frozen visual backbone.
- Self-distillation: a student network and a teacher network share the same architecture; the student is trained by gradient descent while the teacher’s weights are an exponential moving average (EMA) of the student’s weights
- Multi-crop augmentation: several small “local” crops and a couple of large “global” crops are sampled from the same image; the student is fed all crops while the teacher only sees the global crops, forcing the student to learn local-to-global correspondence
- Both networks output a probability distribution over a set of learned prototypes; training minimizes the cross-entropy between the student’s and the (stop-gradient) teacher’s output distributions for matching crops
- Centering (subtracting a running mean of teacher outputs) and sharpening (a low softmax temperature on the teacher) prevent the trivial collapse where both networks output the same constant embedding for every image
DINOv2 (paper link) scales this recipe up with an automatically curated, deduplicated 142M-image dataset, added patch-level and KoLeo regularization objectives, and much larger models, producing a single frozen backbone whose per-patch features transfer well without fine-tuning. Depth Anything (below) reuses this frozen DINOv2 encoder as its semantic prior.
CLIP: Learning Transferable Visual Models From Natural Language Supervision
Jointly trains an image encoder and a text encoder to align images with their natural-language captions, giving a shared embedding space that supports zero-shot recognition without task-specific labels.
- Collect 400M (image, text) pairs scraped from the web (no manual labeling)
- Encode each image with a CNN/ViT image encoder and each caption with a Transformer text encoder into a shared embedding space
- Train with a contrastive loss over each batch: for the $N$ image-text pairs in a batch, maximize the cosine similarity of the $N$ correct pairings while minimizing it for the $N^2-N$ incorrect pairings
- At inference, zero-shot classification is done by embedding the candidate class names into text prompts (e.g. “a photo of a {class}”), embedding the query image, and picking the class whose text embedding is closest to the image embedding
Task-oriented Model
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
A foundation model for monocular relative depth estimation, built by scaling training data rather than by any new architecture — a DINOv2 encoder + DPT decoder trained with a data engine, not a new network design.
- Train an initial teacher model on ~1.5M labeled images from existing depth datasets
- Use the teacher to auto-label ~62M unlabeled images collected from the web, turning them into pseudo-labeled training data (a data engine, scaling supervision far beyond what’s manually labeled)
- Train the final student model jointly on labeled + pseudo-labeled images. Two tricks make the pseudo-labels useful instead of just imitating the teacher’s errors
- Strong perturbations (color jittering, CutMix, strong blur) are applied to the student’s view of unlabeled images, forcing it to generalize beyond what the teacher already memorized
- An auxiliary feature-alignment loss ties the student’s intermediate features to the frozen DINOv2 encoder’s features, transferring DINOv2’s semantic prior into the depth model and stabilizing training on noisy pseudo-labels
Depth Anything V2 replaces the real-image labels with synthetic, geometrically-precise labeled images (avoiding the label noise of real depth sensors/stereo), keeps the same real-image pseudo-labeling + distillation pipeline for generalization, and predicts affine-invariant depth (scale + shift, not metric), trading exact metric scale for much sharper boundaries.
SAM: Segment Anything
A promptable segmentation foundation model: given an image and a lightweight prompt (a point, a box, or a rough mask), it returns valid segmentation mask(s) for whatever object that prompt refers to, without task-specific retraining.
- Image encoder: a ViT pretrained with a Masked Autoencoder (MAE) objective, run once per image, producing a dense image embedding that is reused across many prompts (expensive, so amortized)
- Prompt encoder: encodes sparse prompts (points/boxes, via positional encodings + learned type embeddings) and dense prompts (masks, via convolutions), lightweight and fast enough to run per-prompt
- Mask decoder: a small transformer decoder that cross-attends prompt tokens against the image embedding (and vice versa) to predict segmentation mask(s), an IoU-confidence score per mask, and — since a single point can refer to multiple valid objects (e.g. a shirt vs. the person wearing it) — outputs 3 candidate masks at once to represent this ambiguity
- Data engine: SAM is bootstrapped on its own predictions in a 3-stage loop (model-assisted manual annotation → semi-automatic annotation of remaining objects → fully automatic mask generation by prompting a dense grid of points), producing the 1.1B-mask SA-1B dataset used for the final training
Geometry Foundation Model
DUSt3R: Geometric 3D Vision Made Easy
Replaces the classic SfM/MVS pipeline (feature matching → RANSAC → pose estimation → triangulation → MVS) for a pair of images with a single regression problem, requiring no known camera intrinsics or poses at all.
- Input: a pair of RGB images $I_1, I_2$. uncalibrated (no known intrinsics, poses, or overlap)
- Output: two pointmaps $X_1, X_2$
- one per input image, each holding a 3D point and a confidence value for every pixel of its image
- both expressed in the same coordinate frame (camera 1’s frame).
- Depth, relative camera pose, and pixel correspondence are all read off this one output post-hoc rather than predicted directly:
- depth is just the $z$-coordinate of $X_i$
- camera 2’s pose is recovered by fitting $X_2$ against $X_1$
- pixel $i \leftrightarrow j$ correspondence is found by nearest neighbors in 3D between $X_1$ and $X_2$
- Architecture: two images are encoded independently by a shared ViT encoder (initialized from CroCo, a cross-view completion pretraining), then processed by two separate ViT decoders that cross-attend to each other at every block, letting each view’s decoding be informed by the other view
- Training: supervised with a confidence-weighted regression loss against ground-truth pointmaps (from any dataset with known depth/pose, including synthetic data), so no manual correspondence labels are needed
- Global alignment for >2 images: input becomes a graph of image pairs (not just one pair); the pairwise network above is run once per graph edge, then a lightweight optimization (not a full bundle adjustment) jointly rescales and rigidly aligns every pairwise pointmap into one output global point cloud + per-image camera pose, by minimizing disagreement between overlapping pairwise predictions
MASt3R: Grounding Image Matching in 3D with MASt3R
Extends DUSt3R with a dedicated matching capability, on the insight that two pixels truly correspond only if they observe the same 3D point.
- Input: same as DUSt3R — a pair of RGB images, no known calibration
- Output: DUSt3R’s two pointmaps (as above), plus a dense per-pixel descriptor map for each image.
- Running fast reciprocal matching on the two descriptor maps then yields the actual deliverable most users want: a set of precise pixel-to-pixel correspondences between the two images
- Keeps DUSt3R’s pointmap regression head unchanged, and adds a second head that regresses a dense local descriptor (feature vector) per pixel
- The descriptor head is trained with an InfoNCE-style contrastive loss using the ground-truth correspondences implied by the pointmaps, making descriptors of truly-corresponding pixels similar and everything else dissimilar — i.e., matching is grounded in 3D consistency rather than 2D appearance alone
- Fast reciprocal matching: instead of exhaustive nearest-neighbor search between every descriptor pair, uses an iterative mutual nearest-neighbor scheme seeded from a coarse subsample, giving accurate matches at a fraction of the cost
- Coarse-to-fine matching: for high-resolution images, first matches at low resolution, then re-runs matching only within corresponding local windows at full resolution, keeping compute tractable while preserving pixel-level precision
VGGT: Visual Geometry Grounded Transformer
A single feed-forward transformer that, given one to hundreds of images, directly predicts camera parameters, depth maps, point maps, and 3D point tracks for all of them in one forward pass — no per-scene test-time optimization or bundle adjustment at all.
- Input: $N \geq 1$ RGB images (a single image, an unordered photo collection, or a video — the first image in the set is treated as the reference/world frame)
- Output, per input image, all predicted directly (not derived post-hoc as in DUSt3R/MASt3R): camera parameters (intrinsics + extrinsics, relative to the first frame), a dense depth map, a dense point map in world coordinates, and — jointly across all $N$ frames — 3D point tracks for query points followed across the whole set
- Tokenization: each image is patchified by a DINOv2-initialized backbone; a camera token and register tokens are appended per frame (a distinct set for the designated first/reference frame vs. all other frames)
- Alternating attention: transformer blocks alternate between frame-wise self-attention (tokens attend only within their own frame, preserving per-image detail) and global self-attention (tokens attend across all frames, fusing multi-view cues and establishing correspondence) — this alternation is repeated across many layers instead of one big cross-view attention
- Task-specific heads read out from the shared backbone: a lightweight head regresses each frame’s camera parameters (intrinsics + extrinsics, with the first frame fixed as the world reference), while DPT heads regress dense depth maps and point maps
- Trained with direct supervision on large curated datasets of real and synthetic images with ground-truth 3D annotations, unlike DUSt3R/MASt3R, VGGT is not limited to pairs — all input views interact jointly, so it scales naturally to many-view reconstruction in a single pass
π³ (Pi3): Permutation-Equivariant Visual Geometry Learning
Observes that DUSt3R/MASt3R/VGGT-style models all designate one input image as a fixed reference frame (everything else is predicted relative to it), which is an arbitrary inductive bias: results become sensitive to which image happens to be picked first and to the input ordering.
- Input: an unordered set of $N$ RGB images — unlike VGGT, no image is designated as first/reference
- Output, per input image, symmetrically (no privileged frame): a local point map (a 3D point + confidence per pixel, expressed in that view’s own local camera frame — like a lifted depth map) and a camera-to-world pose (a 4x4 transform for that view)
- Getting the global point cloud: for each view, transform its local point map into world coordinates by applying that view’s own predicted camera-to-world pose — a direct unprojection, with no further pairwise registration or alignment optimization needed. This works because the poses coming out of the network are already mutually consistent with each other
- Removes the reference frame entirely: the architecture is fully permutation-equivariant (closer in spirit to VGGT’s alternating-attention transformer, but without VGGT’s asymmetric first-frame tokens), so all input views are processed symmetrically and produce the same reconstruction regardless of input order
- Where does the world origin come from, if no view is special? Contrast with VGGT, which architecturally forces the origin to be camera 1: frame 1 gets a distinct set of special tokens from every other frame, and training supervises absolute poses expressed relative to frame 1 (ground-truth poses are re-expressed in camera 1’s frame before computing the loss) — swap which image is fed in first, and the whole predicted frame shifts to match, which is exactly why VGGT’s outputs vary with input ordering.
- Pi3 has no such mechanism: it only ever supervises relative poses between every pair of views ($\hat T_i^{-1}\hat T_j$ vs. ground truth), never tied to a designated frame, so nothing forces the output poses $(\hat T_1,\ldots,\hat T_n)$ to land in any particular camera’s frame.
- Pi3 makes the reconstruction result more robust regardless of the order of input frames
- Result: near-zero variance in reconstruction quality across different input orderings (vs. a noticeable spread for VGGT), together with lower trajectory/depth error and faster inference than both DUSt3R and VGGT
Implicit Scene Representation
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Represents a scene as a continuous implicit function (an MLP) instead of any explicit geometry, and recovers that function purely by optimizing it to reproduce a set of posed input photos.
- Model: $F_\theta(x,d) \rightarrow (c,\sigma)$, mapping a 3D point $x$ and a viewing direction $d$ to an emitted color $c$ and a volume density $\sigma$
- Rendering: to render a pixel, cast a ray $r$ through the scene, sample points $x_1,\ldots,x_K$ along it, and accumulate color with the volume rendering integral $C(r) = \sum_i T_i(1-e^{-\sigma_i\delta_i})c_i$, where $T_i=e^{-\sum_{j<i}\sigma_j\delta_j}$ is accumulated transmittance and $\delta_i$ the sample spacing
- Training: minimize photometric error between rendered $C(r)$ and the observed pixel color, over rays sampled from all training images. Requires known camera intrinsics/poses for every image (typically from SfM, e.g. COLMAP) — NeRF only replaces what comes after pose estimation
- Geometry is only an indirect byproduct: a mesh can be extracted by running Marching Cubes on a level set of $\sigma$, but since training optimizes photometric consistency rather than surface accuracy, extracted meshes are typically noisier than classic MVS + TSDF results
- Positional encoding (mapping $x,d$ to a higher-frequency Fourier basis before feeding the MLP) is what lets a coordinate-based MLP represent high-frequency scene detail, which a raw low-dimensional input cannot
3DGS: 3D Gaussian Splatting for Real-Time Radiance Field Rendering
Keeps NeRF’s goal — optimize a scene representation against posed photos with a photometric loss — but replaces the implicit MLP with an explicit, differentiable set of 3D Gaussian primitives that can be rasterized instead of ray-marched.
- Representation: each Gaussian $g_k$ has a position $\mu_k$, an anisotropic 3D covariance $\Sigma_k$, an opacity $\alpha_k$, and a view-dependent color (spherical harmonics coefficients)
- Rendering: project the 3D Gaussians onto the image plane and alpha-composite the resulting 2D splats in depth order — a direct rasterization pass rather than a per-pixel network evaluation: $C(u,v) = \sum_k c_k\alpha_k\prod_{k’<k}(1-\alpha_{k’})$
- Because rasterization is far cheaper than ray-marching an MLP at every sample, 3DGS trains in minutes and renders novel views in real time, while NeRF’s ray-marching is comparatively slow for both
- Initialization and densification: Gaussians are seeded from the sparse point cloud already produced by SfM (reusing the same upstream pipeline as NeRF), then adaptively split/cloned/pruned during optimization based on view-space gradient magnitude and opacity, to add detail where the photometric loss is still high and remove Gaussians that contribute nothing
- Geometry output is a set of Gaussian centers/point cloud rather than a mesh — turning it into a mesh needs extra surface-extraction processing on top