Advanced SLAM Literature Review

Posted on In ,

Literature review for SLAM papers that use neural network / deep learning based approaches.

3D Reconstruction

SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos

paper link

Proposed deep-learning based method to perform 3D reconstruction from video sequence without estimating frame poses and camera parameters

  • Middle frame from the clip will be treated as keyframe
  • I2P (Image to Points) model: output 3D points from a video clip (a sliding window of original video)
    • multi-branch ViT (Visual Transformer)
    • Input: frames from a clip with 1 frame as keyframe
    • Output: features of each frame, 3D points reconstructed and their confidence
    • Model structure
      • Encoder: encode each frame from the clip to features
      • Keyframe Decoder: self-attention for keyframe features, cross-attention for keyframe features v.s. all supporting frames’ features
      • Supporting Decoder: cross-attention for each supporting frame’s features v.s keyframe features
  • Buffering set
    1. Hold up to B registered frames / scene frames for global reconstruction
    2. When a new keyframe after I2P, it may be treated as a scene frame. If chosen as scene frame, it will be inserted into buffering set and replace one of the current frame
    3. When a new scene keyframe comes, its features will be fed into a retrieval model to compute correlation score v.s. each of other scene frame in the buffering set. The top-K best-correlated keyframes will be selected as global reference to fuse the current scene keyframe
  • L2W (Local to World) model: incrementally register local point clouds to the global point cloud
    • Input: K scene frames and a new keyframes with their features and 3D points from I2P
    • Output: new (transformed) 3D points in global reference frame
    • Model structure
      1. Points embedding: encode 3D points from each scene frame into (geometric) features. Combine with visual features from I2P for each scene frame
      2. Registration decoder: self-attention for keyframe features (visual + geometric), cross-attention for keyframe features v.s. all scene features
      3. Scene decoder: cross-attention for each scene features v.s. keyframe features