Overview
Turns uncalibrated monocular video, shot without motion-capture hardware, into editable character animation and camera tracks in Unreal Engine. Body pose and facial expression come from monocular capture; camera trajectory and world coordinates are reconstructed with SLAM and SfM. Both are assembled in one coordinate system as a Level Sequence and rendered. The pipeline covers single-person motion, camera-move cloning and multi-person scenes.
Pipeline structure
The three pipelines build on each other, each solving one more kind of information: camera cloning adds camera motion, and multi-person capture adds each subject’s world position and identity. World-space alignment, foot grounding and orientation correction, Level Sequence assembly and rendering are shared, so new footage only needs its own solving step.
Comparison with the engine’s built-in solution
MetaHuman Animator monocular capture in UE 5.8 returns a single person’s pose in camera space, without camera motion or the character’s position in the scene; multiple subjects must be solved one at a time with no shared reference. This pipeline solves pose, camera and position together in one world coordinate system, preserving the relationships between people and between people and the camera.
Single-person capture: results
The middle column shows the SMPL solve used for verification; the body and facial animation in the engine come from MetaHuman monocular capture, and the camera and world anchor come from TRAM. The top-down inset shows the camera (blue) and character (pink) paths in the same world space.
Single-person capture: solving and assembly
Two branches supply the data. TRAM solves the camera trajectory with DROID-SLAM, estimates focal length with camcalib, and provides the SMPL body and a pelvis world anchor; MetaHuman monocular capture batch-exports body joints and facial curves. Coordinates are converted to UE centimeters, Z-up and left-handed, then foot grounding, root constraints and orientation correction are applied. Per-frame camera jitter is removed by fitting the whole trajectory with cubic control points; their spacing sets the smoothing while keeping the camera move intact. The Level Sequence holds the CineCamera pose and focal-length curves, the body animation track placed by the pelvis anchor, and the facial curve track. Body and Face are separate skeletons, with Torso, Legs and Feet following through Leader Pose. Output is rendered as a PNG sequence with Movie Render Queue and encoded with FFmpeg.
Camera-move cloning
The solved camera motion is stored as editable data, so the same move can be transferred to other scenes and characters. Relative motion in frame can be carried by the camera or by the subject; it is split geometrically into two drivers while the composition stays the same. Because focal length and camera distance are roughly interchangeable in the image, a new focal length can be set and the camera distance solved from it, keeping the subject’s size and position in frame while changing perspective compression and depth of field.
Multi-person capture
Foreground subjects and the background crowd are solved separately and composited. The two-person multi-shot sequence uses COLMAP camera reconstruction with per-shot intrinsics and one shared world coordinate system. PromptHMR solves an SMPL-X track for each person; each subject’s world position is found by intersecting a ray from the camera through the foot point with a RANSAC-fitted ground plane, and a ray through the head gives height in reconstruction units, which sets the metric scale from the character’s real height. Identities are matched across shots by world position, and multi-camera footage is synced by audio onset. Results are checked by the error of the group bounding-box height and centroid relative to frame height. Shots are nested as subsequences in a master sequence, and the final render runs the full sequence so cloth and hair simulation stay continuous across cuts.
References and open-source work
TRAM: Global Trajectory and Motion of 3D Humans from In-the-Wild Videos (ECCV 2024), used for the camera trajectory and world coordinates in the single-person pipeline. PromptHMR: Promptable Human Mesh Recovery (CVPR 2025), used for per-person pose in the multi-person pipeline. SMPL-X: Expressive Body Capture (CVPR 2019), the body model for the multi-person pipeline. Unreal Engine 5.8 MetaHuman Animator: monocular body and face capture, assembly and rendering. Other tools: COLMAP, Python, PyTorch and FFmpeg.
Demonstration video
Technical content adapted from the author’s portfolio and project files.