3D reconstruction of anonymous markers with unknown mutual distances

1. The problem

Papers 1–3 calibrate the cameras with a rigid object whose marker distances are known. On the athlete this is not the case: markers are attached to skin or clothes, distances change with motion and are not known in advance. Each camera sees anonymous blobs; some markers are hidden by the body, two nearby markers can merge into one blob, reflections add blobs that correspond to nothing. The task is to find the 3D positions of the markers using only the calibration, with no body model.

The difficulty is combinatorial: with $C$ cameras and $n$ blobs per camera there are $n^C$ combinations of correspondences. Two rays from different cameras also come close by chance: an intersection of only two rays can be a ghost point. Epipolar geometry reduces the candidates, the consensus of several cameras confirms them.

2. Simulated scene

A person walks (1.4 m/s) or runs (3 m/s) through the 3.5×3.5×2 m volume. The body is made of capsules (trunk, head, upper arms, forearms, thighs, shanks, feet) moving with a gait cycle; $N$ spherical 14 mm markers are spread at random over the capsules, fixed to the segment. For each camera: a marker is visible if the ray from the camera does not cross any capsule; two markers whose image distance is smaller than the sum of their apparent radii produce a single blob at their midpoint; each image receives 0.3 reflections on average; centroids have 0.15 px Gaussian noise. The cameras are the 8 of Paper 2 (two height rows, Full HD, 120° lenses). Each configuration is simulated for 40 frames at 60 fps.

3. Algorithm

For two cameras $i$, $j$ with relative rotation $R$ and translation $\mathbf{t}$, a 3D point seen in normalised coordinates ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ (undistorted) satisfies the epipolar constraint

The distance of a blob from the epipolar line $\mathbf{l}=E{\tilde{\mathbf{x}}}_i$ is $|{\tilde{\mathbf{x}}}_j^{\top }\mathbf{l}|/\sqrt{l_1^2+l_2^2}$; multiplied by the focal length it is in pixels and is compared with the tolerance $\epsilon$.

The cost is dominated by the candidates: $O(C^2{n}^2)$ epipolar tests and $O(KC)$ projections for the support, with $K$ candidates. The "one blob, one point" rule is what removes ghosts: a chance intersection of two rays rarely finds support in a third camera, and if its blobs are already explained by a point with more support it is discarded.

CommentThe epipolar line crosses the body and passes near several blobs: in this frame 243 candidates in total for 24 accepted markers. Only the right candidate is confirmed by other cameras; the others are chance intersections that find no support. This is why a minimum support of 3 removes ghosts.

4. Results

The metrics are: recall (share of markers seen by at least 2 cameras reconstructed within 1 cm), precision (share of reconstructed points that correspond to a true marker: 1 − ghosts), RMS error of the right points and time per frame (Python + NumPy, one core).

CommentWith 4 cameras support 2 finds more markers (86%) but with 13% ghosts; support 3 has no ghosts but finds only 33% of the markers. Tracking recovers part of the markers seen by only 2 cameras when a track predicts them (48%). From 8 cameras on the three variants converge: recall 97.0%, precision 99.9%. The error decreases with the number of cameras (1.34 → 0.85 mm) because each point is triangulated with more rays.

CommentThe tolerance has a wide range where everything works (0.5–3 px); at 5 px ghosts appear (precision 92%) and the error grows because wrong views are accepted. ε = 1–2 px, i.e. about 5–10 times the centroid noise plus the projected calibration error. The 3D error follows the calibration: with 0.02° of rotation 1.31 mm, with 0.05° 2.08 mm, with 0.1° 3.63 mm. Centroid noise weighs the same way: from 0.54 mm at 0.02 px to 3.82 mm at 0.6 px. Without a calibration like that of Papers 1–3, the reconstruction error would be dominated by the calibration.

CommentMore markers mean more nearby blobs and more chance intersections: from 30 to 80 markers precision drops from 99.9% to 96.6% and the error rises from 1.09 to 1.49 mm; time grows with the square of the blobs per camera (56 → 212 ms). Bigger markers merge more often: with 30 mm precision drops to 94.5% and the error doubles (1.09 → 2.09 mm), because merged blobs produce points halfway between two markers. On the body the smallest marker the resolution allows is best, as for calibration (Paper 1).

Resolution reduces the error in proportion to the pixels (the noise in pixels is fixed): 0.61 mm in 4K against 1.67 mm in HD. In running recall drops a few points because constant-velocity tracking struggles with the feet (abrupt velocity changes) and occlusions are shorter but more frequent; with 10 cameras it recovers only in part. Reflections, even 3 per image, do no harm: they find no support.

5. How to implement it

The Python reference implementation takes ~59 ms per frame with 8 cameras and 30 markers: fine for offline analysis, not for real time at 120 fps (8.3 ms per frame). The plan for real time:

From the calibration we need intrinsics, distortion and extrinsics of each camera and, if cameras have no hardware trigger, the estimated delay (preliminary study). During use, two indicators show whether the calibration has degraded (a camera knocked, thermal expansion): the mean residual of the accepted points per camera and the share of each camera's blobs that do not enter any point. If a camera exceeds the thresholds, only that camera is recalibrated with the T.

6. Conclusions

Limits: capsule body, randomly placed markers, no skin-sliding model; the next step is assigning the 3D points to a body model (3D labelling), not treated here.