improving calibration

Updated 19 days ago

an issue i’ve run into somewhat frequently when trying to calibrate folk is the mismatch between the physical area covered by the projector and what the camera can see.

the calibration process involves fitting a set of projected AprilTags with their printed counterparts, where the array of projected tags fill the gaps between printed tags (and vice-versa).

a mismatch between the projector and camera can become a challenge because printed and projected tags must first be manually aligned to start the pose registration process. while folk does provide a simple slider control to adjust projected tag scale, it scales them relative to the top-left of the projector view, leading to situations where the projected tags fit the printed tags but are outside of the view of the camera.

improving the ux of this experience felt like a good way to familiarize myself with some of folk’s inner workings and get to grips with writing TCL. there was (and is) very much a learning curve here, especially coming from writing Rust in an embedded context. something at this size and scope would also be a good exercise in understanding the folk’s existing code and in crafting a specific, focused, relatively uncontentious PR.

my initial thought was to offer an additional set of sliders to control the X and Y position of the projected tag set. the calibration interface already has several sliders to allow the user to set things like camera exposure and tag scale, so reusing the pattern seemed like a natural fit.

but assuming that the center of the projection is more or less guaranteed to be in view of the camera in some form, we can avoid the complexity of added controls by simply centering the tags in the middle of the projected view. i settled on this approach instead.

understanding the code

model.folk provides a set of methods for instantiating and modifying sets of calibration tags, which it Claims as a library for other programs to access under modelLib. calibrate.folk uses these to create a unitless calibration pattern, unitModel.

we can then specify the actual physical size in meters for our projected tags, based on the printed tags that serve as our ground truth:

When camera /camera/ has width /cameraWidth/ height /cameraHeight/ &  
     display /display/ has width /displayWidth/ height /displayHeight/ &   
     the AprilTag detector maker is /makeAprilTagDetector/ &  
     the jpeg library is /jpegLib/ &  
     the calibration model library is /modelLib/ &  
     the calibration matrix library is /matLib/ &  
     /someone/ wishes to calibrate camera /camera/ to display /display/   
         using measurements /measurements/ {

...

set printedSideLengthMm [string trimright $measurements(tagSideLength) mm]

...

set printedSideLengthM [/ $printedSideLengthMm 1000.0]
set baseModel [$modelLib scaleModel [$modelLib unitModel]  $printedSideLengthM]

we then compute a homography to map this set of tags into pixels on the projector plane:

Hold! -key H_modelToDisplay {  
    # Default model and version: nothing rotated.  
    Claim the calibration model-to-display homography is $H_modelToDisplay with   
        model $baseModel version -1   
        frameTimestamp [expr {[clock milliseconds] / 1000.0}]   
        modelTimestamp [expr {[clock milliseconds] / 1000.0}]  
}  

this homography is computed using our mapping, H_modelToDisplay.

set H_modelToDisplay [$matLib estimateHomography [subst {
    {$printedSideLengthM $printedSideLengthM $tagSideLengthPixels $tagSideLengthPixels}
    {$printedSideLengthM 0 $tagSideLengthPixels 0}
    {0 $printedSideLengthM 0 $tagSideLengthPixels}
    {0 0 0 0}
}]]

by adding offsets to this mapping—

set H_modelToDisplay [$matLib estimateHomography [subst {
    {$printedSideLengthM $printedSideLengthM [expr {$tagSideLengthPixels + $xOffset}] [expr {$tagSideLengthPixels + $yOffset}]}
    {$printedSideLengthM 0 [expr {$tagSideLengthPixels + $xOffset}] $yOffset}
    {0 $printedSideLengthM $xOffset [expr {$tagSideLengthPixels + $yOffset}]}
    {0 0 $xOffset $yOffset}
}]]

we can then center our tag arrangement given the appropriate calculations (top-left coordinate system):

set projWidthPixels [expr {$COLS * ($tagSideLengthPixels * 10/6 + $pad)}]
set projWidthPixels [expr {$projWidthPixels * 1.1}]
set projHeightPixels [expr {$ROWS * ($tagSideLengthPixels * 10/6 + $pad)}]
set projHeightPixels [expr {$projHeightPixels * 1.1}]
set xOffset [expr {round(($displayWidth - $projWidthPixels) / 2)}]
set yOffset [expr {round(($displayHeight - $projHeightPixels) / 2)}]

there’s also scaling factors appended to the calibration pattern geometry to offset the labels dynamically appended to all four corners of the tag pattern. there isn’t a way to account for their size cleanly prior to setting the size scale of the pattern itself, so these were set mainly to ensure that the tags would all show up uncropped, even at the largest scale.

ultimately, what i found so interesting about this exercise was that the bulk of the work was not really writing code or thinking through an algorithm, but instead understanding what was already written with a reasonable amount of rigor. once it was clear how the projected calibration pattern was being generated, adjusting it was relatively straightforward.