over the moon

1 day ago

if the work of Seymour Papert has taught us anything, it’s that programming is actually about thinking. the value of computer science—beyond its intrinsic qualities or pragmatic outputs—lies in learning how to understand and shape the world around us through structured systems of representation: modeling, evaluation, debugging, and iteration. the essence of programming lies in cultivating these key metacognitive skills. but is programming, as it has traditionally been understood, the only way to cultivate programmatic thinking?

drawing and written language are rich representations of the world that many people are familiar with; what’s more, we already have a deep, nuanced, and implicit understanding of the physical world around us, whose behavior we constantly model and interact with every day.

to me, the promise of integrating folk and drawtalking has always lied in the promise of a computing environment that bridges programming and the richness of other ways of making sense of the world: where people can program by rearranging teacups, drawing pictures of their dog, or telling stories about the moon.

putting the pieces together

over the past few weeks, Omar has continued exploring different strategies for robust, general-purpose object tracking in folk. with DINOv2, we now have a fairly stable, performant, and embedding-based representation of each object, whose claimed id and position we can find by looking for statements where an /obj/ has center /center/ in /viewport/.

by watching for these statements on drawtalking’s server, i’m able to collect all active objects being tracked by folk and send their id and position in realtime to the drawtalking client (e.g. via a message like exU:object:upd:${id}:${x},${y}\r\n).

it’s important to note that folk expresses object positions in physical units, rather than pixels. drawtalking passes a physical scale factor alongside all of its drawing data, which i apply to match the actual drawn scale of dt entities in screen-space with their projected counterparts. we also use this same scale factor to correctly map the center position of real objects tracked by folk back into pixel space.

each drawtalking entity is made up of a range of associated information, from the strokes that make it up to the properties that are applied or updated in realtime, like its color (applied per stroke), its collision quad, and its transformation matrix. drawtalking’s client currently aggregates all of this information per entity and shares it, as fast as possible, with the server. my server-side code sends this information to folk as discrete Hold!s, keyed to the id of each drawtalking entity. this allows for persistence for an object as its state changes over time:

folk.hold(
  'dt' + event.id,
  tcl`
    Claim ${folkVp} has dtObject with \
    id ${event.id} \
    strokes ${strokes} \
    colors ${colors} \
    transform ${transform} \
    quad ${quad} \
    metersScaleFactor {${dtMetersScaleFactorX} ${dtMetersScaleFactorY}}
  `,
  'folk-comm.js',
);

here we are claiming that all dtObjects are owned by a single discrete viewport in folk, folkVp, which we identify upon initialization. i’ve built the viewport program so that when it has a dtObject, it parses the information related to that entity, applies its associated transforms, scales it to the correct physical scale, and renders it as a series of drawn lines.

learning through play

with a basic bidirectional understanding of physical objects and drawtalking entities between both systems, we were finally able to start experimenting with interactions between them. in the example below, our tracking and rendering capabilities are used to have the drawtalking representation of a teacup mirror the physical object’s actual position:

there’s a unique sensation that comes with seeing physical and digital entities interact this way. it feels particularly pronounced in scenarios where objects and screens closely overlap, which we can observe here as Karl moves our tracked teacup over a tablet running drawtalking:

it also feels pronounced when object and screens are split. here, you can see an example of Omar moving the teacup and having that drive the movement of a drawtalking entity on my screen:

the biggest challenge in getting these demos to work had less to do with performance and more to do with stability; performance between dt and folk over a websocket connection is actually surprisingly decent, even at this early stage of implementation. random socket disconnects, inconsistent tracking based on shifting lighting conditions, and crashes on the drawtalking server were among the issues we encountered getting everything to work. improvements in stability, performance, and scalability are already things i’m working on in a more permanent version of the data pipeline between the two systems.

let’s take a look at some more examples. here, we see a dog moving around to follow the position of our physical cup, the way a dog might follow a ball:

drawtalking has a number of other interesting verbs that allow us to model other types of dynamic behavior, like revolve. think about planets: the way the earth revolves around the sun, or the moon revolves around the earth. what if our cup wasn’t a cup, but a planet instead? and what if that planet had a little moon of its own?

these behaviors and relationships can be composed to express even more complex interactions. our cup is our planet, and our planet has a moon. what if we added a cow, and—as in the old nursery rhyme—we had a cow that jumped over the moon?