dt-folk 09-28

27 days ago

drawtalking is made up of two core components: the drawtalking client (dt-client, which can be compiled as either an iPad or macOS binary) and the drawtalking server (dt-server, a node-based web server which the client connects to and uses for speech recognition).

drawtalking graphics, rendering, and object state are handled entirely client-side. this means that in order to have dt to meaningfully integrate its interactive behavior with another system (like folk), this information needs to be serialized and sent over to the server to be then passed on, and vice versa for entities that folk wishes to share with a client.

first: what sort of interface could we even use to share data between dt and folk?

folk has a library which allows it and another system to communicate over a web socket connection. once a WS handshake is completed, you can send statements and set up reactive subscriptions to fire when Statements are added or removed, matching both against individual statements with watch() or against a collection of statements with watchCollected().

if we understand how object tracking information is exposed in the folk environment, we can match dynamically against the statements created when an object is being tracked and pass those to dt. and (reciprocally) keep track of entities being spawned and modified the dt side.

• dt uses a traditional rendering pipeline that caches core object data on initial creation in a global buffer, then subsequently spawns/applies transforms over that core data to render to screen.
• in folk, we have high-level drawing invocations for creating lines, curves, and so on, but each of these is its own draw call. there’s also the matter of applying coordinate transforms in realtime to a.) transform from dt coordinates to folk coordinates, accounting for physical scale and viewport parameters and b.) the matrix transforms based on the dynamic behavior of entities. where do these operations occur? with many dt entities each spawned as their own call, the overhead from all of these operations could prove significant.
• somewhat in opposition to this, enclosing dt entities fully within their own pipeline obfuscates them from conventional end user access (you can no longer directly modify claims made in the way you might for other programs you make in folk). a lot of the ergonomics of a system like folk is the accretive, global nature of statements reacting to one another, so hiding them away, even if for performance reasons, feels counter to how it should be approached. i am interested in seeing if i can inject TRS information and other necessary transforms alongside dt’s raw object information to improve performance while also allowing for consistency in how people would write for it.
• it feels like there is a trade-off here that will need to be negotiated between performance, accessibility, and the overall philosophy underlying these systems in order to having an integration that feels sensible in both contexts.