bridging drawtalking and folk

5 days ago

drawtalking is made up of two core components: the drawtalking client (dt-client, which can be compiled as either an iPad or macOS binary) and the drawtalking server (dt-server, a node-based web server which the client connects to and uses for speech recognition).

drawtalking graphics, rendering, and object state are handled entirely client-side. this means that in order to have dt to meaningfully integrate its interactive behavior with another system (like folk), this information needs to be serialized and sent over to the server to be then passed on.

first: what sort of interface could we even use to share data between dt and folk?

folk has a library which allows it and another system to communicate over a web socket connection. once a WS handshake is completed, you can send statements and set up reactive subscriptions to fire when Statements are added or removed, matching both against individual statements with watch() or against a collection of statements with watchCollected().

if we understand how object tracking information is exposed in the folk environment, we can match dynamically when an object is being tracked to pass to dt. and (reciprocally) keep track of entities being spawned and modified the dt side.

• dt uses a traditional rendering pipeline that caches core object data on initial creation in a global buffer, then subsequently spawns/applies transforms over that core data to render to screen. in folk, we have high-level drawing invocations for creating lines, curves, and so on, but each of these is its own draw call. there’s also the matter of applying coordinate transforms in realtime to a.) transform from dt coordinates to folk coordinates, accounting for panning and potentially zoom and b.) the matrix transforms based on the dynamic behavior of entities. where do these operations occur? with many dt entities, the overhead from all of these operations could prove significant.
• somewhat in opposition to this, enclosing dt entities within their own pipeline obfuscates them from conventional end user access (you can no longer directly modify claims made in the way you might for other programs you make in folk). a lot of the ergonomics of a system like folk is the accretive, global nature of statements reacting to one another, so hiding them away, even if for performance reasons, is less ideal. i am interested in seeing if i can inject TRS information and other necessary transforms alongside dt’s raw object information to improve performance while also allowing for consistency in how people would write for it.
• in this sense, folk’s flexibility can also feel like its weakness: being able to bolt on arbitrary systems and code is undoubtedly a useful ability (and in this case, essential). but that also means that understanding certain systems can, at times, require heterogenous, somewhat idiosyncratic knowledge that generalizes poorly outside of that one specific domain.