[Live Web] Midterm

Introducing the Schadenfreude Chat!

There are three rules that are advertised on the page through the buttons:

Change Background: Keys the user’s image and adds a image file to the background.

Drunko Vision: Slows down the frame rate and creates trail images with different hues whenever there is movement detected.

Anime Eyes: Attaches a png of stereotypical anime eyes to the PoseNet eye keypoints of the user.

But there is also a hidden rule based on PoseNet keypoint tracking:

If the PoseNet nose keypoints touch between two or more users, the positions of the webcam images will randomly change on the canvas.

Mind mapping process:

Above is an early illustration of how I imagined the code for this piece. I imagined that the webcam and pose would be part of the same data object that would be transmitted through the peer connection and redrawn onto the peer canvases. The webcam would be drawn onto a p5 image at 30 fps so that I could use the p5 loadPixels() method to manipulate the stream. The the image would be added to a JSON object along with the pose data and that would be transmitted via Websockets to a socket server hosted on Digital Ocean as a Droplet. On the the server side, all peer data––an initiator boolean, the initiator socket ID, the receiver socket ID, the webcam image, and the pose––would be stored in an array of peers. Then any user could send and receive their data to the others while on the client side, that data would be drawn onto the canvas.

Here is one of the versions of the mind map I created to try and conceptualize this:

You’ll notice however in this example above, that I have the webcam stream being transmitted directly through the peer connection. This mind map was based on Shawn’s week 5 example from class. I thought maybe that I should separate the stream and send through the peer connection and send the pose through the socket server in order to optimize performance since the pose data would be much heftier.

Another bottleneck in my process occurred when trying to integrate p5 into the examples that Shawn gave. I was attempting to make the same project using two different frameworks: one for Lisa Jamhoury’s Body Everywhere class that used ngrok to create the peer-to-peer connections between users; and the other for this class that hosted the app on a virtual machine on Digital Ocean.