Thesis Week 3: thesis proposal, unity ml-agents, system theory

After submitting my thesis proposal on Friday, my focus on the weekend was on exploring the functionality of ml-agents in unity and reading on system theory, namely the debate between Niclas Luhmann and Juergen Habermas.

I did a few basic tutorials on training agents using reinforcement learning with the unity game-engine with a Tensorflow backend with tensorflow sharp. Here the agent learns to roll a ball towards a cube:

training

 

inference

To learn more complex actions with no immediate reward, researchers at Berkeley recently developed a curiosity mechanism for reinforcement learning that motivates the agent to explore and experiment in its environment. Here it needs to pull a switch (red light turns green) first (no external reward), then a pyramid with a yellow cube on top appears somewhere (still no external reward) and if knocks down the pyramid and the yellow cube on top is touched by the agent, an external reward is added. To motivate the interest of the agent in exploring all the steps above before any external reward is added, the model works with an internal reward mechanism running two neural networks against each other: it compares predicted action and predicted observation with the real observation and action and gives out a reward the higher the difference between expectation and reality is. Meaning the bigger the surprise, the bigger the internal reward. Unity incorporated this into their ml-agents recently to model more complex behaviors - which could help me for my simulation of religion and rituals. I trained the example model for one hour, here the result:

inference

To test how well the trained model is able to abstract objects I changed the environment: instead of a golden cube, the agent has to knock a golden sphere of the pyramid to get an external reward. As this scenario varies slightly from training the agent has difficulties to deal with this new environment:

inference (untrained change in environment)






Thesis Proposal: Eden

statement

I propose an interactive sculpture that connects the audience in a gallery setting with a simulation of religion that is invented by neural networks.

I am fascinated by complex systems and the way individuals dynamically organize themselves around the basic human needs for hope and certainty. Therefore my main artistic focus of my thesis will be on emergent behaviors of systems, namely on how the creation of rituals influences the activity of machine learning agents in their environment. In this simulation, the agents are rewarded for inventing new rituals with the elements of their world. The role of the audience will be to temporarily add true randomness to this complex system - and therefore create non-algorithmic chaos. This uncertainty will have consequences on the agents' rituals. The audience is provoked to think about the paradox of a system that seems to be only partially rule-based.

Living in a world we try to make sense of but will never fully understand is the key to human existence. Establishing a sensitivity for our own hopes and fears when faced with the complexity of the world and ultimately create a more intuitive understanding of it is the main goal of my piece.


research

The theoretical part of my piece is influenced by the works of system theorists/philosophers/sociologists Theodor Adorno, Max Horkheimer, Niclas Luhmann, Juergen Habermas and Jacques Derrida, mathematical approaches in modeling complex systems and current research in the field of neural networks, especially deep reinforcement learning. My artistic approach is rooted in the Fluxus-movement, namely Joseph Beuys' performative focus on art as a collective form of healing. 

While iterating with reinforcement-learning examples in Unity/Tensorflow and rendering possible versions of the final piece in c4d, I will do further theoretical research on Derrida and his view on the relationship between society and religion, more reading on the communication models of Luhmann and Habermas and gain a better understanding of chaos-theory (Lorenz attractor).

I will to talk to Gene Kogan about the main technical setup for the project, contact Ben Light to go through my fabrication plans and schedule an office hour with a faculty member of NYU Department of Computer Science with a background in deep reinforcement learning. For interaction design considerations I will consult Katherine Dillon and artistic advice Danny Rozin. 

bibliography:

Adorno, T. W., & Horkheimer, M. (2016). Dialectic of enlightenment. London: Verso.
Baring, E., & Gordon, P. E. (2015). The trace of God: Derrida and religion. New York: Fordham University Press
Ha, D., and Schmidhuber, J. (2018). World models. arXiv preprint arXiv:1803.10122.
Hui, Y., & Stiegler, B. (2016). On the existence of digital objects. Minneapolis: University of Minnesota Press
Londei, A. (2014) Artificial neural networks and complexity: an overview. Rome: Archeologia e Calcolatori Supplemento 6
Luhmann, N. (2013). Theory of society (Vol. 2). Stanford Univ. Press
Smith, L. A. (2007). Chaos: A very short introduction. Oxford: Oxford University Press

So far in my research I gained an overview over applicable theories, thinking frameworks and tools. Most of it in the field of system theory, complex systems and devotional practices/artifacts in various cultures and times. 

In the next weeks I will go deeper into the mathematical side of complex systems, the relationship between religion and sociology, the nature of rituals and reinforcement learning as a modeling tool for sociological behaviors.

implementation

My academic background is in the fields of sociology, politics and literature. I am familiar with the theoretical frameworks that are relevant for my thesis, especially the sociological and philosophical part. My private interest is in french philosophy of the 20th century with a focus on Existentialism. I have been creating artworks using machine learning / neural networks since three years after taking Gene Kogan's machine learning for artists course in Berlin (1 month intensive course). I recently took a udacity class for a deeper dive into the maths of machine learning and am now proficient enough to build my own basic ml-architectures in Pytorch from scratch. I have used the higher level interfaces of Tensorflow and am experienced with training and deploying GANs and CNNs on various remote server platforms. Last summer I did a 3 month internship at Havas New York as AI researcher in the Creative Department and built multiple prototypes for a client. 
I explored in depth the element of true randomness in Project Development class with Danny Rozin and built a meditative sculpture that lets the audience engage with a rock (via measuring its truly random decay of subatomic particles with a geiger counter and mapping that to the audience actions) and true randomness. 

I will use deep reinforcement learning for my setup. As this is different to the works I have done so far in the field of ML (most of it was language or image based and generative), I will focus on getting a good understanding of the algorithms underlying the Unity-ml agents and then decide which framework is the best for my piece: stick to unity, or building a pure python backend or even entirely browser based. The reward function for sociological behavior has to be developed from scratch and I have to decide if I want a physical entity (robot) in the piece. The latter depends on how much time I have left after settling on the ml-framework. 
The form of interaction of the audience with the sculptural piece is technically set, I can rely on my knowledge from my last piece using geiger-counters to get true random numbers.

I envision the piece to consist of 5 random number generators that are embedded in stone-sculptures. The agent-simulation will be projected on the floor around these rocks with projection mapping techniques. The audience is encouraged to engage with the sculptures through a small hammer. Knocking the hammer against the rocks will trigger and feed random numbers into the simulation and confuse the learning process of the agents. They will then modify their observable behavior and approaches to come up with new robotic rituals - the projection on the floor will change accordingly. 

So far my draft schedule will look like this:

  • February - 2 weeks of theoretical research and first prototypes with unity-ml while getting on top of deep q-learning in Pytorch. Rough draft of exhibition setup/fabrication ideas, early user testing

  • March - 2 weeks narrowing down final version of ml-side and finalizing choices on fabrication/setup, deeper user testing. 2 weeks of building network and fabrication

  • May - debugging, final technical touches, setup

Thesis Week 2: Iteration, Experiments

After a lot of tinkering I finally found a working architecture for my image classifier (still just on flowers, should be fluxus performances frames soon) using densenet and a lightweight one layer as new classifier:

Screen Shot 2019-02-07 at 1.51.34 PM.png

The accuracy on a test -dataset is now around 75%:

Screen Shot 2019-02-07 at 1.45.51 PM.png

While doing this I iterated a lot on traditional catholic holy water fonts, as I wanted to look at spiritual rituals/actions that have been used in traditional religions (I used Catholicism as I grew up with it) and try to rethink their design. I wanted to incorporate the hand movement towards an object that is in the focus of attention (here either cubes or a golden sphere:

ring_cubes2.png
glass_ring_1.png

Some experiments did not work that well and just look strange:

weird_yucca.png

Others had something but need a lot more iteration:

ring_marble1.png
rubber_ring1_0001.png

As I mentioned earlier in my blogposts that I somehow want to use honey in the installation (as a symbol of healing) I started experimenting with the real flow plugin for c4d to get photorealistic renderings of liquids. Unfortunately the renderfarm I used for my usual renderings doesn’t support the plugin and I have to render locally on my mac (which is very very slow - one frame takes a day to render). Anyway, here a little test for honey simulation:

golde_honey_sphere.png

I thought about a design object that consists of a digital and physical layer, combining them in an optical illusion: the golden sphere gets inserted in the object, then gets displayed on the screen, seemingly “floating” through the honey. Ideally the user has the feeling this happens to the real sphere:

IMG_0259.png

Thesis Week 1: Research, Concepts, Ideas

week 1

I decided to write a little bit about my process regarding thesis every day and put everything together in a weekly post.

day 1/night 1

After yesterdays session with our advisor Kat Sullivan, it’s time to go through things for thesis today: A little bit of concept, a little bit of research and a bit of planning for this week. Not too detailed as my concept depends heavily on prototyping, so I will have to come up with milestones after each week - and decide on how to continue.

research question

How can art and technology enrich our spiritual search?

type

“Even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence at the place where it happens to be.” 

(Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction)

real time, real space art performance

concept

create a spiritual performance for robots around a central artistic AI: robots and AI try to save human mankind with art

(c4d render based on  py-c4d tutorial )

(c4d render based on py-c4d tutorial)



ethics

based on Joseph Beuys and greek mythology my thesis tries to “heal” - honey is used as a metaphorical tool for this



(c4d render based on  py-c4d tutorial )

(c4d render based on py-c4d tutorial)

execution

  • stage 1: create a viral AI that helps humans to be artistic every day

  • stage 2: destroy any digital code of the virus, only encrypted code on clay tablets survives

  • stage 3: create a physical sculpture as a devotional object made out of tablets and AI

  • stage 4: create a ritual for commercial mini-robots (anki, alexa, …) where they try to decrypt the tablets (e.g. do fluxus performance around the sculpture) to save human mankind




first steps

  • this week is devoted to choosing and building the right network architecture, training data (performance art in video form, mostly fluxus) and a first trained network as a draft (probably image based, frame-by-frame object detection / transfer learning + image net as a starter)


    Here the very basic and transfer-learning network structure I am experimenting with (based on the Udacity AI Programming with Python online class I took last year):

classifier = nn.Sequential(OrderedDict([
                            ('fc1', nn.Linear(25088, 200)),
                            ('relu1', nn.ReLU()),
                            ('dropout1', nn.Dropout(p=0.5)),
                            ('fc2', nn.Linear(200, 120)),
                            ('relu2', nn.ReLU()),
                            ('dropout2', nn.Dropout(p=0.5)),
                            ('fc3', nn.Linear(120, 102)),
                            ('output', nn.LogSoftmax(dim=1))
                            ]))



I am currently hosting everything on Google Colab for quick prototyping, back to watching the loss going down:

While doing this, playing with particles in c4d offers superb distractions:

rnder0075.png
  • if there is additional time I will try to research more on computer viruses / worms



day 2 / night 2

Today was a day for classes at ITP, so less time to play with the network - but time to hear some amazing things in Gene Kogan’s machine learning class “Autonomous Artificial Artists”. The goal of this class is to create exactly what the title implies: an autonomous artificial artist. A neural network “living” and “working” on the blockchain. This means as well that the training will be distributed and ideally the network will continously retrain itself. This sounds very very up in the air - but is worth pursuing in this 6 week-class. I am very excited about this, especially in the light of my thesis. Here I wanna create a more evolutionary distributed neural network as a form of a positively viral entity. Maybe I bring on the blockchain as well, this might distribute power more evenly. Something to consider.

An interesting and refreshing look on my thesis from a performance (and technology) perspective is this lecture I recently stumbled upon: From Fluxus to Functional A Journey Through Interactive Art by Allison Carter.

Looking at my own little network I have to be a lot more humble: the architecture I chose seems not to be working well, I cannot get the error down fast enough. Here my architecture for the classifier:

Screen Shot 2019-01-31 at 11.09.58 PM.png

And this is the result after an hour of training and 18 epochs:

Screen Shot 2019-01-31 at 11.09.58 PM.png

This is not great. The loss is still high, and stuck around 2.6 - something needs to be fixed, either in my code or my hyper-parameters. Maybe I just need to train longer? Or switch the pre-trained network that I use as a basis for my transfer-learning network? So far I was using vgg16, I want to use densenet tomorrow and see if I get better results. I will need to experiment over the weekend to get an accuracy around 80%.

I still want to avoid to just get an off-the-shelf working network from github and train it with my data. I have done that too often in the past. This time I want to build it from scratch on my own.


day 3 & 4

On Friday I had Designing Meaningful Interactions with Katherine Dillon. Another great class and another great (and maybe different) perspective on my thesis - this time through the lens of design. I technically always saw my thesis project as an art project, not a design project. Since that class it has changed a bit. At least I am trying to think of it how it would shape as a design endeavor. Well, how would it look like? I am trying to think of users now, potential owners of intelligent & connected objects, performers of artistic actions. And I am somehow back to my rock that I created last year: there is something to the idea of designing an object of devotion for people that are not religious but spiritual. I have been thinking a lot about that in the past two days. Will it be for humans only? Or would it be for robots? How would such an object look like and what would it actually do?

Well, lots of questions come to my mind and my original plan is a bit put on hold to answer them, first. In the design class, the main point of designing an interaction is to think of how an object makes the user feel. I like this way of thinking. Maybe that helps me as well to write the fantasy-dream review we are supposed to write for Tuesday.

So I came up with a few emotions that my object and the connected action should trigger:

  • belonging

  • connectedness

  • hope

  • calming

  • sensuality (digital and physical)

  • openness

  • wonder

In that light I just tried to sketch a few objects that should be interconnected and maybe triggered by true randomness? Maybe controlled by one centralized neural network? Ideally those objects should change their shape, the cubes should move around, maybe glow, like a clock that is running on true randomness instead of seconds/minutes/hours. Ideal size: it should fit into the palm of your hand.

dev_object_studies.png
dev_object_studies3.png

On top of this I was reading about the meaning of devotional objects in Renaissance Italy, the ubiquity and use of these animated objects in domestic devotion and a (possible) history of atheism - while moving to a new studio. What a journey so far!

day 5 & 6
After mixed results with vgg16 I tried out a different pre-trained network, densenet121, for image recognition to do transfer-learning on my experimental flowers-test set. Technically I would switch this test set to frames from Fluxus-performance footage. I still wonder if this is the right approach and have to talk to Gene Kogan, our resident here at ITP about ideal training setup. For my test-example gives me a better understanding for the network architecture and transfer learning in general - something I wanted to explore a bit more in depth this semester. I changed the classifier architecture a bit and used only 1 hidden layer:

Screen Shot 2019-02-05 at 1.28.37 PM.png

This leads to faster drop of the loss, but then the loss got stuck at around 1.2 (65% accuracy) - maybe due to this very minimal classifier structure:

Screen Shot 2019-02-05 at 1.28.24 PM.png

On top of that I continued with my c4d-renders for ideation on a certain design-framework for a physical representation of my spiritual art-AI. I just got a post card from a good friend of mine in Germany that summarizes this activity (diving into color and shapes, happiness, obsession and unconditionality):

IMG_0240.jpg

Digital Fabrication Final: Clay VR Future Nudes

Continuing from our explorations with base64 and clay, Kim and I created a VR/physical sculpture that mimics cyborg art in the future.

concept

We meditated the whole semester on the relationship between our physical and the digital body, explicitly on the space between physical and digital object. We tried to deconstruct the notion of sculpture as a form of human memory to the blueprint of code - only to re-imagine it in VR.

We called our piece “future nudes”, as only cyborgs in the future will be able to see the real body behind the sculpture without the help of VR goggles as we imagine them to be able to decipher code in real-time.

process

A depiction of the artists body gets laser-etched in binary code into wet clay, spread out on 12 clay tablets. These tablets get fired in a traditional kiln like the earliest forms of human written media, Babylonian cuneiform. Images of these tablets get put into a immersive VR environment and when the audience is touching the real clay tablet they can see their hands interacting with the physical object of the tablet - which in itself is the depiction of a real person converted into a digital blueprint, binary-code. In the final version the pixels of the body parts on each tablet pop up in the VR environment when the fingers are touching it.

In a first step the highly pixelated image (to minimize the binary file size) gets transferred to binary via a python-script (here one part of the image is shown):

male_torso_illustrator_binary.png

After that we prepared the low-firing raku-clay for laser-etching and kiln-firing. After applying manual focus on the 75 watt laser we used the following settings:

  • 600 dpi raster etching

  • 90 % speed

  • 30 % power

We got the best results when using out-of-packet fresh clay without adding any water to it. The clay fired well at cone 04 for the bisque fire.

Here a few pics from our process:

IMG_7751.JPG
IMG_7827.JPG
IMG_8074.JPG
Oi%H9qYxSQ+WRE0a4f5V7A.jpg
IMG_8350.jpg

And finally the experimental and still unfinished port into VR using Unity VR, HTC Vive and Leap Motion (in the final version the pixels of the body part pop up in the VR environment when the fingers of the audience touch the tablet):

IMG_8114.JPG


Live Web / Machine Learning for the Web Finals: Peer to Peer Meditations

For this final I combined the techniques I learned in “Live Web” with the ones from “Machine Learning for the Web” and built a peer-based meditation piece:

Users can experience the base64-code from a remote web-camera streaming through their own body.

Try it with two browser windows or two computers/mobiles: Peer to Peer Meditations

All of this runs with webRTC, tensorflow.js and node.js as main parts on the backend: peer.js handles the web-rtc peer connection with a remote peer-server, node.js organizes the automatic connection to a random user in the peer-network, tensorflow.js runs person-segmentation (now renamed to “body-pix) on the local-users webcam - all the canvas-handling and pixel-manipulation is done in javascript.

The result is a meditation experience that reminds of the calming effects of watching static noise on tv screens. As it is peer-based, the users body becomes a vessel for the blueprint and essential idea of something or somebody else. This hopefully creates a sense of connectedness and community while meditating.

While staring at a screen for meditation might not be everyones preferred way of doing it, it is worth exploring this from a futuristic perspective: we probably will be able to stream information directly into our retina, therefore an overlaid, peer-based meditation experience might be worthwhile considering in the future.

Here a short video-walkthrough streaming video-frames from my mobile phone camera as base64-code through my own silhouette (the latter is picked up by the camera on my laptop and run through a body detection algorithm). All web-based with machine-learning in the browser powered by tensorflow.js and webRTC peer-to-peer streaming (code):

Understanding Networks: RESTful API and Controller

As outlined in the first blogpost, the assignment for our group was two-folded:

  • build an API & Server for our classmates Vidia and Lucas - they built an interactive limbo-stick after our specifications

  • build a physical controller for our classmates Simon and Keerthana - they built a dance-step projector

The assignment proved to be a great challenge, especially the coordination tasks that come with working as group with two other groups (that do the same) on two different projects over a longer period of time with no pre-defined work environment - but we finally succeeded!

Lucas and Vidia made a video for their limbo-stick machine, built after our API specs/deployed server.

And here a pic and code from the physical controller we built for Simon and Keerthana (and their dance machine): laser-cut/etched acrylic, lcd-screen, arduino mkr1000 wifi, potentiometers, bamboo box

wVoNicRHSQiiz2rabdSpAA.jpg

And a little prototype video of the controller in action (controlling the node-based dance step controller built by Simon and Keerthana):

Machine Learning for the Web: Code Bodies with tensorflow.js

concept

We are code. At least in a great part of our lives. How can we relate to this reality/non-reality with our bodies in the browser?

tech setup

I experimented with tensorflow.js and their person-segmentation model running the smaller mobile-net architecture. I used multiple canvases to display the camera stream base64 code in real time behind the silhouette of the detected person from the webcam. Given the fact that I do pixel manipulation on two different canvases and run a tensorflowjs model at the same time, it still runs relatively fast in the browser - although the frame rate is visibly slower than a regular stream with just the model.

prototypes

A brief screen recording:

Another version with a bigger screen:

 


 

Digital Fabrication Final: Future Nudes

For our final project, Kim and I were experimenting with laser-etching wet clay this week - a poetic and exciting exploration!

Too make it short - so far the results were surprisingly great. That said we still need to fire the clay in a kiln, then we can give a final verdict on it.

Here the basic setup for cutting the clay:

IMG_2948.JPG

We used low firing white raku clay as we will use a kiln that fires at cone 06. It had a nice plasticity and was easy to work with. The tricky part was to get a consistent height for the clay slab. To achieve that, we used a kitchen roller and two pieces of plywood with similar height to roll it evenly over the clay. We then cut it in shape.

IMG_2951.JPG
IMG_2952.JPG

To keep clay particles from falling through the laser bed we used felt.

After applying manual focus on the 75 watt laser we used the following settings:

  • 600 dpi raster etching

  • 90 % speed

  • 30 % power

and it worked well with two goes. Now we just have to let it dry for two days do a first firing, glaze one part and then fire it again. It’s still not clear how it will turn out - something to look forward to!

IMG_2957.JPG

Understanding Networks: RESTful API

update

assignment week1/2

Decide on your application, then describe and sketch the control surface and the specify the REST interface. Describe your application and present your REST API. Don’t build the project, just describe it and its functions. Write a blog post detailing this description. You’ll be assigned someone else’s specification to build this week.

concept

Inspired by the playfulness and shared group experience of parties Sandy, Kellee and I decided to create an interactive and RESTful Limbo-Stick.

context

A little bit of research on the origins of Limbo showed that the dance is considered the unofficial national dance of Trinidad & Tobago, together with steel drums and calypso as national heritage. In its original form the bar is raised slowly from the lowest levels, resembling the entry of death into life. When the dance gained more popularity this form was flipped - dancers are starting now high and the bar slowly lowers.

system overview

 
 

physical interface (draft)

Usual limbo setups look like our rough sketch:

 
 

RESTful API

For the RESTful API for the dance/game we followed an example to define our five basic states: “off”, “on”, “height adjustment”, “idle”, “collision”.

POST-requests

//*On/Off Mode*
POST /switchMain/{bool} 
//value: 1 to turn it on,  0 to turn limbo stick off

//*Height Adjustment Mode*
POST /height/{bool}
// Value: 1 to lower the height of the limbo stick one step on the predefined scale, 0 to keep current position

//*Collision Mode*
POST /collision/{bool}
//value: 1 if collision is detected, 0 if no collision is detected

//*Alarm Mode
POST /alarm/{bool}
//Value: 1 to turn it on, 0 to turn alarm off

//*Home Mode*
// If POST /height counter  > 6, go into home mode:
POST /home/{bool}
//value: 1 to home the stick to highest position on the predefined scale, 0 to keep it current position

GET-request

GET /state/

//returns
{
  "switchMain": {0, 1},
  "height": {0, 1},
  "collision": {0, 1},
  "alarm": {0,1},
  "home": {0, 1}
}

physcial interface/controller

 
Screen Shot 2018-11-19 at 8.08.17 PM.png
 

Digital Fabrication Final Proposal: Future Nudes

Kim and I want to create nude portraits/sculptures for the future (see here more detailed blogpost on Kim’s website .

After laser-etching our portraits in Base64 code on mat-board for our midterms, we decided to continue with this topic and play more with form and materials:


We want to either use Base64 or binary code for the encoding of the image. Regarding the base material for the project we want to iterate with etching wet clay. It seems to be the best material as it preserves our portraits pretty much forever (the earliest form of human writing, cuneiform, was “hand etched” into clay). It has the advantage that we could form it later into a sculpture or play with the shapes before burning. And it has a lot of symbolic meaning regarding the human body in different contexts (bible, Kabbalah, …).

It is pretty experimental to etch into clay and then form it, there are only very few sources online that have tried etching wet clay with mixed success. So we gotta play!

Why nude portraits? We like the idea that the portraits of our bodies will only be visible in a distant future - once humans are able to decipher code naturally, maybe as human-machine hybrids. We abstract our bodies into code and preserve them for a time where our bodies will have changed a lot: we might be half human, half machines by then. This aspect of preservation for a distant future reminds of the skor-codex from a few years ago.

For the looks of ceramics we are inspired by rough surfaces of dark stoneware, dark clay and playful sculptural explorations.

An interesting iteration would be a collecting a couple of different nudes, then cutting up the pieces into a row of beaded curtains that people could walk through - so they could interact with our bodies in code.

Machine Learning for the Web Class 1: Playing with tensorflow.js

For our first assignment I played with the exmples provided in the tensorflowjs-models git. I used the posenet-example to create a mini-skateboarding game for the browser:

The user has to jump to avoid the rocks that are coming along the way. And that’s it! I simply changed the color, strokes and dots of the skeleton and attached a few shapes between the two ankle-points (plus some minor canvas & font additions).

This construct does work somehow, as long as the light in the room is good. Still a lot of it is pretty rough, the error detection (board hits the rock) is not very accurate and needs audio-feedback, the rocks are only on one level of the canvas and there is no counter for points. But so far so good for some first browser-machine-learning.

I tried to deploy it on my server but got lost in ES6 vs Node errors. So for now just a video and ran locally.

For those wondering about the title: It is taken directly from an old NES game from 1988.

I had a lot of fun jumping around to test the game, I guess my neighbors downstairs were not really that amused … :


LiveWeb / Machine Learning for the Web - Final Proposal

concept

For my final I want to start building an interactive, decentralized and web-based devotional space for human-machine hybrids.

background

Heart of the project is the talking stone, a piece I have been working on over the last semester already and showed at ITP Spring Show 2018. Now I would like to iterate on it in a virtual space, with a built in decentralized structure as a form of a “community”-backbone via the blockchain and as an interface geared towards future human entities.

system parts

The connection to universal randomness: a granite rock.

IMG_2739.JPG

Rough draft of user interface (website):

IMG_2936.jpg

And here the full system as an overview (as well a rough draft):

decentr_stone_schematics.png

iteration with tensorflowjs

I built a web-interface as an iteration on the idea that is fed by the camera input and displays it as Base64 code within the silhouette of the users body:

project outline

Here the basic setup: The rock is displayed via sockets as a Base64Stream on the website for logged in users. Users have to buy “manna” - tokens with Ethereum (in our case the Rinkeby-Testnet blockchain), then they get an audio-visual stream of decaying particles of the rock to start their meditation/devotion: The rhythm of particles decaying is happening - according to quantum theory - in a true random manner. Therefore the core of this devotional practice of the future is listening to an inherent truth. The duration of each “truth” - session is 10 minutes. I will have to see how “meditative” the random decay actually is - I remember it as pretty soothing to just listen to the particle decay that gets hammered into the rock with a solenoid, but I will have to find a more digital audio coloring for each particle decaying. That said - this piece is pure speculative design. I am interested in the future of devotional practice, especially for people who consider themselves non-religious. So trial and error is probably the way to go with this project as well.

If the user moves significantly from the prescribed meditation pose (Easy Pose — Sukhasana) in those 10 minutes, tokens get automatically consumed. The same happens if the user does not open a new ‘truth’ - session within 24 hours after finishing the last session.

On the screen, little dots display active users that are in a session as well. The size of the dots changes according to how much time they have left in their daily meditation practice.

The goal of this experimental devotional space is to give users an audio stimulation that is scientifically true and therefore easier to identify with - in a sense a random rhythm of the universe to meditate with. By buying into the practice with tokens, only dedicated users are using the space and their motivation to actually perform the mediation is higher as they paid for it. They only have to pay again if they do not perform the practice in a regular manner or interrupt a session.

The visuals will be dominated by code - as this is a devotional meeting place for future human-machine-hybrids that find peace and solitude in true randomness (the opposite of their non-random existence).

tech specs

  • granite rock with geiger-counter attached to it

  • raspberry pi with camera running sockets to display rock and send random decay patterns

  • remote server running node.js and sockets

  • tensorflow.js running posenet

  • ethereum contract (solidity based) running on Rinkeby-Testnet (dev blockchain)

  • MetaMask chrome extension (to access blockchain from browser with a secure wallet)

challenges

That sounds like a lot to do in 6 weeks, but I want to give it a try. I experimented with blockchain and true randomness last semester already in two different projects and the current LiveWeb / Machine Learning for the Web classes seem a great framework to give this a virtual and AI guided setting. I am still a bit uncertain about the blockchain backbone as this is the part where I feel the least at home at the moment. I only remember fragments of Solidity, web3js and MetaMask, connecting all layers together was tricky and the documentation sparse. Well, we’ll see!

Understanding Networks: Packet Analysis

Assignment:

Capture and analyze traffic on your home network while going about your usual network activities. Present your results in summary form, using graphical analysis where appropriate.

background traffic analysis

I used wireshark throughout the assignment in the ITP-sandbox network.

After a few basic experiments with querying a very basic webpage and looking at the sent/received packets, I went back to zero: I started measuring just traffic when all applications are off over the course of 5 minutes. Surprisingly there is still a lot of background network traffic happening, especially application data, without me actively engaging in any online activities.

OS X source traffic (my IP: 128.122.6.150)

OS X destination traffic (my IP: 128.122.6.150)

Ubuntu source traffic (my IP: 128.122.6.149)

Ubuntu destination traffic (my IP: 128.122.6.149)

I used two operating systems to compare the background traffic: OS X and Ubuntu. Ubuntu only shows one third of overall network traffic compared to OS X.

The latter one shows a lot of connections to IP addresses starting with 17 - these are registered to Apple in Cupertino. A more detailed view shows a variety of different activities that my OS X operating system is performing and sending data back and forth between Apple servers and my machine:

Screen Shot 2018-10-27 at 3.59.12 PM.png
Screen Shot 2018-10-27 at 4.43.15 PM.png
Screen Shot 2018-10-27 at 4.42.20 PM.png
Screen Shot 2018-10-27 at 4.37.04 PM.png



Some further research on the sent TCP-packets showed that those processes refer to Apple cloud services (as I am using iCloud to sync my photos, calendar etc.) or programs like iTunes.

On my Ubuntu machine, I do not use any cloud service that that is tied to the operating system, therefore 1/3 of background activities.

As I got interested in this “hidden” traffic I did some further research on the TLS - Layer and used wireshark to go through each step of the TLS -protocol:

 
 
tls.png
 
 

Looking at the cipher change protocol specifically, I did further research on the encryption part (here the negotiated cypher-suite) of TLS - and finally understood Diffie-Hellman and RSA encryption. That was worth the extra hours … ! I have to confess that trying to sketch all components of RSA encryption still gives me headaches, compared to that Diffie-Hellman seems a bit more simple and elegant. To me it was not obvious to choose which one over the other, intuitively I would choose Diffie-Hellman over RSA. And I wondered why Apple is “downgrading” my (possible) SHA384 encryption to SHA 256 in the cipher suite negotiation in the protocol.

Screen Shot 2018-10-27 at 7.25.07 PM.png

After reading the Diffie-Hellman vs RSA post on stackexchange, I was a bit less confused: Non-ephemeral RSA encryption seems to be the industry standard for now as its generally faster to compute. Diffie-Hellman is more secure, but more difficult to compute.

I suspect Apple is using the lower encryption standard (RSA with SHA 256) due to the sheer volume of traffic on their servers.

usual daily network traffic analysis

For my usual daily network traffic analysis I had a couple of browser windows open, mail and terminal. In one of the browser windows I was running an online-radio station (nts.live). I captured data over the course of 5 minutes.

Here the output looking at the source-traffic (my IP: 128.122.6.150, running OS X):

Here the destination side of traffic:

Finding out on which data channel my online-radio station is running proofed to be difficult as the stream is hosted not under the website-ip address but on a different server. I suspected either one of the ip addresses below as they showed a continuously high traffic via TCP (initially I expected UDP but as the stream is https, TCP makes sense). Both pointed to amazon-servers.

Screen Shot 2018-10-29 at 11.30.57 PM.png

To find out which one might be the one hosting the stream, I just closed the tab running the online radio station. Here the results:

no_radio.png

To my surprise both were muted now and didn’t appear on the traffic overview anymore. As the website is hosting two streams at the same time, I guess it might load both from different servers even when I would only be able to listen to one? My fellow classmate Beverly pointed me into the right direction: I should check the sent packets in Chrome directly - and here two streams are loaded at the same time! This is for sure eating into the bandwidth …

Screen Shot 2018-10-29 at 11.48.07 PM.png

Surprisingly the connection to the Apple servers was somehow quiet during multiple wireshark-captures while running Chrome and Mail. The server-IP starting with 17 (apple server range) does not appear in the traffic overview at all. Why this is the case, is not quite clear to me. Maybe background processes are only run while no other traffic is using the bandwidth? I can only guess at that point.

Now enough of packet sniffing, TCP, TLS and UDP - I learned a lot and got a lot more interested in encryption, which will be the topic of a future class in a few weeks. Awesome!

Digital Fabrication Midterm: Future Portraits

“In another reality I am half-human, half-machine. I can read Base64 and see you.”

final iteration

Kim and I created self-portraits for the future.

concept

The self-portraits are Base64 representations of images taken by a web-camera. The ideal viewer is a human-machine hybrid/cyborg that is capable of both decoding etched Base64 and recognizing the human element of the artifact.

process

We went through several iterations with felt, honey and rituals: Our initial idea of capturing a moment in time and etch or cut it into an artifact morphed into an exploration of laser-cut materials with honey. We were interested in the symbolic meaning of honey as a natural healing material. Our goal was to incorporate it into a ritual to heal our digital selves, represented by etched Base64-portraits that were taken with a webcam and encoded. We used felt as it is made out of wool fibers that are not woven but stick to each other through applying heat and pressure. This chaotic structure seemed a great fit for honey and clean code:

IMG_2855.JPG

Soon we dropped the idea of using felt as it seemed to be too much of a conceptual add-on and reduced it to honey and digital etchings - the healing should be applied directly onto the material in a ritual.

IMG_2854.JPG

After gathering feedback from peers and discussions about the ritualistic meaning we struggled with a justification for the honey as well: most of the people we talked to liked the idea better that only human-machine hybrids of a near or far future are technically able to see the real human portrait behind the code. After a few discussions about both variations of our concept dealing with digital identities and moments in time we favored the latter one and dropped the honey.

So we finally settled on etching timeless digital code onto a physical medium that ages over time - self-portraits for the future: Maybe we look back at them in 20 years and can see through the code our own selves from back in the day?

A little bit about the technical process: The image is taken with a nodejs chat app that I created for LiveWeb: It takes a picture with the user’s webcam every three seconds and shows the Base64 code on the page - again an example of an interface for human-machine hybrids or cyborgs of the future.

machine_facetime.gif

After taking portraits with my webapp, we copy/pasted the code for each of us into MacOS TextEdit, exported it as a pdf, copy/pasted the contents of it into Adobe Illustrator and outlined the font, Helvetica. We chose this font as it is a very legible font, even at a very small font size. Our laser computer did not have Helvetica installed, therefore we outlined all letters.

fullsizeoutput_310.jpeg

The files were very large, as portraits varied between 160,000 and 180,000 characters.

These digital preparations were followed by 2 days of test-etchings on crescent medium gray and crescent black mounting-boards and experiments with different laser settings (speed, power and focus).

IMG_2912.JPG

We discovered that getting the focus right proved to be difficult: The laser beam seems to get weaker once it travels all to the right of the bed. This makes the etching illegible on that side, whereas the far left it is fine. Focusing a bit deeper on the material with the manual focus produces satisfying results with white cardboard fixed this issue, whereas the fonts looked blurry on the black cardboard on the left side - it was etching too deep. Finding the right balance for the focus fitting both sides equally well took a long time and a lot of trial and error.

IMG_2915.JPG

Once we got the focus right, we started with the the final etchings: Speed 90, Power 55, 600 dpi and focus slightly closer to the material than usual turned out the best results on our 75-watt laser.

Each portrait took 1:33 hours to etch, in total we completed 4 portraits.

We see them as a starting point for further iterations, as “exploration stage one”: The concept of creating physical artifacts for the future that preserve personal digital moments in time with code and laser is very appealing to us.

We will iterate on our portraits for the future probably for the final.

IMG_6999.JPG
IMG_2887.JPG
IMG_2864.JPG
IMG_7004.JPG
IMG_6998.JPG

LiveWeb: Midterm Project Idea

Quite often the rapid prototyping process that we celebrate here at ITP (and can be really difficult to get used to as I like very much to spend more time with a project before going on to the next one …) has the great side effect that you go through a lot of different ideas, topics and digital narratives that can be told. This sometimes means you end up finding a hidden gem that really sticks with you for a while.

Last semester it was the rock-project and its devotional aspects that kept me occupied (and still does).

This semester I am fascinated by creating apps for human-machine hybrids for a probably not so distant future.

For the last Live-Web class I developed an image-chat app that shows bare Base64 chat images instead of decoded (human readable) images. This means only a human-machine hybrid can fully enjoy the chat: a machine itself is limited to the non-conscious decoding of data and can’t enjoy the visual image, a human cannot enjoy it either as the other participants are hidden behind a wall of code (Base64). Only a human with machine-like capabilities could fully participate and see who else is in the chat.

So far so good.

But the prototype was very rushed and lacks a few key features that are conceptually important:

  • real continuous live stream instead of image stream

  • hashing/range of number participants/pairing

  • audio

  • interface

  • further: reason to actually be in such a chat/topic/moderation

I would love to try to address these with the following questions for my midterm (and possibly as well final as this seems to be a huge task):

  • can this be live-streamed with webRTC (as code) instead of images every 3 seconds?

  • how and by who can it be encoded that the stream is only readable by a select circle that possesses a key? Is the rock coming back as a possible (true random) moderator?

  • how would the audio side sound like? or is there something in the future that is kind of like audio, just as a pure data stream that opens up new sounds? and how does data sound like?

  • how to compose the arrangement of interfaces for the screen?

  • which design aesthetics can be applied to pure code?

  • a bit further out there: how would a chat look/feel/sound like in the future with human-machine hybrids? what lies beyond VR/AR/screen/mobile as interfaces?

Let’s get started!

Another iteration on this for Digital Fabrication:

IMG_2860.JPG


Digital Fabrication: Midterm Iterations on Healing, Honey and Felt

“In your backbone you feel a pointed something and it works its way up. The base of your spine is tingling, tingling, tingling, tingling. Then n|om makes your thoughts nothing in your head”

[Kxao ≠Oah - a healer from |Kae|kae area, quoted in Biesele, Katz & St Denis 1997:19]

(taken from JU|’HOANSI HEALING SONGS on NTS-radio)

IMG_6856.jpg



Screen Shot 2018-10-11 at 12.28.37 AM.png

For our midterm in Digital Fabrication, we iterated on the idea of the swarm based behavior. After reading more about Joseph Beuys’ use of felt and honey in his fluxus-performances in the late 60s and the JU|’HOANSI tribe and their use of healing through dance, we shifted our focus to the idea of a healing ritual for the digital age.

We are thinking about making a piece that uses three components in an interactive installation:

  • an image in Base64 code on paper (used for image encoding on the web)

  • felt with a perforated structure

  • honey

As shown in the image above, a depiction of the artists in a waterfall-like silhouette made from Base64 encoding is printed on paper. This is the foundation of the sculpture that reflects ourselves that we want to heal. Above the paper is a layer of felt cut in strips and shaped in a web. It filters the honey dripping from above which acts as a part of the healing ritual: Beuys saw honey as a healing material as it is gathered by bees, which represented a “peaceful” entity.

A coincidence we found out while cutting felt with the laser: it smells like honey afterwards.

IMG_2852.JPG

LiveWeb: MachineFaceTime

I created an image chat application that can be fully used or seen only by machines. Or you need to be fluent in Base64 to see who you are connected to. Every three seconds an image from the chat participants is displayed and sent around via sockets - it is kept in the original data format for image encoding on the web, Base64 and shown in a very small font. The focus of the eye shifts on the code as an entity and creates rhythmically restructuring patterns. The user-id is displayed along on top with each received image.

To make it work I had to hide various elements on the page that are still transmitting data via the sockets. It works as well on mobile. Just in case you are in need of some calming Base64 visuals on the go.

(code)



ezgif.com-video-to-gif.gif


Live Web: Collective Drawing in Red and Black

I modified our example from class a little bit … and added a button to change the collective drawing color: it is either red or black, everybody is forced to adapt to the new color but can choose the canvas they want to work in. As the canvas is split between a red and black background the “pen” can be either used as an obscurer (black on black/red on red) or a highlighter (red on black/ black on red). The results look strangely beautiful as the dots seem to have depth or resemble a point-cloud:

Screen Shot 2018-10-02 at 4.57.17 AM.png

… and here some code. And more random drawings:

Screen Shot 2018-10-02 at 11.26.58 AM.png

Understanding Networks: Traceroute

assignment

Traceroute at least three of the sites you regularly visit. Do it from all of the locations you regularly connect from. Save the trace in a file, and make a map of the routes, indicating the network providers that show up every time. Identify who the major tier 1 providers are in your life. Identify the networks your traffic crosses in the course of your daily life. Figure out whose hands the data about your life goes through on a regular basis. Look for patterns from your network-browsing habits through analysis and graphing of your network traces.

setup

I traced four websites that I frequently visit:

I chose two networks/locations for tracing that I spend most of my time at:

  • university wifi (+ mobile network comparison in same location), at Washington Square/Manhattan

  • home wifi (+ mobile network comparison in same location) in Brooklyn

I wrote two custom tracerouting tools to automate the process as far as possible:

  • tracer.py: combined traceroute and isp-lookup/geoip for given list of urls, returns json-txt file

  • mapper.ipynb: matplotlib based tool using basemap to create traceroute maps from json-txt file

server footprint from home

 
 

I live in Brooklyn. This is where I spend the few hours that I am not on the floor at ITP.

Looking at the footprints I observed that connecting to my homepage (which I registered in Europe with Squarespace) from my homw wifi causes the trace to bounce back and forth between EU and US - until finally reaching the US - and the routes my signal travels on are owned by Time Warner Cable and Akamai Technologies:

The route looks similar when using a mobile connection, but with different providers: Zayo Bandwidth and Level 3 Technologies stick out, Akamai Technologies comes up again as ISP in Europe:

Looking at another footprint (nts.live), Time Warner Cable dominates the servers in the US:

The same trace with mobile emphasizes again Zayo Bandwidth for mobile-networks:

Connecting to NYU and Stackoverflow does not yield too many interesting results, both stay more or less close to or entirely in New York. The only strange behavior comes from trying to traceroute Stackoverflow on mobile - it does not allow to traceroute, the signal gets stuck in a local private network (in all locations).

Here the connection to Stackoverflow via wifi which travels to Denver and then comes back to New York:

 

server footprint from ITP/NYU

 

ITP/NYU is situated at Washington Square in Manhattan. Here I spend most of my time, logged into NYU wifi or the mobile network.

Comparing the traces of the two wifi-networks (home and university), the paths through the servers look different for my homepage - in the US the network provider is not clearly identifiable, NYU provides the last visible trace before it goes to Europe. There GTT Communications comes up as a provider:

The trace for nts.live shows a connection to Europe that does not come in the map from my home wifi, the network provider that pops up is Hibernia Networks Netherlands. Why? It might have to do with NTS having multiple IP-addresses, maybe the server was more easy to access from NYU. Maybe. I can only speculate at the moment. Anyway, here the map, accessed from NYU wifi:

On mobile the connection stays in the US (and again in the hands of Zayo Bandwidth as ISP):

Takeaways

To make it short - this is all very interesting! My online life is in the hands of very few network providers, they vary depending on which type of network I am connected to - and the routes vary sometimes substantially, detours to Europe are not always explainable for me. I really enjoyed understanding much more of this physical layer of the internet and how every request from a laptop of phone travels around the globe at -literally - light-speed.

I thoroughly enjoyed building custom tools in python for it and dive a little bit into data visualization with matplotlib and basemap - although I encountered quite a few challenges on my way: nested dictionaries are great, but need a logical setup, building tools takes way more time than actually collecting and visualizing the data.

Let’s finish this blogpost with a screenshot from parts of the json-data (a little bit obscured):