Thesis Week 3: thesis proposal, unity ml-agents, system theory

After submitting my thesis proposal on Friday, my focus on the weekend was on exploring the functionality of ml-agents in unity and reading on system theory, namely the debate between Niclas Luhmann and Juergen Habermas.

I did a few basic tutorials on training agents using reinforcement learning with the unity game-engine with a Tensorflow backend with tensorflow sharp. Here the agent learns to roll a ball towards a cube:




To learn more complex actions with no immediate reward, researchers at Berkeley recently developed a curiosity mechanism for reinforcement learning that motivates the agent to explore and experiment in its environment. Here it needs to pull a switch (red light turns green) first (no external reward), then a pyramid with a yellow cube on top appears somewhere (still no external reward) and if knocks down the pyramid and the yellow cube on top is touched by the agent, an external reward is added. To motivate the interest of the agent in exploring all the steps above before any external reward is added, the model works with an internal reward mechanism running two neural networks against each other: it compares predicted action and predicted observation with the real observation and action and gives out a reward the higher the difference between expectation and reality is. Meaning the bigger the surprise, the bigger the internal reward. Unity incorporated this into their ml-agents recently to model more complex behaviors - which could help me for my simulation of religion and rituals. I trained the example model for one hour, here the result:


To test how well the trained model is able to abstract objects I changed the environment: instead of a golden cube, the agent has to knock a golden sphere of the pyramid to get an external reward. As this scenario varies slightly from training the agent has difficulties to deal with this new environment:

inference (untrained change in environment)

Thesis Proposal: Eden


I propose an interactive sculpture that connects the audience in a gallery setting with a simulation of religion that is invented by neural networks.

I am fascinated by complex systems and the way individuals dynamically organize themselves around the basic human needs for hope and certainty. Therefore my main artistic focus of my thesis will be on emergent behaviors of systems, namely on how the creation of rituals influences the activity of machine learning agents in their environment. In this simulation, the agents are rewarded for inventing new rituals with the elements of their world. The role of the audience will be to temporarily add true randomness to this complex system - and therefore create non-algorithmic chaos. This uncertainty will have consequences on the agents' rituals. The audience is provoked to think about the paradox of a system that seems to be only partially rule-based.

Living in a world we try to make sense of but will never fully understand is the key to human existence. Establishing a sensitivity for our own hopes and fears when faced with the complexity of the world and ultimately create a more intuitive understanding of it is the main goal of my piece.


The theoretical part of my piece is influenced by the works of system theorists/philosophers/sociologists Theodor Adorno, Max Horkheimer, Niclas Luhmann, Juergen Habermas and Jacques Derrida, mathematical approaches in modeling complex systems and current research in the field of neural networks, especially deep reinforcement learning. My artistic approach is rooted in the Fluxus-movement, namely Joseph Beuys' performative focus on art as a collective form of healing. 

While iterating with reinforcement-learning examples in Unity/Tensorflow and rendering possible versions of the final piece in c4d, I will do further theoretical research on Derrida and his view on the relationship between society and religion, more reading on the communication models of Luhmann and Habermas and gain a better understanding of chaos-theory (Lorenz attractor).

I will to talk to Gene Kogan about the main technical setup for the project, contact Ben Light to go through my fabrication plans and schedule an office hour with a faculty member of NYU Department of Computer Science with a background in deep reinforcement learning. For interaction design considerations I will consult Katherine Dillon and artistic advice Danny Rozin. 


Adorno, T. W., & Horkheimer, M. (2016). Dialectic of enlightenment. London: Verso.
Baring, E., & Gordon, P. E. (2015). The trace of God: Derrida and religion. New York: Fordham University Press
Ha, D., and Schmidhuber, J. (2018). World models. arXiv preprint arXiv:1803.10122.
Hui, Y., & Stiegler, B. (2016). On the existence of digital objects. Minneapolis: University of Minnesota Press
Londei, A. (2014) Artificial neural networks and complexity: an overview. Rome: Archeologia e Calcolatori Supplemento 6
Luhmann, N. (2013). Theory of society (Vol. 2). Stanford Univ. Press
Smith, L. A. (2007). Chaos: A very short introduction. Oxford: Oxford University Press

So far in my research I gained an overview over applicable theories, thinking frameworks and tools. Most of it in the field of system theory, complex systems and devotional practices/artifacts in various cultures and times. 

In the next weeks I will go deeper into the mathematical side of complex systems, the relationship between religion and sociology, the nature of rituals and reinforcement learning as a modeling tool for sociological behaviors.


My academic background is in the fields of sociology, politics and literature. I am familiar with the theoretical frameworks that are relevant for my thesis, especially the sociological and philosophical part. My private interest is in french philosophy of the 20th century with a focus on Existentialism. I have been creating artworks using machine learning / neural networks since three years after taking Gene Kogan's machine learning for artists course in Berlin (1 month intensive course). I recently took a udacity class for a deeper dive into the maths of machine learning and am now proficient enough to build my own basic ml-architectures in Pytorch from scratch. I have used the higher level interfaces of Tensorflow and am experienced with training and deploying GANs and CNNs on various remote server platforms. Last summer I did a 3 month internship at Havas New York as AI researcher in the Creative Department and built multiple prototypes for a client. 
I explored in depth the element of true randomness in Project Development class with Danny Rozin and built a meditative sculpture that lets the audience engage with a rock (via measuring its truly random decay of subatomic particles with a geiger counter and mapping that to the audience actions) and true randomness. 

I will use deep reinforcement learning for my setup. As this is different to the works I have done so far in the field of ML (most of it was language or image based and generative), I will focus on getting a good understanding of the algorithms underlying the Unity-ml agents and then decide which framework is the best for my piece: stick to unity, or building a pure python backend or even entirely browser based. The reward function for sociological behavior has to be developed from scratch and I have to decide if I want a physical entity (robot) in the piece. The latter depends on how much time I have left after settling on the ml-framework. 
The form of interaction of the audience with the sculptural piece is technically set, I can rely on my knowledge from my last piece using geiger-counters to get true random numbers.

I envision the piece to consist of 5 random number generators that are embedded in stone-sculptures. The agent-simulation will be projected on the floor around these rocks with projection mapping techniques. The audience is encouraged to engage with the sculptures through a small hammer. Knocking the hammer against the rocks will trigger and feed random numbers into the simulation and confuse the learning process of the agents. They will then modify their observable behavior and approaches to come up with new robotic rituals - the projection on the floor will change accordingly. 

So far my draft schedule will look like this:

  • February - 2 weeks of theoretical research and first prototypes with unity-ml while getting on top of deep q-learning in Pytorch. Rough draft of exhibition setup/fabrication ideas, early user testing

  • March - 2 weeks narrowing down final version of ml-side and finalizing choices on fabrication/setup, deeper user testing. 2 weeks of building network and fabrication

  • May - debugging, final technical touches, setup

Thesis Week 2: Iteration, Experiments

After a lot of tinkering I finally found a working architecture for my image classifier (still just on flowers, should be fluxus performances frames soon) using densenet and a lightweight one layer as new classifier:

Screen Shot 2019-02-07 at 1.51.34 PM.png

The accuracy on a test -dataset is now around 75%:

Screen Shot 2019-02-07 at 1.45.51 PM.png

While doing this I iterated a lot on traditional catholic holy water fonts, as I wanted to look at spiritual rituals/actions that have been used in traditional religions (I used Catholicism as I grew up with it) and try to rethink their design. I wanted to incorporate the hand movement towards an object that is in the focus of attention (here either cubes or a golden sphere:


Some experiments did not work that well and just look strange:


Others had something but need a lot more iteration:


As I mentioned earlier in my blogposts that I somehow want to use honey in the installation (as a symbol of healing) I started experimenting with the real flow plugin for c4d to get photorealistic renderings of liquids. Unfortunately the renderfarm I used for my usual renderings doesn’t support the plugin and I have to render locally on my mac (which is very very slow - one frame takes a day to render). Anyway, here a little test for honey simulation:


I thought about a design object that consists of a digital and physical layer, combining them in an optical illusion: the golden sphere gets inserted in the object, then gets displayed on the screen, seemingly “floating” through the honey. Ideally the user has the feeling this happens to the real sphere:


Thesis Week 1: Research, Concepts, Ideas

week 1

I decided to write a little bit about my process regarding thesis every day and put everything together in a weekly post.

day 1/night 1

After yesterdays session with our advisor Kat Sullivan, it’s time to go through things for thesis today: A little bit of concept, a little bit of research and a bit of planning for this week. Not too detailed as my concept depends heavily on prototyping, so I will have to come up with milestones after each week - and decide on how to continue.

research question

How can art and technology enrich our spiritual search?


“Even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence at the place where it happens to be.” 

(Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction)

real time, real space art performance


create a spiritual performance for robots around a central artistic AI: robots and AI try to save human mankind with art

(c4d render based on  py-c4d tutorial )

(c4d render based on py-c4d tutorial)


based on Joseph Beuys and greek mythology my thesis tries to “heal” - honey is used as a metaphorical tool for this

(c4d render based on  py-c4d tutorial )

(c4d render based on py-c4d tutorial)


  • stage 1: create a viral AI that helps humans to be artistic every day

  • stage 2: destroy any digital code of the virus, only encrypted code on clay tablets survives

  • stage 3: create a physical sculpture as a devotional object made out of tablets and AI

  • stage 4: create a ritual for commercial mini-robots (anki, alexa, …) where they try to decrypt the tablets (e.g. do fluxus performance around the sculpture) to save human mankind

first steps

  • this week is devoted to choosing and building the right network architecture, training data (performance art in video form, mostly fluxus) and a first trained network as a draft (probably image based, frame-by-frame object detection / transfer learning + image net as a starter)

    Here the very basic and transfer-learning network structure I am experimenting with (based on the Udacity AI Programming with Python online class I took last year):

classifier = nn.Sequential(OrderedDict([
                            ('fc1', nn.Linear(25088, 200)),
                            ('relu1', nn.ReLU()),
                            ('dropout1', nn.Dropout(p=0.5)),
                            ('fc2', nn.Linear(200, 120)),
                            ('relu2', nn.ReLU()),
                            ('dropout2', nn.Dropout(p=0.5)),
                            ('fc3', nn.Linear(120, 102)),
                            ('output', nn.LogSoftmax(dim=1))

I am currently hosting everything on Google Colab for quick prototyping, back to watching the loss going down:

While doing this, playing with particles in c4d offers superb distractions:

  • if there is additional time I will try to research more on computer viruses / worms

day 2 / night 2

Today was a day for classes at ITP, so less time to play with the network - but time to hear some amazing things in Gene Kogan’s machine learning class “Autonomous Artificial Artists”. The goal of this class is to create exactly what the title implies: an autonomous artificial artist. A neural network “living” and “working” on the blockchain. This means as well that the training will be distributed and ideally the network will continously retrain itself. This sounds very very up in the air - but is worth pursuing in this 6 week-class. I am very excited about this, especially in the light of my thesis. Here I wanna create a more evolutionary distributed neural network as a form of a positively viral entity. Maybe I bring on the blockchain as well, this might distribute power more evenly. Something to consider.

An interesting and refreshing look on my thesis from a performance (and technology) perspective is this lecture I recently stumbled upon: From Fluxus to Functional A Journey Through Interactive Art by Allison Carter.

Looking at my own little network I have to be a lot more humble: the architecture I chose seems not to be working well, I cannot get the error down fast enough. Here my architecture for the classifier:

Screen Shot 2019-01-31 at 11.09.58 PM.png

And this is the result after an hour of training and 18 epochs:

Screen Shot 2019-01-31 at 11.09.58 PM.png

This is not great. The loss is still high, and stuck around 2.6 - something needs to be fixed, either in my code or my hyper-parameters. Maybe I just need to train longer? Or switch the pre-trained network that I use as a basis for my transfer-learning network? So far I was using vgg16, I want to use densenet tomorrow and see if I get better results. I will need to experiment over the weekend to get an accuracy around 80%.

I still want to avoid to just get an off-the-shelf working network from github and train it with my data. I have done that too often in the past. This time I want to build it from scratch on my own.

day 3 & 4

On Friday I had Designing Meaningful Interactions with Katherine Dillon. Another great class and another great (and maybe different) perspective on my thesis - this time through the lens of design. I technically always saw my thesis project as an art project, not a design project. Since that class it has changed a bit. At least I am trying to think of it how it would shape as a design endeavor. Well, how would it look like? I am trying to think of users now, potential owners of intelligent & connected objects, performers of artistic actions. And I am somehow back to my rock that I created last year: there is something to the idea of designing an object of devotion for people that are not religious but spiritual. I have been thinking a lot about that in the past two days. Will it be for humans only? Or would it be for robots? How would such an object look like and what would it actually do?

Well, lots of questions come to my mind and my original plan is a bit put on hold to answer them, first. In the design class, the main point of designing an interaction is to think of how an object makes the user feel. I like this way of thinking. Maybe that helps me as well to write the fantasy-dream review we are supposed to write for Tuesday.

So I came up with a few emotions that my object and the connected action should trigger:

  • belonging

  • connectedness

  • hope

  • calming

  • sensuality (digital and physical)

  • openness

  • wonder

In that light I just tried to sketch a few objects that should be interconnected and maybe triggered by true randomness? Maybe controlled by one centralized neural network? Ideally those objects should change their shape, the cubes should move around, maybe glow, like a clock that is running on true randomness instead of seconds/minutes/hours. Ideal size: it should fit into the palm of your hand.


On top of this I was reading about the meaning of devotional objects in Renaissance Italy, the ubiquity and use of these animated objects in domestic devotion and a (possible) history of atheism - while moving to a new studio. What a journey so far!

day 5 & 6
After mixed results with vgg16 I tried out a different pre-trained network, densenet121, for image recognition to do transfer-learning on my experimental flowers-test set. Technically I would switch this test set to frames from Fluxus-performance footage. I still wonder if this is the right approach and have to talk to Gene Kogan, our resident here at ITP about ideal training setup. For my test-example gives me a better understanding for the network architecture and transfer learning in general - something I wanted to explore a bit more in depth this semester. I changed the classifier architecture a bit and used only 1 hidden layer:

Screen Shot 2019-02-05 at 1.28.37 PM.png

This leads to faster drop of the loss, but then the loss got stuck at around 1.2 (65% accuracy) - maybe due to this very minimal classifier structure:

Screen Shot 2019-02-05 at 1.28.24 PM.png

On top of that I continued with my c4d-renders for ideation on a certain design-framework for a physical representation of my spiritual art-AI. I just got a post card from a good friend of mine in Germany that summarizes this activity (diving into color and shapes, happiness, obsession and unconditionality):