I collaborated with Aazalea Vaseghi on this chat-project: A self correcting/censoring chat application that only allows positive sentiments. It runs sentiment analyses on each user input and matches the darkness of the message background accordingly. When the message is too negative it gets censored with an entire black background so that the negative message disappears. Code is on github.
For my summer internship 2018 I worked at the advertising agency Havas New York in the creative team for a corporate client as AI-researcher: For two and a half months I was exploring machine learning techniques for rapid prototyping with existing neural networks.
As this was client based work I will add results of my work to my portfolio once it is publicly available.
As part of my internship I worked as well on an intern-project for a non-profit as creative technologist.
I thoroughly enjoyed my work at Havas New York, it was a great learning experience that helped me to clarify my focus for the next year in school: I want to explore machine learning on a deeper level, clearly understand the maths behind it and start building my own models and networks.
I am currently enrolled in the Udacity course “AI programming in Python” that covers the math basics and combines it with the necessary libraries in Python to build networks from scratch at a lower programming level. I hope that this course will reinforce my existing Python knowledge, help me to understand neural networks from a mathematical perspective and give me a solid background of the model mechanics when using higher level APIs in Tensorflow or PyTorch.
Thanks Havas New York for giving me such a great opportunity to use my machine learning skills in an agency landscape and especially Ali Madad, Marc Maleh, Marc Blanchard, Joseph Delhommer and Nick Elliott for their guidance and mentorship.
I created a game controller with the RaspberryPi Zero W using websockets in Python and the GPIOs connected to switches. I chose this setup as the PiZero offers a terminal to run shell commands, can be accessed headless via ssh and offers the possibility to run Python scripts. It is also very portable and can be mounted on a small controller.
The connection to the game server via sockets was very smooth and reliable, the GPIO connection with the switches worked well - after quite a few hickups regarding the wiring of the switches: I initially wired all switches directly into one power source without separating them via resistors. As a result all switches fired with each other once one was triggered. After a few hours of hardware debugging (and de-soldering …), I mounted the resistors in between each power connection and the switch and everything worked smoothly.
I finally added a LED to indicate a successful socket connection to the server.
Here the code for the socket connection and GPIO wiring in Python:
import RPi.GPIO as GPIO # Import Raspberry Pi GPIO library import socket import time s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) s.connect(('Server-IP',Port)) GPIO.setwarnings(False) # Ignore warning for now GPIO.setmode(GPIO.BOARD) # Use physical pin numbering GPIO.setup(35, GPIO.OUT) GPIO.output(35, GPIO.LOW) #check connection data=s.recv(1024) if len(data) > 0: print('yay, we are connected') GPIO.output(35, GPIO.HIGH) msg1='l' msg2='r' byt1=msg1.encode() byt2=msg2.encode() msg3='l' msg4='r' byt3=msg3.encode() byt4=msg4.encode() #for i in range(0,50): # s.send(byt2) # time.sleep(2) # s.send(byt1) # time.sleep(2) #s.close() def button_callback_l(channel): print("go left!") s.send(byt1) def button_callback_r(channel): print("go right!") s.send(byt2) def button_callback_u(channel): print("go up!") s.send(byt3) def button_callback_d(channel): print("go down!") s.send(byt4) GPIO.setup(11, GPIO.IN, pull_up_down=GPIO.PUD_DOWN) # Set pin 10 to be an input pin and set ini$ GPIO.add_event_detect(11,GPIO.FALLING,bouncetime=500, callback=button_callback_l) # Setup event o$ GPIO.setup(7,GPIO.IN, pull_up_down=GPIO.PUD_DOWN) GPIO.add_event_detect(7, GPIO.FALLING,bouncetime=500, callback=button_callback_r) GPIO.setup(13,GPIO.IN, pull_up_down=GPIO.PUD_DOWN) GPIO.add_event_detect(13, GPIO.FALLING,bouncetime=500, callback=button_callback_u) GPIO.setup(15,GPIO.IN, pull_up_down=GPIO.PUD_DOWN) GPIO.add_event_detect(15, GPIO.FALLING,bouncetime=500, callback=button_callback_d) message = input("Press enter to quit\n\n") # Run until someone presses enter GPIO.cleanup() # Clean up
I used the video playback speed functionality in html to make the most out of a very short moment in time of my life - waking up in the morning a few days ago. The user can wake me up again and again, the moment is played back at random speeds and either extended or shortened and finally looped.
Here the code (including code sources that where used):
As part of our assignment we should as well look for interactive websites that offer live interactions to its users. I chose NTS.live, an online radio-station with two live shows and the possibility to listen back to shows in an archive. The interesting interaction here is the live radio show that is broadcasted over the internet, users can interact with the hosts via all social networks, these interactions are then re-told/narrated/moderated by the show-hosts. This is a highly filtered interaction, as not every communication between user and show host will be re-broadcasted or commented on - but a very charming mix of the old-fashioned live radio show with social media.
As the semester moves towards its end, my project development studio piece is getting more and more real. Here a quick rendering of the setup that I made for my ITP spring show proposal:
And some changes to the code as I am trying to get rid of a few bugs.
This weekend I took my Energy final project out in the sun and it worked - at least most parts of it: I can power the geiger-counter entirely with a 12 V / 3.4 Watt solar panel - no LiPo involved. I cannot get enough current out of the panel to power a solenoid on the side as well. To avoid using another solar panel and keeping the project still portable I decided to slim this part down and accept the final outcome: a solar powered geiger counter with an Arduino Nano as a data outlet. I cannot run my entire art installation from project development on a small solar panel - but it is remarkable that the small panel can power the geiger-counter and therefore produce voltages up to 400V (needed to activate the geiger-mueller tube)!
Here the prototype test outside:
And a live test:
Thinking about the course and my projects in it so far I thought since a long time that it might be worth revisiting my only partially finished project for the midterms.
It took me a few hours to successfully debug my hardware + code and had three insights:
- a bigger power supply (in my case a 6600 mA Li-Ion Pack) is great for the Raspberry Pi 3
- don't copy paste into Github from terminal - check for tabs and spaces
- always check if your project share a common ground
I got the wallet / blockchain connection and the servo part working on solar (with a gentle support from the Li-Ion pack ...), here the rough prototype (here just running from Li-Ion pack):
After compiling ffmpeg on my raspberry pi locally (takes a long time ...) I tested streaming to youtube from the pi and then embed this into my website. Streaming from the Pi to Youtube following this tutorial worked, at full resolution only if the Raspberry Pi is plugged into a standard power socket - the Li-Ion Pack can only support a streaming resolution of 320x240 at the moment, maybe the bottleneck is the booster. Here the battery powered prototype:
After a lot of tweaking of the ffmpeg-stream parameters I finally decided to re-solder some of the connections - using thicker wires. And I finally got all parts (digging the blockchain, running the servo while streaming live to youtube - with battery/solar power only) working. Very buggy, it crashed after a minute ... but it worked!
I am super happy! And I know that I need to iterate on this: This is a temporary fix as I ideally would like to the use webRTC as I used it on the mac-version of the project.
This class really was a journey, a lot of insights and learning moments - so far am so satisfied that I can run both projects independently with solar on a raspberry pi and an Arduino!
I am super happy that my classmate Yen, who happens to work at ITP as well, heard about my project and gave me a beautiful big natural stone that he kept in his office . Very likely it is granite and therefore a good source for natural radiation / random numbers.
So last night I sat down and did a few experiments measuring the amount of decaying particles with and without a stone close to the geiger-counter. I conducted 3 test rows, so three times with the stone, three times without the stone.
Here are the results measured over the time of 3 minutes each:
The stone from Yen's office seems to be slightly radioactive, around 15 - 20% compared to "normal" conditions that mainly detect gamma rays from cosmic radiation. It is still very low compared to carbon-monoxide detectors, big granite kitchen countertops, old watch luminescent hands, which emit many more decaying particles over this amount of time.
installation prototype setup
I decided to do a quick run to test parts of the setup. Here a solenoid knocks the random particle decay from inside the stone that gets detected by the geiger counter back into the stone - the heart of the stone is talking to its audience:
So far so good - tomorrow I have to test the setup powered entirely with two solar panels - let's hope the sun is out and the panels can generate enough current to power the entire installation!
As I am currently looking for a digital translation of the physical interface of the random-project (stone and knocks with hand or hammer / solenoid as the "hammer of randomness") that might be part of the installation piece and create a more immersive space for the audience, I am exploring cinema4d shapes and materials to find the right fit for the "devotional device" that the installation tries to create: the rhythm of the animation would be triggered by the particle decay inside the granite (and some gamma-particle noise probably caused by solar flare). The materials of the digital objects should remind of devotional objects: precious stones, gold and fur. These become animated and "alive" through true randomness.
And I am quickly realizing the relationship between computational power, render-time and animation/material complexity - it takes forever to render a short animation in high-res ....
Here a few more screenshots and animations:
(animation above based on mograph-tutorial)
For my final in my energy-class I decided to create a solar powered version of my installation piece for project development studio: In this iteration of my project, the audience can only "connect" to the true randomness of the granite when the sun is out.
I will build a more complex installation for the spring show that will not run on solar, this one for my energy-final will run entirely on it.
Here a modified wiring schematics that is based on an Adafruit tutorial for using the piezo and solenoid in combination:
I am using a logic level converter between Arduino nano running on 5V and the geiger-counter running on 3V. The solar panel is providing power for the Arduino and the solenoid. The piezo is running on a separated power-circuit provided by the Arduino. This circuit feeds as well into the logic-level converter which is providing power to the geiger counter (convert voltage down) and enables listening to its pulse (convert signal voltage up).
The code for the installation is still in the making, I still have to merge the solenoid trigger into the piezo and geiger counter code. The comparison of geiger counter beeps and knocks to identify whether the knocks of a user are in sync with the randomness of the particle decay inside the granite needs to be improved as well.
Last week I took a few moments to play with cinema4d - I somehow felt explorative with creating animations for my random-project. So far I am not sure how exactly it could fit in, but there is something to this motion ... maybe three screens / panels on each of the side-walls of the room that are triggered by the random-ness of the geiger-counter underneath the black granite?
This past week I focused on sourcing a bigger piece of granite (granite is by nature slightly radioactive and will be the source for generating random numbers with the geiger counter) and the circuit for the user interaction. I used an example from the adafruit-learn section as the basis for my code which will interface with the geiger-counter. The circuit will perform the following tasks:
- listen to knocks of the users against the stone with a piezo element
- listen to geiger counter (convert 3.3 V of geiger-counter pulse to 5 V of Arduino digital-in)
- compare the knocks of the user with the inner true random decay pattern of the granite (measured with the geiger counter)
- trigger a solenoid for 10s in the true random decay pattern of the granite if user and stone pattern align / if user and stone are "in sync"
Two weeks ago I had the idea to center the devotional piece around a granite rock as source of randomness - with a geiger counter measuring the (random) decay of radioactive particles from the rock. Here a chart tracing the decay of Radon gas and an explanation of the true random nature of this subatomic process:
Last week I assembled a geiger-counter kit - lots of soldering and lots of fun:
Now the setup can be more finalized:
After countless days and hours of debugging, two bricked/one broken SD cards and a final kernel panic when running a simple python sketch it is time to take a little break and resume. So let's have a look where I was a week ago with my project - at that time everything was running fine, the Pi was accessing the blockchain and the servo reacting to changes in my account. All of that streamed in real time. At that time with two laptops:
So far so good: listening to changes in the chain and the servo re-action happened battery powered on the Pi Zero W, the streaming was done with webcams and laptops, the server part plus frontend ran on my Amazon Instance.
My next step was to try to run the Pi-part with solar power. It worked as expected with a 5V booster and a Li-Po between Pi and solar panel. Then I tried to run the main local part (camera stream and querying blockchain) with battery (a pre-step before using solar exclusively) - after a few days of debugging and hacking web-rtc for streaming live from the pi via my server this worked as well (with a great lag due to the slow network). Great!
Motivated by this success I tried it with solar power on a sunny day - from an energy perspective a success, not so much from a computational one. My streaming code and/or Pi image started to behave strange - it had a bug. And here I probably made a few too optimistic assumptions about the energy provided by the small solar panels, running the Pi at full CPU usage requires more amps than expected. And more than the panels can provide. It should take me a few days to fully realize this - and (spoiler alert!) finally delay the final project vastly.
But at that time I was optimistic to solve this issue in the next days (I assumed it is a code issue, not an energy one) and started working on the enclosure for my project.
After running the separate codes on the py I started to see a mysterious "Bus error" - which seemed to indicate corrupt SD cards, a common issue on the Pi. I re-imaged different cards a few times - just to run again into the same IO-Bus error. I was still optimistic to solve this - I knew that the code worked a week before on the Pi (at that time with a steady battery supply) - why should it not work anymore now? I hadn't changed a single line as far as I could remember.
I spent 3 days in a row trying 2 new SD cards (which are completely unusable now), broke one other SD card, tried numerous debugging strategies - only to run into a kernel panic after a fresh install of Raspbian during a very late night session.
That was the point where I had to laugh out loud - too many things went wrong before and this appeared just absurd. I realized that I could not solve this quickly - but that I had done as much as I can to solve it. It was indeed disappointing to work this hard on a project only to see that nothing work=ed anymore at the end of the week. I basically started on the top and ended up at the bottom. After another day of debugging and presenting my progress in class more and more clues popped up that this might not be a software/code issue, but related to the power supply - in my case the LiPo/Solar setup. I finally stumbled upon this forum post:
I realized that I probably need more Amps to power my application - meaning a LiPo that is more powerful. The solar panel should be fine, if exposed to direct sunlight as much as possible. In a real life installation outdoors the conditions (temperature) would affect the battery life as well, for this project I assumed indoor use.
So far in this project I learned a lot: Solar works, computation powered by it on a Pi works as well, the sleep/wake up external circuit is simply great - but better stack a few more Amps into the circuit when running computational intense operations on a Pi.
idea & prototype
After experimenting with blockchain implementations for IoT devices I decided to use a Raspberry Pi Zero W for a first iteration on my solar project for energy: a physical pay-for-a-smile interface for the blockchain.
basic circuit setup
Pi Zero W + Camera, micro servo, 5V booster, 3.7 V 1200mAh LiPO, charging/load circuit, solar panel
I re-measured the power consumption for the project and averaged the values over time/runs. The Pi Zero showed a significantly lower power consumption compared to the Pi 3 (in mA).
I am still contemplating on how to create a deep sleep mode that fits the concept of the project. At the moment, the Pi is listening constantly on changes in the blockchain, then reacts by turning the servo when a user sends money into the account. Conceptually the project would be online forever (so that users can continuously trigger the smiley face), the camera would need to stream constantly as well (something still not finished in pi sketch, at the moment I am using my laptop camera). This is not possible with the current circuit as it will be down after 4.5 hours if running exclusively on battery.
LiPO specifics: 3.7 V, 1200mAh
LiPO + booster: 5 V, 890mAh (rounded)
runtime for continuous sketch (ignoring the servo peaks): 4.5 hours (rounded)
This would mean it would inevitably stop at nighttime when there is no daylight to charge the LiPo from the solar panel. Assuming the physical interface would be mounted outside, it would make sense to power it off at night - as the camera would not be able to film the interaction anyway. This would as well emphasize the connection of the piece to a real environment - which a physical web-interface tries to evoke anyway.
Nevertheless, an external deep sleep circuit is necessary. I am currently waiting for an external hat for the pi that will manage the power with an automatic shutdown/wakeup schedule.
Hopefully the sun will come out in the next few days so that I can run the sketch at least for one or two days to test the performance of the circuit over time and alter it if necessary.
I want to experiment in the next few days with writing data to the blockchain directly with the Pi Zero W to store environmental data from a temperature sensor in the network permanently with solar power.
As an iteration on last weeks ideas I am thinking of not using the quote at all and let users listen to a rhythmic pattern (generated by random numbers that are measured by a geiger counter, a granite stone is the source of radiation and random radioactive decay), and repeat it with a hammer on the same source of radiation - as if they were trying to influence the infinite randomness of the universe. Maybe humans can? Probably not. But we can still try like Camus' Sisyphus:
"The struggle itself [...] is enough to fill a man's heart. One must imagine Sisyphus happy" .
The rhythmic signal in the headphones gets recorded and played back automatically from speakers in the background of the installation.
This setup would not need the the prediction of the quote by Emile Borel, it could serve as the title of the piece:
"Whatever the progress of human knowledge, there will always be room for ignorance, hence for chance and probability."
The process of generating the rhythmic pattern would be started the moment the users gaze at the granite (as a hint at the role of the observer of quantum processes). The duration of the rhythmic pattern would be determined by the first random number generated by the stone.
Here an overview of the setup:
- Mighty-Ohm geiger counter
- 2 x Raspberry Pi Zero W with phatDAC (for audio out to speakers and headphones)
- speakers and headphones
- NVIDIA Jetson Tx2 plus external camera for gaze detection
After last weeks feedback session I sat down with my classmate Azelia who decided to join the project and discussed different aspects of the piece:
marbles or candles
During our conversation we talked about the role of marbles in visualizing the random process as a hint at infinity and spirituality. Azelia suggested to go even further and use candles for the visualization - I would love to build such a candle lighting machine, although I am aware of the issue of fire safety and possible misunderstanding when a candle is not lit up (audience might think the machine is not working correctly). So far we will stick to the marbles then and work on the mechanics of the machine to transport the marbles back after usage.
clean/minimalist looks vs unsettling traditional referrals to religion
We were as well discussing the possibility of using traditional materials and looks like in churches to create an unsettling atmosphere - like in installation pieces of Beuys or Schlingensief. We later decided for a more minimalist look with cast concrete and a granite stone in the middle (that will be a natural source of radiation for producing the random numbers).
flow of user/audience interaction
So far we are not sure what would be the best setup of the installation regarding the user interaction: The disconnect between the random guessing process that happens in some sort of "black box", the machine, and the ritual of putting the marbles with their hands on the concrete ring is still there. Azelia had the idea to activate each process via gaze tracking/user observation (which hints nicely at quantum mechanics where the particles fall into a definite state once they are observed which is the basis of the random guessing of the piece) - I think that is a great idea. It raises as well the question if the user should be part of the ritual afterwards or leave it with this ":gaze activation".
quantum computer or decay of radioactive particles
After some further research and experiments on the IBM quantum computer (which is very slow at the moment) I found another way to create real randomness with quantum processes locally: with a geiger counter connected to a raspberry pi measuring the decay of radioactive particles (e.g. a granite stone) - which is based on quantum mechanics as it is impossible to predict when specifically the particles will decay. This setup would be much more tangible for the audience the granite stone which is slightly radioactive could serve as the centerpiece of the installation. I am . not sure if the quantum computer can achieve the same presence.
Over the past week I worked on the marble detection (distinguish black and white marbles) with a RaspberryPi 3 and a camera:
So far openCV worked great for the colors red and green but not very good for white. After a few attempts in getting greater accuracy I abandoned openCV on the Pi and went for a much simpler solution - compare brightness values. Now the camera takes a picture of the marble in the slot and a python script using the python library skimage analyses the brightness of the overall image. As the background of the marble slot is always black, the overall brightness changes significantly when a white marble is in the slot (compared to a black marble).
Here the simple code:
This means I can now distinguish black and white marbles using the PiCamera and send the right marble-color to the audience who will put them on the installation piece.
Early prototyping-setup for testing with Pi and PiCamera:
My focus this semester in many courses is the blockchain: I believe that a server-less, decentralized and consensus-based network is the future of the web. I want to explore different use cases for DApps (decentralized apps) and DAOs (decentralized autonomous organizations) in an art and design context.
In my "Energy"-class we were assigned with the task to power computation (in any form) with solar power. After investigating the energy consumption linked to the proof-of-work algorithm in blockchain applications like bitcoin or Ethereum I decided to explore how far this could be powered by solar energy. After some research I realized that the proof of work is theoretically a great idea - but not sustainable from an environmental and economic perspective in the longer run. Ethereum is planning to replace the proof-of-work with a proof-of-stake algorithm which relies on a stake-based verification/mining system rather than the cryptographic-puzzle that needs to be solved for proof-of-work. This will require less computational power (and real energy) as a hardware arms-race (miners competing for mining a block first) is avoided by design.
Based on my research that I presented in class I decided to look at use-cases for micro-controllers and blockchain. These should be low energy as they are powered by solar energy only and store environmental data gathered by sensors in the blockchain - creating a 'memory' of environmental data. So far the open-source IoT devices capable of running the necessary systems (node & geth) are rare: I found one git about measuring temperature with a Node 8266 board. This will be my starting point for further exploration. I plan to base my further work on this project on the hack and measure the necessary power consumption of the board, then explore an optimized use of solar energy (without relying too much on battery storage). I will investigate if it is possible or necessary to try to run a node directly on the device (highly independent but probably more power hungry) or use the device as an ultra low-energy transmission tool to another central device running a node via wifi/bluetooth/lora.
I plan to start with measuring environmental data, but the use cases could be quite varied: I am working in my Project Development Studio Class on an installation piece that runs possibly for an infinite time trying to randomly guess with quantum processes a certain quote on probablity - it would be conceptually interesting to store the failed attempts of this process "forever" in the "world memory" of the blockchain. Solar power would make sure that this guessing runs for an infinite time (excluding material failure).
To familiarize myself better with more abstract concepts in solidity (the language for Ethereum) I started with CryptoZombies, an interactive, game-based learning platform that teaches programming for the blockchain - by building a zombie game on the blockchain.
Directly related to our topic of energy:
In Solidity, your users have to pay every time they execute a function on your DApp using a currency called gas. Users buy gas with Ether (the currency on Ethereum), so your users have to spend ETH in order to execute functions on your DApp.
How much gas is required to execute a function depends on how complex that function's logic is. Each individual operation has a gas cost based roughly on how much computing resources will be required to perform that operation (e.g. writing to storage is much more expensive than adding two integers). The total gas cost of your function is the sum of the gas costs of all its individual operations.
Because running functions costs real money for your users, code optimization is much more important in Ethereum than in other programming languages. If your code is sloppy, your users are going to have to pay a premium to execute your functions — and this could add up to millions of dollars in unnecessary fees across thousands of users.
This means I should look into the most efficient ways to write contracts and store data in the blockchain - probably based on trusted open-zeppelin contract templates and decentralized storage with IPFS.
Here the PComp side of the setup: ESP 8266 wifi-module, LiPo, Solar-Panel + adafruit mcp73871 solar charger
We tried and tried and tried to get the kinectron working - and got it sometimes working, except the p5-code ;)
But we got xyz-values from the kinect (v2) to p5!
Today I ran my first live experiment on the public IBM Quantum Computer that will be the backend of my installation piece - I am so excited!
I tried an entanglement setup:
"Two or more quantum objects are entangled when, despite being too far apart to influence one another, they behave in ways that are 1) individually random, but also 2) too strongly correlated to be explained by supposing that each object is independent from the other." (ibm-guide)
Above the user interface with the applied gates to create a superposition (h-gate in blue), entangle (+ sign) and measure (pink gates) the states of the particles (q0, q1). And here the results: