Data Science & Arts Hack your Brain: creating a significant audio-immersive literature experience

Shortcuts / Table of Contents
Intro - The Preparation - The Modelling - The Experiment - The Future - 
Continuation of the Project – Part 2 

This audio-immersive literature experience is made possible using the brainwave measuring tool Emotiv Insight and further code which is developed for this purpose. We proudly present the HSLU Data Science project & research team:

Professor Mr. Peter Gloor, MIT Massachusetts (Boston) Institute of Technology:
Innovative Researcher, Scientist, Lecturer & Entrepreneur

Data Science finds more and more fields of application, also the art sector is getting interested in it. Together with Maison du Futur, it has been decided to plan an experiment, where the faces and brainwaves of the participants and their emotions will be recorded to measure the effect of "resentment", expressed through spoken language and referring to an "object" (here: a book) on a recipient's emotions. 

Maison du Futur is a national innovation center for the arts, where all forms of audio-visual and performative storytelling are developed and presented in combination with new technologies, such as Artificial Intelligence, Augmented Reality, and Big Data – and we were taking part in it. After an intense preparation phase, the experiment took place in Zurich and the data could successfully be tracked and is ready for further analysis in the next semester. 

As our society becomes more and more individualistic and people are used to being treated individually, especially online, this field of study is very interesting.

The most significant part of this research was to collect enough brainwave data to develop a model. Friends, family, and colleagues were invited to take part as test subjects and let us “borrow” their brains with the help of the Emotiv Insight measuring tool. This device measures activity in all cortical lobes of the brain, providing in-depth information typically found only in research devices. We did not miss the chance to also test the device on our heads.

Our test subjects, including ourselves, watched an approximately 10 minutes long video by COIN's Personality Prediction Project which triggers different emotions, such as happiness, anger, disgust, sadness, and neutrality. Data was collected not only with brain waves but also via face recognition. So, at this point, data about their emotions recognized by face and real-time display of raw Electroencephalography (EEG) and performance metrics were available.

An electroencephalogram records the electrical signals of the brain by using electrodes that are attached to the scalp. Brain cells communicate with each other using electrical impulses and brain activity will show up on an EEG recording as waves. Six basic measures of mental performance (engagement, excitement, focus, interest, relaxation, and stress), derived directly from mental activity. Each measure is automatically scaled to suit the normal range and base level of each condition. Now different models could be built.

Emotions of the brainwave data were matched with the expected emotions from the face recognition and then labeled. With the help of Deep Learning, a model to recognize emotions was built. A Long Short-Term Memory (LSTM) network was chosen which is a type of recurrent neural network. LSTM is capable of learning long-term dependencies in data. Promising results were found during the training phase of the model with high accuracy. In case our model would not be accurate enough in the new setting, we decided to add additional emotions collected by face emotion detection. Thus, new approaches had to be taken. Luckily, we were also able collecting emotions from face detection. 

The day of the experiment was here. Again friends, family, and colleagues were brought together to collect more insights about their emotions – but this is not all.

At the Max Frisch Bad, each experiment participant was brought to a room marked by literature. They were set down on a chair, facing two books – one supporting feminism and one opposing it - both written by women. The books were set in dark lighting to not put too much attention on them. The measuring device was carefully installed on their heads – sometimes with more hassle than intended, depending on the person's hairstyle. Calming music started for the introduction phase in which the person also was asked to read two articles about the feminist and anti-feminist female authors.

After the 5-minute introduction phase, an emotionally loaded audio reading by the Swiss author Robert Walser’s “Der Spaziergang” was played. The audio reading was about a specific scene in a bookstore, in which the protagonist was really opposed to a certain book, which was not named. After enough suspense has been built, THE BOOK was lit up in front of the subjects. The lit-up book was the one mentioned in articles, which was very affirmative to feminism. The same reading, but in a neutral voice, started again and this time the anti-feminism book was lit up.

EXPERIMENT IMPRESSIONS
Some impressions of the experiment "creating an audio-immersive literature experience", taking place at the Max Frisch Bad in Zurich

Subsequent to the experiment, a survey looking for the participant’s political orientation was handed out. The goal was to see, if they would react differently to the two books which were highlighted, depending on their political orientation. Thereby, for the emotionally loaded audio stronger emotions were expected to be measured.

With an initial analysis immediately after the experiment, the data-based evidence showed clear results. The version with the angry audio and pro-feminism book provoked more emotional reactions than the second version. However, the data also indicates that these emotions were triggered less by the provocative book and more by the speaker's tone of voice. The point, at which the voice tips into shrillness is clearly visible visually, opposed to the point when the book was lit up.

Through deeper analysis of the recorded data and a further development of the algorithm, more subtle emotional reactions, such as in our example the feelings caused by a polarizing book, the political attitude or the influence of the setting, should become detectable in the next step. For us, the most exciting further development of the project also lies in this direction. 

The detection of emotions with EEG data still has a slightly futuristic touch. Especially outside of laboratory conditions and standardized data sets. And this is precisely, where the greatest appeal of this project is: exploring new ideas at the intersection of data science, art, and emotions, without knowing exactly, where the journey will lead. Maison du Futur in their role as artists "producing" emotions and experts in the search for something new, were of course a perfect partner in crime for this experiment.

The fact, that we learned something about literature in addition to neural networks and brain waves, was more than just a pleasant side effect. And that such partnerships between art and data science are more than just fun. The fact that they are also highly relevant outside of term papers is no longer a secret, at least since the buzz of AR, VR, and Metaverse. 

In the meantime, the student team has worked another semester analyzing the data collected from the experiment. Their main insight was that when the participant was listening to the angry voice and shown the pro-feminism book, it provoked more emotional reactions than a calm voice and showing an anti-feminism book. However, the data also indicates that these emotions were triggered less by the provocative book and more by the speaker's tone of voice. The point at which the voice tips into shrillness is clearly visible visually, while the point when the book was lit up triggered much lower emotions. Thus, by deeper analysis of the recorded brainwave data, it was shown that different reading styles decisively influence the emotions of the listener. This was tentatively confirmed by comparing brainwave data with face emotion data of the listener. In conclusion, this project illustrates that using EEG for the detection of emotions provides a valuable new tool for enriching and improving audio art project installations. 

We want to thank the "Hack your Brain" Team and Maison du Future for sharing these interesting insights into their mindblowing project!

Authors: Ardiana Hoxha, Daniel Gisler, Nicole Hediger, Lisa Marino
Professor & Lecturer: Peter Gloor, MIT Massachusetts (Boston) Institute of Technology:
Innovative Researcher, Scientist, Lecturer & Entrepreneur