TACTILE EXPLORATIONS
Making and Consultations
For week 4, I started on a new experiment that explores the addition of touch interaction within a Multisensory Interface. After that, I presented it along with my newly focused research topic to Andreas for the group consultation session we had for the week.
To explore the addition of tactile interaction within a audio-visual interface through a interaction with a rattan board.
A touch-visual-audio interface. Participants touches a rattan board that triggers a visual change on the image on screen (Image goes from B&W to colored). A audio interview also starts playing when the image is fully colored.
Ng Si Ying is a Singaporean rattan artisan (@atinymaker) that specialises in the crafting with Rattan. She has crafted bags, rings and even airpods cases just by using rattan. This experience is meant for participants to feel the touch of the material she works with and also to hear about her unique story as a rattan artisan.
Experiment 2 - Rattan Artisan's Story
B&W Portrait with Instructions
Post-interaction colored portrait
I wanted to have a visual transformation on screen that seems obvious. Thus, I thought of changing the image from black and white to colored, symbolizing the image coming to life upon interaction. To achieve this, I started by downloading a portrait of the rattan artisan and created two images: one black and white and one colored. I added some instructions on the black and white version to help guide the particiapnt on what to do.
Currently, I do not yet have a Arduino and Conductive tools to allow me to use conductivity as a way to register interaction data with the rattan board. Thus, a work around I had was to use hand detection instead. To achieve this, I used the MediaPipe plugin for TouchDesigner, which is a extensive plugin that facilitates embodied interactions with TouchDesigner visuals. It was quite simple for this experiment as all I needed was for the plugin to register that a hand is in frame.
Two Images linked to Cross TOP. Cross Value references Hand Data
I first imported both versions of the rattan artisan portrait (colored and BW) with a MovieFileIn node. The whole idea here is to fade in and overlay the colored image over the base BW image, to give the illusion that the image is visually transforming. I done this by using a Cross TOP. The more the Cross value in the Cross TOP increases (0-1), the more prominent the colored image will be, until it hits 1 and completely overlaying the BW image. The key here is for me to map the Hand Tracking data to the Cross value and allow it to trigger the Overlay effect. This ultimately gives the result I want: Hand Interaction > Image becomes colored (coming to life).
Importing the interview and soundscape audio into TouchDesigner and mapping hand data to trigger "play" and "reload"
Next is to make the rattan interaction also trigger the interview segment. To start, I downloaded an audio clip of the Rattan Artisan's interview and imported it in TouchDesigner with AudioFileIn. I then mapped the hand data to act as a trigger for the "Play" and "Reload" button. Mapping the hand data to "play" is simply because I wanted hand detection to start the audio playing. Mapping it to "Reload" is to ensure that the audio plays from the start instead of starting mid interview. To enhance the immersivenenss of the experience, I also pieced together some ambience and SFX to recreate the sooundscape of the Rattan Artisan's working room. You can hear the sounds of rattan crackling and some scissors snipping, simulating the sounds of the artisan at work.
To be really honest, there was not much technical challenges faced. It was not that hard to piece together this experience as it was using tools in ways which I am familiar with. Perhaps this is not really a good thing because maybe I am not challenging myself enough?
Andreas pointed out that there is more potential within the touch interaction with the rattan board. I can look deeper into using the physical characteristics of the rattan, such as its textures, to create some unique visual or audio interactions with the image. Right now, it seems quite straightforward.
I do feel that it kind of hits the objective of the experiment, where I have successfully introduced tactile interactions within a audio-visual interface. However, Andreas feedback aligns with my initial gut feeling that the experiment is too straightforward and lack some unique form of visual/audio interaction effect. Personally I feel its kind of a "meh" experiment as it does not really generate a super unique experience. Perhaps this is the consequence of having "little to no technical challenges" doing this experiment. It was a indicator that it was too "simple"! But I still do feel its a good first step for now.
1. To explore and integrate more of the material texture to generate visual and audio change
2. To pay attention and understand the different textures that materials have
3. To challenge myself more (always good to be uncomfortable and grow!)
Group Consultation with Andreas
We had our first group consultations with Andreas this week. I consulted two main things: (1) My experiment 1 and (2) My new research direction. Above I have already discussed the feedback I received for my experiment so this section I will talk more about the feedback he gave for my new research direction.
Andreas stated that it is important for me to really establish a strong understanding of how we perceive the different sensory experiences when we interact with the world. This serves as a foundation when I move onto designing the interactions within the interface.
I should also seek to explore different and suitable technologies that compliments the materiality of the physical objects used. As in the case of the Rattan Board i used, maybe I can look into the use of conductive technology to allow more possible forms of interaction with the material.
Currently, I am looking specifically to base my interface design within a gallery space, since I am looking into augmenting visual arts with multisensory interfaces. However, Andreas encouraged that I keep my options open and not be constrained by the environments of a gallery space. This is to allow me to have more fun and opportunity to explore other possible applications.
Andreas also suggested that I look to define more specifically what kinds of audience engagement and meaning making I am looking into. Perhaps I can give further examples and define it clearer within my writing (RPO etc).
Definitely a informative feedback session. I understand that I need to pay more attention and study the materials that I am working with (I.E its textures). This can help me unlock different ways of interaction that leads to more unique outcomes. It was also liberating that I was advised to not be too fixated on the applications of the interface and I should instead focus primary on Multisensory Interface Design. This means more room for exploration and more fun to be had!