Week 0
PROLOGUE
Getting back into the Groove

Coming back from a 3 month break where I have worked tirelessly at an advertising agency, focusing mainly on content creation through Visual Design and Photography, I have to adapt back my focus onto my research topic which is about a different field of expertise. Carrying forward the topic from my year 2 research log, I will be revisiting the Design of Multisensory Interfaces and its impact on Human Emotions and Behavior.

The main themes in current research topic

Recap on Research Topic

My research topic is about understanding how Sensorial Principles, such as Congruency of Sensory stimuli affect Human Emotions and Behavior towards Multisensory Interfaces and Experiences. Through this understanding, I hope to leverage sensorial principles in the design of Multisensory Interfaces to create an emotionally engaging interface that encourages enjoyable interactions with machines.

  • Multisensory Interface Example 1:
    Nutrigrade Interactive Vending Machine by HPB, users use body motion to catch falling cans of drinks.
    Watch video demo

  • Multisensory Interface Example 2:
    Rain Room by Random International, participants walk through a raining room without getting wet.
    Watch video demo

HOW DO THESE MULTISENSORY INTERFACES CREATE
EMOTIONALLY ENGAGING EXPERIENCES?
PAUSE.
Let's understand some Keywords

I suspect I would lose the reader's understanding at complicated terms like "Congruency of Stimuli" and "Sensorial Principles", so let's establish some key terms so its easier to follow along.

Keyword - Sensorial Principles

Sensorial Principles are guiding concepts that explain how the body interacts with and absorbs sensory experiences. Understanding these principles is crucial for informing effective design interventions within a Multisensory Interface.

Example - Congruency of Stimuli

Let's look at the principle of Congruency of Stimuli as a example. The principle states two key points as follows.

1. Multisensory stimuli are perceived cohesively, not just a sum of its individual components. 2. Hence, each stimuli has the ability to influence one another positively or negatively, affecting the cohesiveness of the multisensory experience.

To summarise, due to our embodied perception of multisensory experiences, different stimuli affect one another. In design, this means individual sensorial interventions must work cohesively to ensure a pleasant user experience.

WHERE DID I GET THESE INFORMATION ABOUT
SENSORIAL PRINCIPLES?
Reading 1 - Emotional Responses to Multisensory...
What is the book about?

Emotional Responses to Multisensory Stimuli is a research article that looks into how humans perceive multi-sensorial experiences and how they influence our cognition and emotional responses. The reading also discusses key sensorial principles, such as Congruency of Stimuli, to underscore the importance of the interconnected nature of multisensory experiences.

Some key learning points 1. The Inter-connected nature of multisensory experiences

Multisensory experiences are not just a sum of its individual sensorial components, but a complicated, interweaved relationship between all the involved sensory stimuli. Each stimuli have the ability to influence each other. A example given from the book discusses how when multiple audio beeps play alongside a single visual flash, it can give the illusion of multiple flashes happening even though there is only one.

2. Congruency of Stimuli's impact on our emotions and behavior

Congruency of Stimuli can have a big impact on how we perceive multisensory experiences. When stimuli is congruent (working together harmoniously), it is more likely to encourage positive reception and behavior. In contrast, while incongruent stimuli might be more memorable, it results in more negative feelings towards the experience.

Reading 1: Emotional Responses to Multisensory Stimuli
by Schreuder et al.

CASE STUDY 01
Rain Room What is the project about?

Rain Room is a immersive experience where participants get to walk in a perpetually raining room without getting wet. Through using motion sensing technology, the rain stops whereever human motion is detected, creating the impossible ability to "control" rain.

Some key learning points 1. The Magic of Multisensory Interfaces

One of the biggest compelling factors of "Rain Room" was the magical ability to be able to walk freely in the rain without getting wet. This case study highlighted the capabilities of Multisensory Interfaces in creating the impossible through the power of technology and the inclusion of embodied technological interactions.

2. Congruency of the Multisensory Experience

Relating to the reading above, it is a clear demonstration of the sensorial principle of Congruency. The sounds, visuals and smell of the multisensory experience all align with our natural understanding of rain. Thus, making the overall experience very realistic, yet magical due to how we never get wet.

3. Exploring Tangible Output and Input

From many of the Human Computer Interactions I observe, most of them involves having a screen as a source of input/output due to practicality. However, Rain Room fully relies on physical input (body movements) and generates a physical experience (walking in the rain). This was what I felt contributed heavily to its magical experience.

WHY DID I CHOOSE TO WORK ON THIS
DISSERTATION TOPIC?

  • Arduino Wishing Well by Wen Soon and Janessa
    Watch video here

  • Audio Reactive Thunderstorm by Chiang Wen Soon

  • Natural Forms with Code by Chiang Wen Soon

A Developed Interest in Creating Coding

Throughout my creative coding learning journey in year 1 and 2, I have managed to learn more about creative applications such as p5js/TouchDesigner/Arduino and managed to create rewarding and fun outcomes with it (Arduino Wishing Well, Audio Reactive visuals etc). This led me to understand and appreciate the amazing capabilities of creative coding and how it can realize ideas which transcends concerns of practicality, ultimately motivating me to study deeper into it's craft.

One of my favourite creative coding projects done in Lasalle

Survival Strategies
Chiang Wen Soon, Janessa Aw, Syafiq Shukor

Signify is Singapore's archive of the nation's rich biodiversity history, with thousands of detailed images of the country's unique specimens. To extend the outreach of Signify beyond enthusiasts of biodiversity, my team and I developed a interactive exhibition through TouchDesigner making use of simple gestures to create a interactive experience to rope people into the wonders of biodiversity. The experience centers around 3 different themes: Camouflage, Mimicry and Aposematism. These are techniques which animals and insects use to escape predators and survive out in the wild.

  • Tree Colored Wings to Tree Bark

  • White Leaf Pattern to Cabbage

  • Eyespot Pattern to Owl

The Art of Mimicry
My portion of the project

Some animals and plants have evolved to mimic the appearance of other species or objects to deceive predators or prey. This can make them seem more dangerous than they really are, or allow them to blend in with their environment by resembling something unthreatening.

The Technology

By integrating both MediaPipe (Interactive Plugin) and StreamDiffusion (AI Generation) into TouchDesigner, I was able to use hand gestures to activate the prompt intensity from StreamDiffusion. Pinching collects the particles to form a image, while sliding adjust the AI prompt's intensity. To further strengthen the visual outcome, the generated image was then put through particle systems, where it takes on a particle look which also helps to smooth out the framerate from the image generation.

Eyespot Pattern to Owl

I WANT TO USE CREATIVE TECHNOLOGY TO
MAKE MAGIC FOR PEOPLE