KICKOFF
Getting Started with Making
Andreas encouraged us to begin the process of experimenting and making as soon as possible for us to get back into the mentality of making. While initially I was struggling for ideas, I decided to start by looking at the tools that I have and see what I can do with it. That tool happened to be a camera.
A Busker along Orchard Rd street
Photographing the Buskers
Going on a Photowalk
I grabbed my camera and headed out to snap some photographs to be used for my experiment. I did not really have a theme at that time but chose Orchard Road as a location. I just wanted to go with the flow and just have fun capturing photos. As I walked, the sounds and visuals of busker performances caught my attention. With that, I found a interesting subject for my photographs.
Er Hu - Uncle Jing
-
Puppet Show - Mei Lee
-
Guitar + Song - Raymond
The Final Portrait Shots
I thoroughly enjoyed the process of taking these photographs. There's something meditative about being present in the moment and observing life unfold. I was pleased with the shots I captured as they managed to freeze the buskers in motion, whether playing guitar or preparing for their puppet show. Overall, they encapsulated small stories of their busking performances and environments, which was exactly what I aimed for on this photowalk: to capture mini narratives.
The Soundscape
Other than taking pictures, I also recorded the soundscape with my Iphone of the Er Hu and Guitar
buskers'
performances, less the puppet show as I made the mistake of recording on earbuds (bad quality). I wanted
to
add more elements in the soundscape for me to play with, so I AI generated voices (for Er Hu uncle) and
enhanced the Er Hu
instrument’s audio with post editing.
Press the buttons below to hear the audio and press
again to stop.
Throughout the CPJ, I will be adopting the Experiment Log approach that is conventially used in a Critical Journalling (CJ) since CJ is my overarching research framework. I also feel that the template of Experiment Logs helps to keep my analysis of my experiments clear and reflective.
To explore the addition of sensory modalities to photographs, by starting with and focusing on the addition of the sense of hearing through adding soundscapes.
An audio reactive photograph that has boxes with the object/person’s name popping up when the said objects/person are audible in the soundscape.
-
Experiment 1a - Guitar Busker Raymond
-
Experiment 1b - Er Hu Busker Uncle Jing
Boxes drawn on Illustrator
Creating the boxes was straightforward. I drawn boxes that covers the elements which I want to highlight and coupled it with a text label. I used the image as a reference throughout this process to gauge the size of the boxes.
This was slightly tricky. I loaded the soundscape in and had to listen to it intensely while observing the soundwaves. I had to identify which portions of the soundscape affects which type of sound frequency. For example in experiment 1b, I observed that rhythm responds to the road traffic, while mids responds to Er Hu's music.
After identifying the frequencies, I used a select chop and mapped the values with a math chop to a usable range of values for me to work with.
I mapped the respective frequency values to the relevant boxes by referencing to the brightness of the boxes (with level nodes). Thus, the boxes will "appear" when their coupled sound is heard in the soundscape.
Issue: High frequency data picks up voice too. Audio on to hear.
The success of this method of creating audio-reactive visuals relies heavily on having one sound element mapped to only one type of frequency data (Traffic to Rhythm etc). However, I faced a issue when I tried to isolate the sound of a coin drop (for busker's donation box). While I found out that the High frequency range reacts to the coin drop, It also reacts to the voice of the Er Hu uncle. This present the issue when the uncle speaks, the coin box also was highlighted (which is wrong). Thus, I took out the sound of the coin drop from the soundscape as I could not come up with a solution despite messing with the threshold parameter.
While I faced with the fustrating issue mentioned above, I do like the visual outcome of the experiment. It was a good first try at making a visual photograph multisensory by adding a soundscape to it. The overlaying boxes gave it a unique visual twist, but not too distracting to take away from the photograph. On a critical note, I feel that the outcome is quite straightforward as it seems to fulfill more a more functional purpose (highlighting elements that is making the audio).
Extracting from the feedback in week 2, Andreas did highlight that perhaps I could explore the possibility of using the visuals to trigger the sound instead. Meaning, using the image/pop-up boxes to transform the soundscape. The outcome was also too literal, where it just simply highlighted what object was making the sound (similar to what I reflected on).
1. To explore the use of visuals as the catalyst of sound.
2. To explore less literal outcomes, more creative visual/audio intepretations
3. Figure out a more efficient way to create accurate, audio-reactive effects