Category Archives: Creative Tech

Technology behind ‘The Ring’

This blog post will be dedicated to the hard ware part of the installation. The final execution differed from the intended form. I will be talking now about its intended form. The difference from the final executed version and reasons behind will be explained in another blogpost. Intended form of installation consists from:

  • 16 programable LED strips approx 190cm tall (107 programmable LEDs) positioned in the circle.
  • Pair of WiFi motion tracking gloves
  • Control Station
  • Pole with ToF sensor and Bluetooth for triggering the animation (or AV Game)
Programmable LED strip WS2812B – The building block of the visual interface for The
Ring

Gloves contain micro-controller ESP32, IMU sensor BNO055 for tracking the motion, DC-DC buck converter to bring small LiPo battery voltage of 3.7V down to desired 3.3V (operational voltage for both, ESP32 and BNO055). Raw x, y, z data from sensor are processed in micro-controllers and sent via WiFi on separate channels into Control Station.

Control Station

Control Station contains two ESP32 receivers for each glove. Data those are being split and sent into Teensy 4.0 containing logic which converts specific angles from the 360 degrees radius into specific MIDI note of pre-programmed scales, as well as one MIDI CC control. Each MIDI note covers 45 degrees of the circle. Teensy 4.0 can be directly connected with any DAW. Second avenue from the data split is into two Teensy 4.1 which controls LED interface. Each Teensy 4.1 is handling 8 strips. I have chosen this way because I found necessary to use library octows2811. This library enables very fast LED animations simultaneously in comparison to other libraries like FastLED or NeoPixel. I tried those too and found them very inefficient for real-time and fast applications. Control Station also contains a Bluetooth module which receives data from the pole’s ToF sensor.

Pole with ToF sensor and the roest of the Control Station

The Pole contains ToF (Time of Flight) sensor which essentially measures the proximity from and object. Distance data are being sent via Bluetooth module to the Control Station. In the installation it is used to trigger the initial animation and sound upon entering The Ring.

Inspiration for a New Project: A MIDI Theremin with Visual Feedback

After extensive work with LED strips, particularly the WS2812B, I found myself inspired to create something new while working on The Ring. One improvised idea that emerged during the development of the scales in The Ring was to design a Theremin-like MIDI controller. This device would allow the user to trigger MIDI notes from a pre-programmed scale without physical contact, relying on motion sensors and providing visual feedback through LEDs. The concept was a natural extension of my work, utilising my growing expertise in coding, microcontrollers, and sensor integration.

To bring this idea to life, I built the MIDI controller using two ultrasonic sensors (HC-SR04), a 60 cm programmable WS2812B LED strip, and an Arduino Micro. The Arduino Micro, equipped with the Atmega 32u4 chip, was particularly suitable for this project as it supports direct MIDI communication with DAWs (Digital Audio Workstations) and other MIDI-compatible instruments. This eliminated the need for additional hardware or software bridges, making the device streamlined and efficient.

I utilized the MIDIUSB and NeoPixel libraries in C++ to program the device. The ultrasonic sensors were configured to detect hand movements within a certain range, triggering MIDI notes based on the distance of the user’s hands from the sensors. Each sensor was assigned to a different musical scale, similarly like gloves in The Ring, creating a dual-layered experience. To add a layer of visual feedback, I programmed the LED strip to light up in distinct colors corresponding to each scale. This ensured that users could easily distinguish between the two scales, enhancing both the functionality and the aesthetic appeal of the device.

The result was a responsive and visually striking MIDI instrument that combined gesture-based control with dynamic lighting. The experience of using this MIDI Theremin went beyond sound; it became a multisensory interaction where movement, sound, and light converged seamlessly.

The MIDI Theremin was successfully performed during the Chronic Illness XXIII event, showcasing its potential in a live performance setting. Watching it in action during the event confirmed its versatility, not just as a standalone instrument but also as a tool for enhancing interactive installations or live sets. I definitely plan to incorporate this MIDI Theremin as a permanent feature in my setup for live musical performances.

Performing MIDI Theremine at Chronic Illness XXIII
Performing MIDI Theremine at Chronic Illness XXIII

Creating ‘AV’ Games

As I touched on briefly in my previous blog post, the Circle series is centered around the concept of audio-visual games, where a designated “conductor” takes control. Positioned within the circle and equipped with motion-tracking gloves, the conductor manipulates sound and visuals in real time, creating an immersive, interactive experience. The LED interface, consisting of 16 LED strips arranged in a ring, serves as the visual canvas for this dynamic interplay.

Building on my prior experience with creating interactive gloves and using motion to control sound, I feel confident in generating and manipulating audio elements through hand gestures. This familiarity has allowed me to focus more intently on exploring and refining the visual components of the installation. My goal is to design an engaging and intuitive system where light and sound not only complement but also amplify each other.

The “Entry Game”: A Gateway to Interaction

The first element I’ve programmed for The Circle series is the “Entry Game.” This game is designed to trigger automatically as the conductor steps into the circle. The concept behind the Entry Game is to provide an immediate, engaging introduction to the system. Upon entry, the motion-tracking gloves activate a sequence of lights on the LED strips, signaling that the conductor has entered a new interactive domain. This game acts as a gateway, setting the stage for deeper levels of interaction while ensuring the conductor feels immersed from the outset.

“Digital Hula Hoop”: A Work in Progress

Another game currently in development is the “Digital Hula Hoop.” This element focuses on creating a visual and sonic interplay that responds dynamically to the conductor’s movements. The idea is to program two light circles in different colors, representing the conductor’s hands. These circles will move and tilt within the LED ring based on the motion data captured by the gloves.

At this stage, the animation for the Digital Hula Hoop is automated and does not yet include sound integration. However, the visual elements are being refined to ensure smooth and intuitive responsiveness. The next step involves linking the motion-tracking data to control the position and orientation of the light circles dynamically.

On the auditory side, I envision pairing the light movements with evolving drone sounds. The amplitude and distortion of the sound would change in response to swift horizontal hand movements, creating a sense of energy and tension. Additionally, vertical hand motions could modulate the pitch, adding depth and variety to the soundscape. The ultimate goal is to achieve seamless synchronization between sound and visuals, where each gesture transforms the conductor into a performer and the LED ring into a living, reactive instrument.

I added dramatic sound design which is suppose to evoke entering to the cybernetic liminal space.

Arrangement of samples in Ableton Live – 8 drum hits slowly panning to the left corresponds with the animation movement of 8 LED strips with the ‘shrink’ drone in the end.
Test of the ‘Entry and the shrink drone in limited light sequence 1, 2 and 8 only – Amount of lights is this time limited due space restrictions in the studio.

While the The Ring series is still in its early stages, the progress so far has been exciting and illuminating. The combination of intuitive hand-controlled soundscapes and visually dynamic LED animations offers immense creative potential. Moving forward, I aim to refine the interaction mechanics, ensuring that the system is not only responsive but also rewarding for both the conductor and the audience. Each game in the series will build on the others, gradually increasing in complexity and encouraging deeper engagement with the installation.

Automated Digital ‘Hula Hoop’ animation on LED interface recently expanded in length.

Interactive Audio-Visual Installation ‘ The Ring’

I have decided to incorporate my Sonokinetic Arduino gloves to complement the programmable LED strips in The Ring. This marks an initial step toward a broader interactive audio-visual installation.

The Ring explores the convergence of audience and artist roles within club culture through sonic and visual mediums. It examines how the commercialization of DJ culture and the rise of social media impact immersive experiences in nightclubs, where the focus often shifts from artistry to consumerism. The installation also considers whether the constant evolution of digital technologies might provide an alternative solution through the creation of hybrid art forms.

The piece aims to democratize and decolonize club spaces by encouraging direct audience participation, disrupting the traditional dynamic where the DJ serves as the central focus while the crowd remains passive. Instead, The Ring invites attendees to actively engage, turning them into co-creators of the experience.

Drawing on Haraway’s (Haraway, 2016) cyborgian narrative, The Ring integrates the communal aspects of a club night, affective immersion, and the blurring of boundaries between artist, artwork, and audience. It creates a semi-virtual space where sonic, visual, and social elements converge into a hybrid form, challenging conventional distinctions and offering a reimagined experience of club culture.

Bibliography:

Donna Jeanne Haraway (2016). Manifestly Haraway. University of Minnesota Press.

Commission for motion tracking gloves

The inspiration for The Ring began in an unexpected and somewhat serendipitous way. I came across a striking visual of an unknown installation while scrolling through social media. The image captured my imagination, and I immediately thought, “I’d like to replicate something like this.” What began as a visual exercise—a simple attempt to recreate the aesthetic appeal of the installation—soon evolved into a much more ambitious project. As I delved deeper, I realized the potential to expand the concept by incorporating sonic and interactive elements. These additions aligned with my broader interest in creating immersive, multisensory experiences.

The idea began to take form after a shift in priorities around a separate commissioned piece I was working on at the time. Suddenly free to explore my own creative directions, I decided to use this opportunity to build upon the initial inspiration. What started as a purely visual experiment grew into an exploration of audience interaction, embodiment, and the integration of sound and motion.

In September, my friend Matteo Chiarenza Santini approached me with an intriguing request. Matteo was collaborating on a live performance for FKA Twigs and had been tasked with sourcing a pair of simple, interactive motion-tracking gloves for the performance. He reached out to me, asking if I could create a prototype that would meet the technical requirements.

Excited by the challenge, I began working on the gloves. Using my experience with Arduino and similar technologies, I designed a simplified version of an earlier prototype. The gloves featured BNO055 IMU sensors for precise motion tracking and ESP32 microcontrollers for data collection and Wi-Fi transmission. Each glove was capable of sending raw x, y, z axis motion data to a Teensy board, which interfaced with Max For Live, enabling users to control parameters in real-time. Additionally, the gloves supported direct MIDI communication, making them compatible with Ableton Live and other DAWs.

Although the gloves were completed, they were ultimately not used in FKA Twigs’ performance. Initially, this was disappointing. However, the experience of building the gloves and the creative potential they represented sparked a new wave of ideas for me. What if these gloves became the foundation for the interactive installation I had been contemplating? The thought of integrating motion-tracking gloves into an installation seemed like the perfect opportunity to explore the interplay between movement, sound, and interactivity on a deeper level.

This unexpected twist marked a turning point in the development of The Ring. The gloves became the starting point for an installation that not only reflected my fascination with visual aesthetics but also pushed me to explore how movement could shape soundscapes and create immersive environments. What began as a technical experiment transformed into a project driven by the potential to blur boundaries between performer, audience, and artwork.

Custom LED system for music performance of Ona Tzar

One idea which emerged from my growing interest in creative technology was creating custom light system which could ad to a live music performance strong visual aspect.

Based on my previous work with programable LED strips and micro-controllers I have made MIDI reactive strips. Strips are circuit-bended floor lamps. Lamps themselves initially contained a chip for automated animations however I found them not very attractive as well as audio reactivity was quite basic. The design on lamps, portability and fact that they contain easily programmable LED lights inspired me to create something new. I have removed the chip and replaced it with 3.5mm female jack so the lamp becomes a part of a “screen” of 7 lamps.

Separate lamps are connected to separate outputs from Arduino Leonardo (contains chip Atmega 32u4 which allows direct MIDI control). Arduino Leonardo is programmed to receive MIDI notes from a DAW or MIDI instrument. Each lamp is a single MIDI note on a scale from 0 to 127. A single MIDI note can contain different colour, different pattern or animation. With 7 lamps we then have possibility of 18 unique series of animated colour patterns on a single MIDI channel. If we need more we can simply program more animations on different MIDI channel (16 in total, so in this case we can get 16 x 18 = 288 variation)

Light patterns can be either played live from MIDI controllers via Ableton Live sending MIDI note messages to the Arduino Leonardo or hand-written in piano roll.

For the live performance of Ona tzar we have decided to create hand-written piano roll MIDI clips so they can be as live light loops together with audio clips.

Light pattern hand written in piano roll – allows the exact timing and respons to the music or sound adequately.
Test of various animations with Ona Tzar’s single Hypnagogia
Test of various animations with Ona Tzar’s single Hypnagogia with aditional strobe lights connected to relays triggered also by MIDI notes
Final video of Ona Tzar’s live performance of Hypnagogia. The first video from coming up triptych of live performances .