«That’s kind of the amazing part about video mapping because when you don’t know what is entirely possible, then anything is! »
Recently, the team has been very impressed by the work of Jesse James Allen, a one-of-a-kind projection artist who combines HeavyM with other software to create sensational projects. We had a great talk with Jesse, who defines himself as a Renaissance Man because he has always been this very artistically curious person who loves to learn from many disciplines.
Jesse James Allen – a portrait in front of his last work
Jesse entered the entertainment industry in the early 90’s, kind of the dawn of the digital age. Through the years, he has acquired a wide set of skills ranging from Record Mastering Engineer to Media Creator for Theme Parks and tried his hand at many different tools. Jesse considers mediums as pieces of the puzzle to tell better stories along the way, to give more emotionally impactful experiences. He has been part of 53 video games that have really shaped his career as a creative and we can see how the video gaming influences in some of his projection mapping projects. He sometimes uses gamepads to control the projection:
Using a PS4 controller to play with the projection
One step torward interactivity with HeavyM
Jesse started getting curious about projection mapping, he stumbled upon projection-mapping.org and tried out the various programs they had listed. HeavyM really resonated with him as an intuitive and powerful program to work with. He has been doing professional sound design for over 20 years so that skill, combined with all the visual creation tools of HeavyM, allows him to come up with some unique experiences.
Jesse experiments the software to push the creation further for a lot of personal projects, driven by the idea to develop the accessibility to enter a virtual world around physical objects for everybody. He particularly appreciates the ability of HeavyM Live to accept MIDI, OSC, and Spout&Syphon protocols, with which it creates 100% interactive applications with the addition of sensors.
In this example, Jesse created particles with the Unity graphics engine. The movements of his hand are tracked through a Leap motion sensor and control the particles. All this is sent to HeavyM Live via Spout.
See the full video here.
« I have been doing research and development with HeavyM for about a year now and I feel like I am just scratching the surface of what is possible ».
Jesse sees a tremendous amount of potential for HeavyM in interactive experiences and art exhibits. He is working with traditional artists such as painters and illustrators, as they bring the tactile physical world into the mix. To him, it just makes the end result feel more organic. According to Jesse: « There is something about mixing physical and digital art together this way that creates a special kind of magic ».
His collaboration with Charlie Jicha resulted in a picture painted with conductive ink. This ink is connected to an electronic board (Touch Board) which allows, for example, to convert the information into MIDI for HeavyM. The HeavyM sequencer is controllable in MIDI, so Jesse has linked parts of the painting to particular sequences. The result is stunning!
Interactive mapping with conductive ink, Jesse James Allen & Charlie Chicha
When prototypes become public works of art
On the basis of this experience, Jesse James Allen has imagined and built an 8ft tall Seraph Angel sculpture, showcased on October 19th at Immerse, a massive Orlando arts festival hosted by Creative City Project.
“The Seraph” combines projection mapping, gestures, touch and sound as an interactive experience. This is the largest HeavyM project that he has done to date, allowing all the viewers to interact with the installation with their bodies and voices.
First, most of the projected visuals are ISF shaders (new features included in HeavyM 1.10). In addition, the visitor can act on the projection thanks to the movement recognition via leap motion, to the touch interaction with the conductive ink. Finally, the sound analysis module of HeavyM was used to take into account the sound of its voice visitors via a microphone.
Video teaser of the “Seraph” at Immerse Festival
Jesse is a member of Falcon’s Creative Group, made up of ambitious artistic people such as award-winning artists, engineers, architects, designers, writers, and feature-film level filmmakers. This company focuses on creating and making immersive entertainment experiences worldwide, for destination resorts, theme parks, museums, and attractions.
With the illustrator Bruce Sereta, Jesse got to talking about augmenting a physical mural with projection mapping. Falcon’s Creative Group hosted a Themed Entertainment Party and they decided to make it happen. While Bruce illustrated all the line work physically on a 4ft mural, Jesse augmented it with HeavyM: he had to use every available layer in the software and to really get creative with the layout and functions of each layer: « Kind of a trial by fire but that is how you really get to learn what a program can do! »
Mermaid, a project only done with HeavyM, by Jesse James Allen and Bruce Sereta
With the artist Chaya, Jesse went further in augmented reality by combining HeavyM Live with SMode and Electrifly AR. A truly impressive XR experience, celebrating the opening of Orlando Game Space.
Next projects, ideas, and work in progress
- The pursuit of interactive experiments including mapping moving surfaces and fabrics,
- New collaborations,
- Possibilities of working with performance artists.
« I am very happy to work with tools like HeavyM. Thank you for creating them! »
Get inspired and follow Jesse James Allen on Vimeo.
Jesse James uses all the power of HeavyM Live including MIDI, OSC protocols and video streams exchanges by Spout&Syphon.