Harmony Ray: A MIDI Visualization Pet Project

Skip to the bottom

  • @kable  Jan 29

    Hello everyone!

    Let's jump straight to the point.



    I have been programming a MIDI visualization tool that I will be giving a test run for the first time this FAWM. The tool is called Harmony Ray, built from the ground up in C++, and it allows me to design abstract 2D visuals from MIDI input. This idea has been a passion project of mine for quite some time now, and I've been adding to it on-and-off over the course of a year, though realistically, the development time strung together in a continuous sequence probably lasts about a month or two.

    Some early footage I captured of it running many months ago:
    https://www.youtube.com/watch?v=pDXWrlXdDP8
    https://www.youtube.com/watch?v=DbCy2lW5zCU

  • @kable  Jan 29

    Let's talk about what you can do with this tool in its current state:

    1. Import mono or stereo WAV / OGG files of just about any sample rate or bit depth you'd like, and then play them back through my custom (so-so) linear sample rate converter at a lovely 48kHz.

    2. Choose from the three implemented 'instruments', which visualize the MIDI information in different ways, and tweak their MIDI ranges individually (shown in this brief video at https://www.youtube.com/watch?v=40mN6JZdkIg), as well as their visual properties (like color). The names of the instruments added so far are Rhombify, Tiler, and Verticallity.

    3. Drag and drop MIDI clips into the sequencer, to which the instruments will then react.

  • @kable  Jan 29

    The most recent thing I got working was getting the visuals to adjust based on the pitch bend wheel (which you can check out at https://youtu.be/BNIzgPE2abQ). Over this weekend, I'll be making the sequencer tool more useful, and adding more instruments. Hopefully I can do it all in time . . . ?

    Anyway, I'll be adding to this thread with any noteworthy updates on the state of the tool. At the moment, this program is so terrible that no one but me should have to bear the pain of using it, but if you have music that relies a lot on MIDI (like I do), let me know and I'll try and send you a copy when it doesn't suck as much.

  • @dzdandcunfsd  Jan 29

    @kable that's actually really cool! Best of luck polishing it up! I don't do much with MIDI, but looks like something I would use if I did at least to play with from time to time, about as far as I've gotten digging into Jeskola Buzz which I keep meaning to do more with.

  • @tesla3090  Jan 29

    This is really cool! I'd be interested in checking it out once you've got a release!

  • @pipewrench67  Jan 29

    Very cool, kind of reminds me of Rhythmus 21
    https://www.youtube.com/watch?v=FYPb8uIQENs

  • @yam655  Jan 29

    @kable That looks really great! Quality music visualization can be hard to find, and integrating it with MIDI is something that I've been wanting for years.

    Do you have lyric support on your list of potential features? Like, a MIDI line for a singer that has all the timing information, coupled with the text lyrics that are displayed? (Lyrics either in the MIDI itself or -- my preference -- as plain-text with simple formatting to help with note alignment, but could be later be extended.)

  • @kable  Jan 29

    @yam655 ah, I have considered adding text features at some point. Totally doable. It would probably just be its own dedicated lane in the sequencer.

    @pipewrench67 that's a great reference video, and that was 1921??? Just goes to show how someone's always had your idea before you haha!

  • @pipewrench67  Jan 29

    @kable Different method (I'm fairly certain).

  • @kable  Jan 29

    @pipewrench67 right, it looks like some kind of stop motion, possibly claymation? Although my technique is different, the end result is yet more abstract shapes moving to music haha!

  • @kable  Jan 29

    Oh, I forgot to mention Wayne Lytle, who made a tool that could do this with 3D instruments in the early 2000's, and called the project Animusic. He is one of my biggest inspirations to make MIDI-powered software.

  • @yam655  Jan 29

    @kable If it's going to be public software, it's something that could be supplied via a patch at some point. Let me know if you decide to open-source it.

  • @kable  Jan 29

    @yam655 totally possible at some point. It will be cool once I get past the "would have to pay you to use this" leg of the development journey. Who knows?

  • @kable  Jan 31

    === UPDATE THE FIRST ===



    See anything different?

    Yeah, neither do I. (that is, except for the tiny all-red icon in the top left corner)

    I spent all of today, on-and-off, swapping out the graphics back-end for this tool. Before today, everything ran on an extremely flimsy Vulkan / GLFW graphics framework, but now, everything has been moved to a more basic, robust (and easier to use) OpenGL / SDL framework. Vulkan was a mistake.

    With this new framework, I found a way I can program the tool to export image sequences to be combined into a video. This is a huge feature that was previously missing; I don't have to use a capture card to manually capture the animations anymore!

    With all of that cleanup, I'm ready to make the final main improvement I need before FAWM begins: assign MIDI clips to specific instruments.

    That's all for now.

  • @kable  Feb 1

    === UPDATE THE SECOND ===

    Took me a few hours, but video exporting finally works. One click of a button, and I have an .MP4 file. Magical.

    https://www.youtube.com/watch?v=1RMvpz7tntQ

  • @swampjaw  Feb 1

    Sort of in the same universe -- I've got an app on the app store that will trigger video with MIDI. Load in a video, set cue points, and then MIDI notes will seek to the different cue points. MIDI CC values can be used to rotate, scale, etc..

    The general idea is to be able to sequence video in time with music. I've got some more things that'll go into the app when I get more done.

    The app is free for iOS, and available now -- just good clean fun.
    https://apps.apple.com/us/app/visual-synth/id1493237895

    Some users have posted a few things:
    https://www.youtube.com/watch?v=T36GEGeNiDU&feature=emb_logo

    https://www.youtube.com/watch?v=v-0guJq_z8M&feature=emb_logo

  • @apertome  Feb 1

    Cool stuff, @kable ! Does this tool support MIDI CCs? I sometimes use stupidly-high-res CCs to glitch things out.; wonder what some of those would look like in a tool like this.

  • @kable  Feb 1

    @apertome I'll try to support any CC's you can export directly to a MIDI clip (no 'online' processing for now, sorry!), and MPE is also something I'll have to eventually consider, though it looks particularly daunting. Based on how the tool currently still has artifacts with how it interpolates the pitch bend wheel, I don't think I'm ready for any more MIDI controls yet, not before I fix that..

  • @kable  Feb 1

    @swampjaw ah nice, this is a workable approach for VJ'ing. Usually I've seen video manipulated manually or based on audio peaks / spectrum information, so congrats on pushing MIDI into it! I'd download your app, but I only have an Android phone, sorry. How are you streaming the video? As in, any helpful libraries for the video codecs?

    Video support was not on the to-do list, but never say never I guess. Well, in this case, I'll say never for this month haha! Solid color backgrounds on the other hand, or static image backgrounds for that matter . . . still on the menu!

  • @swampjaw  Feb 1

    @kable iOS has a bunch of widgets for displaying and controlling video - instantiate an object on the screen, call the method to load a video, and then wait for MIDI -- telling the widget to seek to a new location when a particular note comes in (or scaling/rotating on CC). The code for the app is absurdly short. I do most of my work on iOS/Mac; the APIs and tools are a ton of fun to play with.

    Are you working in Windows? Android? Linux?

  • @kable  Feb 1

    @swampjaw gotcha. Very resourceful, thanks for the background! It's been forever since I developed anything on iOS, and I wasn't ever particularly good at it.

    Ok, so for this tool, I'm developing / deploying on Windows, but all my libraries should port to OSX / Linux as well. OSX has me a little worried because Apple has a pretty aggressive push towards their 'Metal' graphics API which I will most certainly not use and might prevent this app from ever touching a Mac.

  • @kable  Feb 5

    === UPDATE THE THIRD ===

    Here's the first full animation ever produced with Harmony Ray!

    https://youtu.be/qCW0mRUVSC0

    Tagging @postcardhelicopters, @apertome, @swampjaw, @yam655, @tesla3090, and @pipewrench67.

  • @yam655  Feb 5

    @kable That's looking really great. Please let me know when you have a version I can play with.

  • @kable  Feb 7

    === MINI UPDATE ===

    I just got the serialization to work again. This means that the tool can now save / load projects. I'll try and post a demo later if there's time. This is the last day I can make meaningful changes before I have to just focus on making music.

  • @kable  Feb 11

    @postcardhelicopters will do, and yes, there might be big changes by next year, too!

    This year, I had the epiphany to mix the outputs of my tool with another, called Magic Music Visuals (https://magicmusicvisuals.com). It's a node-based VJ tool to synchronize video effects with audio. Still no good MIDI support, but I can supplement that part!

  • @kable  3 weeks

    I've continued to make good progress on the tool, but I have to wrap up the updates this year. Tune in next year where, in theory, this is a more fully-fledged program for making cool-looking visuals!

  • @tcelliott  3 weeks

    I'll be looking forward to seeing what you have next year then. Missed this earlier, but it's cool.

  • @gm7  3 weeks

    Super cool...
    Well done and a great idea .

Leave a Message. Log in to FAWM or sign up first...