Polyrhythm Visualizations and Auralizations Using HTML5

Last updated on November 1, 2017

The implementation depends on HTML5 canvas and HTML5 audio, is experimental, and works best in Google Chrome (version 10.0 as of this writing).

The experiment, which uses HTML5 features.

Before the HTML version, I wrote a polyrhythm visualizer some time ago as a Java applet, when I was learning a number of the Chopin nocturnes. This was spurred on in particular by Op. 27, No. 2 in D-flat major. It’s in 6/8 time (i.e., two beats per measure), with the left hand playing six sixteenths per beat throughout. The ending involves two beats of: seven notes in the right hand and six notes in the left hand (a ratio of 7/6). Not only was the 7/6 a big challenge, but I started noticing patterns that called for some exploration.

Visualizations

There’s a pattern formed by which notes “fire” closest to each other in each hand.

Lines are drawn between top and bottom to emphasize when notes in the left and right hands are closest. The ebb and flow of the time distance itself has a pattern. In the example above, the left hand increasingly trails the right hand, until the midpoint, and then the left hand decreasingly trails the right hand, until they resynchronize.

One key for handling close ratios is that the midpoint involves an even trade-off between hands.

I didn’t want to draw too many lines between top and bottom, so the closeness visualization trumps the equidistant visualization.

Auralizations

After I rewrote my Java applet using HTML canvas, it seemed I should be able to bring it to life with HTML audio. Because of the precise timing required to render the audio, I wasn’t very optimistic about this in JavaScript, and it’s by no means perfect, but it was successful enough to go public.

Update: I’ve tried this on a few different systems now, and it functions horribly and unacceptably except on my development system. Google Chrome and IE9 RC work very accurately on my development system, I swear it. For now, the auralization is best described as experimental.

Timing in JavaScript, Part #1

To keep things simple and somewhat accurate, I wanted to use window.setInterval(...) rather than try to have sequences of window.setTimeout(...) daisy-chained together. I didn’t know what to expect across browsers. My conclusion is that timers in all browsers are very accurate, with Chrome being the most accurate. Chrome timers are least affected by CPU activity within the browser itself and other processes.

Timing in JavaScript, Part #2

The primary weakness bumped into seems to be that simply playing sequences of Audio elements is susceptible to random delays now and then. That said, Chrome is so reliable, it’s almost completely acceptable for the purpose here—essentially a metronome. IE9 RC is also very reliable. Firefox 3.6 is perhaps just under the threshold of acceptability. I found Opera to be too erratic.

Safari 5.0 on Windows delays the playing of all audio elements, thus the UI and the audio are totally out of sync.

Sequencing Audio elements

There are three things worth noting here:

1. I didn’t find any problems with playing multiple Audio elements simultaneously. The sounds played okay and blended okay.
2. The biggest hurdle was in realizing that I couldn’t get away with replaying the same Audio element each time it was needed. I needed to create pools of identical Audio elements and cycle through the pools.
3. I found that repeatedly calling play() on an Audio element sounded erratic, as if the sound got queued up to play but didn’t necessarily play immediately. In fact, I’d venture to say this is the primary weakness of all browsers. A big improvement here, at least for Chrome, was to call play() only when playing the sound for the first time, and using currentTime = 0.0 to play it again later.

To expound on #2: If you play with the demo, you’ll notice there are only three different sounds. I spent a lot of time trying to get three Audio elements to play and replay and blend acceptably. This was a losing battle. The result was almost random noise.

Rather than work with three Audio elements, I’ve created three pools of ten Audio elements. (Choosing ten was arbitrary; a much smaller number would probably work just as well.) For example, playing ten hits of the hi-hat has played ten instances of the same sound (and playing twenty hits has played each sound twice). Using this approach cleaned up the sound tremendously.

Timing in JavaScript, Part #3

As I’ve tried more browsers on more systems, I see that the audio performance varies wildly. It seems that performance is irreparably bad on old, slow hardware. But even on faster, newer hardware, performance varies a lot. I’ve implemented two different approaches to playing audio, and which approach is used can be selected at run time:

1. The default choice is to load the sounds once and replay them when needed. This seemed like the obvious approach to me, but this often results in random delays playing the sounds.
2. I’ve found that on some systems, performance is better if a new Audio element is created and loaded (and played) each time a sound is needed. (Note: only reloading the audio did not make a noticeable difference. Creating a new Audio element each time is what made the difference.)

Relevant Links

• 2011-03-09 The State of HTML5 Audio
  • Web Audio API
  • Audio Data API
• 2012-06-18 The HTML5 audio troubleshooting guide (this explains a lot)
• 2015-11-25
  • Haven’t looked at this in a couple of years, but it works much better now in all browsers!
  • To look at: https://goldfirestudios.com/howler-js-modern-web-audio-javascript-library