Polyrhythm Visualizations and Auralizations Using HTML5

The implementation depends on HTML5 canvas and HTML5 audio, is experimental, and works best in Google Chrome (version 10.0 as of this writing).

The experiment, which uses HTML5 features.

Before the HTML version, I wrote a polyrhythm visualizer some time ago as a Java applet, when I was learning a number of the Chopin nocturnes. This was spurred on in particular by Op. 27, No. 2 in D-flat major. It’s in 6/8 time (i.e., two beats per measure), with the left hand playing six sixteenths per beat throughout. The ending involves two beats of: seven notes in the right hand and six notes in the left hand (a ratio of 7/6). Not only was the 7/6 a big challenge, but I started noticing patterns that called for some exploration.

Visualizations

There’s a pattern formed by which notes “fire” closest to each other in each hand.

14 against 5

Lines are drawn between top and bottom to emphasize when notes in the left and right hands are closest. The ebb and flow of the time distance itself has a pattern. In the example above, the left hand increasingly trails the right hand, until the midpoint, and then the left hand decreasingly trails the right hand, until they resynchronize.

One key for handling close ratios is that the midpoint involves an even trade-off between hands.

7 against 6

I didn’t want to draw too many lines between top and bottom, so the closeness visualization trumps the equidistant visualization.

5 against 6

Auralizations

After I rewrote my Java applet using HTML canvas, it seemed I should be able to bring it to life with HTML audio. Because of the precise timing required to render the audio, I wasn’t very optimistic about this in JavaScript, and it’s by no means perfect, but it was successful enough to go public.

Update: I’ve tried this on a few different systems now, and it functions horribly and unacceptably except on my development system. Google Chrome and IE9 RC work very accurately on my development system, I swear it. For now, the auralization is best described as experimental.

Timing in JavaScript, Part #1

To keep things simple and somewhat accurate, I wanted to use window.setInterval(...) rather than try to have sequences of window.setTimeout(...) daisy-chained together. I didn’t know what to expect across browsers. My conclusion is that timers in all browsers are very accurate, with Chrome being the most accurate. Chrome timers are least affected by CPU activity within the browser itself and other processes.

Timing in JavaScript, Part #2

The primary weakness bumped into seems to be that simply playing sequences of Audio elements is susceptible to random delays now and then. That said, Chrome is so reliable, it’s almost completely acceptable for the purpose here—essentially a metronome. IE9 RC is also very reliable. Firefox 3.6 is perhaps just under the threshold of acceptability. I found Opera to be too erratic.

Safari 5.0 on Windows delays the playing of all audio elements, thus the UI and the audio are totally out of sync.

Sequencing Audio elements

There are three things worth noting here:

  1. I didn’t find any problems with playing multiple Audio elements simultaneously. The sounds played okay and blended okay.
  2. The biggest hurdle was in realizing that I couldn’t get away with replaying the same Audio element each time it was needed. I needed to create pools of identical Audio elements and cycle through the pools.
  3. I found that repeatedly calling play() on an Audio element sounded erratic, as if the sound got queued up to play but didn’t necessarily play immediately. In fact, I’d venture to say this is the primary weakness of all browsers. A big improvement here, at least for Chrome, was to call play() only when playing the sound for the first time, and using currentTime = 0.0 to play it again later.

To expound on #2: If you play with the demo, you’ll notice there are only three different sounds. I spent a lot of time trying to get three Audio elements to play and replay and blend acceptably. This was a losing battle. The result was almost random noise.

Rather than work with three Audio elements, I’ve created three pools of ten Audio elements. (Choosing ten was arbitrary; a much smaller number would probably work just as well.) For example, playing ten hits of the hi-hat has played ten instances of the same sound (and playing twenty hits has played each sound twice). Using this approach cleaned up the sound tremendously.

Timing in JavaScript, Part #3

As I’ve tried more browsers on more systems, I see that the audio performance varies wildly. It seems that performance is irreparably bad on old, slow hardware. But even on faster, newer hardware, performance varies a lot. I’ve implemented two different approaches to playing audio, and which approach is used can be selected at run time:

  1. The default choice is to load the sounds once and replay them when needed. This seemed like the obvious approach to me, but this often results in random delays playing the sounds.
  2. I’ve found that on some systems, performance is better if a new Audio element is created and loaded (and played) each time a sound is needed. (Note: only reloading the audio did not make a noticeable difference. Creating a new Audio element each time is what made the difference.)

Relevant Links

The Great void of JavaScript

In JavaScript, you often see code like this:

// 1

(function() {
    a bunch of code;
})();

The code in the function runs immediately and allows you to avoid namespace collisions with other code. If the code includes (inner) functions, those functions are essentially private. In fact, they will go away if there are no references to them when the code completes.

People seem to prefer the above syntax, though the code is equivalent to:

// 2

(function () {
    a bunch of code;
}());

When I first felt the need to do this sort of thing, I tried the more intuitive syntax:

// 3

function () {
    a bunch of code;
}();

That is what one would expect to work. It makes intuitive and syntactic sense. It clearly creates an anonymous function and then invokes it. However, it causes a syntax error because, according to here, a statement that begins with the function token must be a function declaration (as opposed to an invokable function expression).

Noting first that the following works without complaint:

// 4

var discard = function () {
    a bunch of code;
}();

I then arrived at this:

// 5

void function () {
    a bunch of code;
}();

Note that void is a unary operator in JavaScript that discards its operand and evaluates to undefined. It is neither a function, as has been misreported, nor a data type, nor the absence of a data type, as it is in Java.

To me, #5 is more readable than #1 and #2. What I don’t like about #5 is that it’s still a bit cryptic. It may be cryptic, but I think it’s less syntactically disconcerting than #1 and #2, where I really don’t like the mysterious outer parentheses.

In case you missed it

Cases #1, #2, and #5 are equivalent. To me, #5 looks the simplest and the least prone to errors. I do wonder if there any insidious differences deep under the hood of any browsers.

Encapsulation? Or clean air?

This is a technique in JavaScript commonly used to avoid namespace pollution, and it’s often referred to as encapsulation, even though it’s really not. Booch defines encapsulation as “serving to separate the contractual interface of an abstraction and its implementation.” To this end, in Java and ActionScript, we have the interface. In JavaScript, we don’t.

Notes

  • 2012-11-20. Douglas Crockford refers to case #1 as “dog balls” here and prefers case #2.

Memory Leaks in IE8 and IE9 (Fixed in IE10)

In June 2007, Microsoft famously fixed a problem with memory leaks in Internet Explorer 6. IE8 leaks memory worse than IE6 ever did, yet I haven’t been able to find any mention of it. I say worse because:

  1. No closures or circular references are required.
  2. I cannot find a workaround. (Update: See comments from May 9, 2011 for a way to avoid the problem.)

Update: It was fixed in IE10.

Problem statement

The memory used by IE8 to create certain DOM nodes is never freed, even if those DOM nodes are removed from the document (until window.top is unloaded). There are no closures or circular references involved.

The DOM node types that leak are form, button, input, select, textarea, a, img, and object. Most node types don’t leak, such as span, div, p, table, etc.

This problem only occurs in IE8 (and IE9 and IE10 preview). It does not occur in IE6, IE7, Firefox, or Chrome.

In case you missed it: every image element, anchor element, etc. added to a page uses memory that is not reclaimed even if the element is removed from the page.

In practical terms

Think intranet web application rather than Internet web site—that is, something that you may leave running for multiple days. If you have a web page that never reloads as a whole but pulls over many updates from the server (e.g., images and anchors are updated Ajax-style), each update will leak memory. Note that update here means existing DOM nodes are removed and new DOM nodes are created.

Depending on the size and frequency of the updates, IE8 can end up using all available virtual memory, start thrashing within itself, and eventually failures occur; see below for more details.

The user hitting F5 occasionally (if the page can handle it) will unload window.top and free its memory, but there is no apparent programmatic workaround. If the page is in an iframe, unloading the iframe has no effect. It seems that window.top must be unloaded.

In terms of code

First example

Here’s the basic idea of the leak, which is to repeatedly execute the following:

function leak1() {
    var node = document.getElementById("TO_AREA");
    node.innerHTML = "<img>";
    node.innerHTML = "";
    node = null;
}

Notes:

  • You can leave off the second innerHTML assignment, as well as setting the node to null, which I put in just for clarity/emphasis.
  • There’s an initially empty div with an id of TO_AREA in the body.
  • Example innerHTML that doesn’t leak: <span></span>. See the problem statement above for a list of problematic node types.
  • As I mentioned, there are no closures or circular references involved here.

Second example

Here’s a similar piece of code that avoids innerHTML and has the same leak:

function leak2() {
    var node = document.getElementById("FROM_AREA").cloneNode(true);
    node.id = "NEW_AREA";
    document.body.appendChild(node);
    document.body.removeChild(node);
    node = null;
}

Notes:

  • The FROM_AREA contains HTML like in the first example. Still, there are no closures or circular references.

Third example

This is the simplest and most straightforward:

function leak4() {
    var node = document.createElement("IMG");
    document.body.appendChild(node);
    document.body.removeChild(node);
}

Replace IMG with (e.g.) SPAN, and there is no leak.

Other notes

These examples leak in IE8 but not other browsers. I’ve tried IE8 on Windows XP, Windows Vista, Windows Server 2008, Windows Server 2008 R2, and Windows 7. I’ve tried IE in IE7 backwards compatibility mode, as well as every possible variation of quirks and standards mode. Though that’s by no means every variation of everything, it makes me comfortable that it’s not a fluke.

Live example

Here’s a link to an example page that shows the problem highly amplified. What I mean by amplified is that the job of the page is to show the leak, highly exacerbated; it has no other purpose. There are start/stop links on the page, so don’t worry about the page doing something terrible on its own.

http://test.hemiola.com/leak-ie8.html

Characterizing the leak

The memory size (called private working set on Windows 7) of iexplore.exe grows steadily over time, but this doesn’t lead directly to failures. IE memory usage does seem capped, and it does seem to be trying to manage memory from a DHTML perspective within certain constraints.

After it reaches its self-imposed cap, it seems to start thrashing within itself. Operations that could be performed at maybe 100 times per second begin taking over one minute per operation. Eventually, DHTML-style out-of-memory errors can occur.

In closing

I’m very interested in hearing about workarounds to this problem. I will post any meaningful updates as they are uncovered.

Update 2010-03-26

Microsoft made an IE9 preview available on 2010-03-16. It runs my leak test and does not leak. This is very promising. Additional update on 2010-06-24: We are now up to preview #3, and still no leaks.

There is a lot of memory churn in IE9 for the elements that leak in IE8, and there’s no churn at all for elements that don’t leak in IE8. The memory usage pattern in IE9 could be characterized as following a sawtooth pattern (low amplitude, high frequency), but with no long-term growth. I ran the test for an hour with no memory growth in IE9, whereas IE8 grew by about 1 GB in the same time.

Update 2010-09-16

Microsoft released the first beta of IE9 yesterday. The leak is back, at a rate of about 1 GB an hour: the same elements are leaking as before, at about the same rate as before.

Update 2010-10-28

Microsoft released the sixth preview of IE9 today. The leak is still there, same as in the beta.

Update 2010-11-17

Microsoft released the seventh preview of IE9 today. There’s the same churn and sawtooth as in the original three previews, and likewise, there is no leak. This is turning into a nail-biter.

Update 2011-02-10

Microsoft released the IE9 RC today. The leak is back.

Update 2011-03-14

Today is π Day, and Microsoft released IE9. The leak is still there.

Update 2011-04-12

Microsoft released IE10 Platform Preview 1. In terms of leaking, this is very similar to the IE9 preview: no overall leak, but a lot of churn and the sawtooth pattern is back.

Update 2011-06-29

Microsoft released IE10 Platform Preview 2. No leak. I suppose there is no SmartScreen Filter in these preview releases, which might account for the leak being absent from them.

Update 2011-09-13

Microsoft released the IE10 Platform Preview 3 as part of the Windows 8 Developer Preview. The leak is still there if the SmartScreen Filter is off; otherwise (the default), the leak is not there.

Update 2011-11-29

Microsoft released the IE10 Platform Preview 4 for the Windows 8 Developer Preview. The leak is not there, but I suspect the SmartScreen Filter is also not there; it’s hard to tell.

Update 2012-02-29

Microsoft released the IE10 Platform Preview 5 for the Windows 8 Consumer Preview. The leak is not there, and the SmartScreen Filter definitely is. This is the best it’s looked in years.

Update 2012-05-31

Microsoft released the IE10 Platform Preview 6 for the Windows 8 Release Preview. The leak is not there, and the SmartScreen Filter definitely is. This is the best IE has looked in years. And Windows 8 is looking pretty good, too (after going to the desktop and staying there, that is).

Update 2012-11-13

Microsoft released the release preview of IE10 for Windows 7. The leak is back, just as before: with the SmartScreen Filter disabled, the leak is present; with the SmartScreen Filter enabled, the leak is not present.

Update 2013-02-26

Microsoft released IE10 on Windows 7. The leak is gone! With the SmartScreen Filter enabled, the sawtooth pattern is there. With the SmartScreen Filter disabled, it looks even better: no manic memory usage and no memory growth.

Update 2013-06-26

Microsoft released an IE11 Preview with the Windows 8.1 Preview. I haven’t had a chance to look at it.

Update 2013-07-25

Microsoft released the Developer Preview of IE11 for Windows 7. It looks fine.