Silverlight 5 – Poor man’s sound effects

One of the beauties of Silverlight is that is has or is a web based software delivery mechanism. A user clicks a link (icon, image, and video) on a web page and is offered an application that in many instances can be executed within the confines of a web browser as well. Once installed the application can update itself when new version become available. Recently Silverlight has been extended to also use pretty arbitrary native software components using the PInvoke mechanism. This last extension is a winner, and we will see in this blog that it is because it totally opens the Windows platform for use by Downloaded Silverlight Apps.

In Phase three of the Hollow Moon we have seen that 3D-Graphics can be Combined with playing a video within the virtual world created with 3d-graphics. We see a video screen, rotating within the Hollow Moon. A chase camera allows us to pass through the moon, and the screen (below, over, left, right – anything). Of course, what we would like now is sound effects that place the origin of the video soundtrack at the sound columns. And we want more.

Volume and frequency effects

In approaching, the sound intensity should increase, in a natural way. A nice touch would be that the volume increases stepwise when we pass through the moon wall. Nicer even, would be if we could filter out the high frequencies when outside the moon, leaving a kind of muffled version of the sound track that clears up when entering the moon. We would also like a velocity dependent Doppler effect when passing the video screen, possibly at high speed.

Panning effects

When approaching the video screen, the sound should seem to be located at the sound columns as, mentioned earlier. Since the video screen rotates, we also would like the sound to pan in such a way that the sound seems to originate at the rotating sound columns. This panning effect should also take into account the position of the camera, especially if it is behind the screen, or below or over it. It should take into account the movements of the camera; i.e. if we move along with one of the sound columns, the pan effect should not be noticeable anymore.

Silverlight and sound effects for video soundtracks

So, now that we have specified what we want, can Silverlight provide us with the means to build this? Eh, no.

Silverlight has some means to manipulate sounds, but these do not apply to video sound tracks. If you want to use these facilities, you first have to extract the sound track from the video. That’s not what we want. We want the user to select the video to be shown, and then play it. We might even want the user to be able to play arbitrary streaming video. Easy and elegantly.

At first that is some kind of a disappointment. Why don’t they (Microsoft) just provide what I need???

But if you think about it: the intelligent solution is not to port all the Microsoft software to Silverlight – that would turn Silverlight into a monstrum – but to make all software available to Silverlight. And that is about (not all software is available) what has been done by the introduction of COM Interop (earlier) and PInvoke (version 5) mechanisms.

The possibilities available in Silverlight to manipulate a video soundtrack are the volume and the balance. Balance is not equal to Pan, although the difference is a bit subtle. The balance is the distribution of the stereo channel volume over the speakers. Pan is the distribution of the soundtrack channel volumes over the output stereo channels. So, what we can have is an increase in volume when approaching the video screen, including a stepwise increase when coming through the Moon perimeter, and some play with the balance.

Poor man’s sound effects

To see how far this could be pushed I devised some poor man’s sound effects.

Sound intensity I (volume) decreases as a function of distance d like:

I(d) is proportional to 1/ (1 + d^2)

Which is used to modify an initial intensity, resulting in:

I(d) = I(0) / (1 + d^2)

By using a high power (7, in this case), you get steep changes in volume close by the video screen. This provides at least some reference to a Doppler effect. Thus we get:

I(d) = I(0) / (1 + d^7), with I(0) = 1, and d typically in [0, 3000].

For intensities outside the confines of the moon, we subtract 10%:

I(d)outside = 0.9 * I(0) / (1 + d^7)

Values below 0.1 are clipped. The lower bound provides a faint trace of the sound track that is supposed to lure the user towards the video screen.

You can also swing the balance a bit to emulate panning the sound to match the positions of the sound pillars. In this case the balance swings between ± 0.5. By keeping the swing limited I hope to fool the listener in hearing a faint panning effect :-).

Alternatives

To be honest, the result is not very satisfying, especially with regards to the emulation of panning, and I am not going to accept this ‘unpremium’ sound effect situation. Well, I will for now, but I will drastically change my technological scope, see below.

As indicated above, the situation initiates a need to take the route via either COM or PInvoke to available technologies that do provide enough control over the sound track of the video. What are our options?

XNA Game Studio

We could create COM wrappers for XNA dll-s. And hope to access the relevant XNA functionality this way. But Hey! XNA cannot apply its sound effects to video sound tracks either. This will get us nowhere.

Silverlight MediaStreamSource

We could implement Silverlight’s abstract MediaStreamSource class. That will be a solution for .wmv file video’s (ASF format). The MediaStreamSource class allows for separate access to the video and the sound streams in a file. This however, feels like rewriting already existing software. Windows, DirectX, etc. already contain various software facilities to quickly access media files containing both sound and video, so we like to use those existing facilities.

Examples of COM and PInvoke

SharpDX and SlimDX

There exist Managed wrappers for DirectX technologies. In particular SlimDX and SharpDX. There may be others, of course, but these I found.

SlimDX is an open source .Net library that wraps DirectX. It makes intelligent use of C++ by loading an activating the native DirectX COM components, and wrapping it by .Net C++ code. The SlimDX library itself is a .Net assembly (that cannot be referenced by Silverlight directly). When I first heard of SlimDX, I thought it might be rebuilt against the Silverlight runtime, thereby solving the problem we’re facing definitely. However, since it is written in .Net C++, rebuilding it as a Silverlight application is a very extensive job. Nevertheless, When rebuilt as a COM server, we may be able to use it from Silverlight after all. It would bring the advantage that marshalling to / from the CLR already has been taken care of.

SharpDX is a direct competitor to SlimDx, but the source code is C#, and the Library is to a large extent generated from the DirectX api’s (wow!). It is also faster than SlimDX. A major advantage is also that it built cleanly on my PC, and SlimDX does not – it wants me to install VS2008 first. The advantage of SharpDX over SlimDX is that it seems possible to port it to Silverlight

Both SlimDX and SharpDX do not provide access to libraries that process video files. Too bad. Both are valuable sources of knowledge on how to access native software from Managed code.

NESL

NESL stands for Native Extensions of Silverlight. A open source collection of COM-servers that wrap Windows 7 functionality. Interesting about NESL is not only what it is, but that it also has an elegant mechanism to install required native software on the user’s machine (requires Administrator rights, though). Of course, Silverlight 5 introduces PInvoke, thereby providing direct access to the Windows system. Nevertheless, NESL provide good knowledge on the aforementioned install mechanism.

Windows 8

In Windows 8, The use of Xaml with C++ is a standard option. This makes DirectX (etc.) directly available for use in Xaml based applications; Only issue is that rendered 3D-graphics are to be re-rendered on a Xaml control like a BitmapImage, using a WriteableBitmap and a blit operation.

So, Conclusion

There is much to learn from NESL, SharpDX and SlimDX. SharpDX may be portable to Silverlight, providing a much better alternative than the current partial XNA port. In principle what is required is to provide native DirectX technology as COM servers in Silverlight apps. This can also be done on a per application basis; build a DirectX application, Wrap it in COM, use it in Silverlight, Require the user to have DirectX 11 and or Windows 7(+).

But this is not all. We also see the advent of Google’s NaCl (Native Client) which allows HTML5 applications to use C/C++ code in a sandbox. Microsoft will presumably follow suit in order to provide comparable performance, and sustain the market dominance of IE. But wait! It used to be possible to reference COM components from HTML pages. Does this still work in HTML5?

In this blog post , a lot of options have been discussed, and a policy has been set out: to create COM-Components, or PInvoke dll-s that can be executed by Silverlight applications to use the full power of the DirectX and Windows Media libraries, that can also be elegantly installed within the context of the Silverlight application.

I expect and hope for seamless integration of a DirectX surface in Xaml and the possibility to execute native code from an HTML5 application in order to draw the results on the HTML5 canvas, both as parts of Windows 8.

Sample Application

A Sample application exhibiting the poor man’s sound effects can be found here. Scroll down to ‘The Moon Poor Man’s Sound Effects’.