Posts Tagged ‘moon’

Silverlight 5 – Moon boundary transition effects

Playing around with the Hollow Moon Video Show, it came to me that the transition from outer space into the moon, vice versa, needs transitional effects.

I also noted that it is hard to stop at a exactly the position one desires around the video screen. This calls for a throttle!

Transitional effect 1: Entering the moon

It seems nice that in approaching the moon the hull seems to open up and show the interior world – as an invitation to come and explore. Next question is how to implement such a feature. The choice for a shader effect seems preferable over a vertex based effect here. A vertex based effect means that an animation will move parts of the sphere (the moon) around to open up. But, the same vertices are used to project the inner world, which would then be invisible where the moon surface was has been removed. So, a shader effect it will be.

In order to complicate matters not too much, the expansion shader used earlier was reused. The effect however presents itself entirely different in this new settings; expansion is now an inverse function of the distance to the surface, and it works on a sphere, not a flat surface. This provides for a quite nice effect.

You can’t immediately see the video screen now, I’ve increased the size of the moon significantly in order to improve the illusion one is travelling through an enormous and desolate landscape. See the section on throttling for more.

Transitional effect 2: Leaving the moon

The shader effect above was relatively simple to implement. However, now the challenge is to create a shader effect within an environment mapping effect where shader sampling is in 3D space. Behold, it is not the math, it is the shader engine (level 2) that responds to any creative idea with the message that the shader is too complex to process. A disappointment. So, for now I’ve settled for a simple black spot (hole in the moon, if you like) that shows the stars in the background (to be implemented J ).

The idea is simple: a small black spot is projected onto the moon interior surface where we will leave the moon. We approach, and the spot grows bigger, then we pass through. We create the hole by just clipping the shader texels at the desired location. But how does one find the right position for the black hole?

The answer to that question was not easy, and it is not quite satisfactory in its final form. The Silverlight XNA library contains an Intersect method for both the BoundingSphere and the Ray classes. The idea is simple: if you have a position and a direction – and a chase camera usually has, you can define a ray, and the Intersect method provides you with the distance to impact, if any. However, this method does not work for rays that originate within the BoundingSphere. OK, no problem, I’ll just project myself out of the Bounding sphere by a known distance, turn around, and then obtain the distance to impact. This gives you a value, BUT, the Intersection method provides you with a distance to the center of the BoundingSphere, not to its hull, and not in the provided direction.

So. As an alternative I found a handy article at StackOverflow containing a decent treatment of how to mathematically solve the intersections of a sphere and a line (ray). A static method implementing the theory takes 0.01 second to computer the distance to either zero, one or two intersection points.

This solution works fine most of the times, but if you navigate through e.g. the top of the moon, the black spot gets distorted into an asymmetrical form that resembles an ellipse, and it gets displaced. It seems to me that the distortion indicates the limits of the rendering system, rather than an error in the algorithm that positions the black spot. The image below depicts the black hole in effective mode.

OK, I do feel an obligation to improve on this. But it will have to do for now. As soon as I thought out an dramatic effect that the shader engine can render, I will implement it. I promise!

Throttling and the parallax-ish illusion

The idea of environment mapping is that the camera is always unboundedly far from the camera, so that a parallax effect is created: the background doesn’t move with the camera (or observer). One technique to implement this is to move the background with the camera (same translation), or alternatively to stitch the background to the far plane.

Although this is theoretically convincing, and sometime right in practice, I find it unsatisfactory. The background seems dead, not just far away. I’ve been in open ground a lot, and when you travel around, you will notice that your view of the landscape changes, especially if you watch the seem scenery for a prolonged time. It changes slowly, though.

In the current situation, that of the Hollow Moon, there is also little else for visual orientation than movement of the landscape. So we need, to move the landscape a little in order to generate the illusion of travelling through is. If not, encountering the video screen, or the boundaries of the moon itself, come a complete surprise.

Now, if we would move slowly relatively to the moon, and also give the impression that the moon is huge, it would take forever (well, weeks) to get from the starting point to the center of the moon. So, we need a throttle. The throttle can be used to move towards and into the moon quickly, and to travel inside the moon increasingly slowly. A throttle has been implemented and is controlled by the S (slower) and F (faster) keys. I’ve also double the maximum speed and increased the size of the moon relatively to the video screen. It is now about 1250 times the size of the screen. How to operate the throttle is also indicated on the start screen message.

Demo Application

A demo application of phase 5 of this project can be found here, scroll down to “Moon boundary transition effects” (no, the # tags still don’t work).

Wish list

It turns out that the further the development of this moon thing progresses, the more wishes arise that are not (yet) implemented. That in itself is curious, but well. This section is to be a recurring section of those wishes, if only as a reminder of imperfection, and a source for philosophical contemplations. So far we have the following wishes.

Dramatic exit effect when leaving the moon

As discussed above, but feasible for the shader compiler level 2, while in the context of environment maps.

Elaborate sound effects that include panning

As noted, it is not easy to implement 3D effects on the sound sources, split by stereo channel. WE cannot apply panning to the individual sound channels of a video. If we could, the user would experience the oral effect of the sound pillars rotating around. Also a Doppler effect would be nice.

Textures that indicate speakers at the sound pillars

The sound pillars that flank the video screen are evenly colored (gold). They could do with some grid like textures that indicate the presence of speakers at those positions. We wouldn’t go as far as to implement moving speakers (woofers), would we?

Higher resolution graphics and MipMapping

If the camera approaches the moon surface or the interior surface, the low resolution of the images used begins to show. So, we would like to be able to use higher resolution. We also should implement mipmap levels (when you get closer to the surface, a higher resolution image is presented), I know. Mipmapping has the drawback that is greatly increases the amount of data that has to be sent over the internet. There has to be stuck a balance here.

Advertisements

Silverlight 5 – Poor man’s sound effects

One of the beauties of Silverlight is that is has or is a web based software delivery mechanism. A user clicks a link (icon, image, and video) on a web page and is offered an application that in many instances can be executed within the confines of a web browser as well. Once installed the application can update itself when new version become available. Recently Silverlight has been extended to also use pretty arbitrary native software components using the PInvoke mechanism. This last extension is a winner, and we will see in this blog that it is because it totally opens the Windows platform for use by Downloaded Silverlight Apps.

In Phase three of the Hollow Moon we have seen that 3D-Graphics can be Combined with playing a video within the virtual world created with 3d-graphics. We see a video screen, rotating within the Hollow Moon. A chase camera allows us to pass through the moon, and the screen (below, over, left, right – anything). Of course, what we would like now is sound effects that place the origin of the video soundtrack at the sound columns. And we want more.

Volume and frequency effects

In approaching, the sound intensity should increase, in a natural way. A nice touch would be that the volume increases stepwise when we pass through the moon wall. Nicer even, would be if we could filter out the high frequencies when outside the moon, leaving a kind of muffled version of the sound track that clears up when entering the moon. We would also like a velocity dependent Doppler effect when passing the video screen, possibly at high speed.

Panning effects

When approaching the video screen, the sound should seem to be located at the sound columns as, mentioned earlier. Since the video screen rotates, we also would like the sound to pan in such a way that the sound seems to originate at the rotating sound columns. This panning effect should also take into account the position of the camera, especially if it is behind the screen, or below or over it. It should take into account the movements of the camera; i.e. if we move along with one of the sound columns, the pan effect should not be noticeable anymore.

Silverlight and sound effects for video soundtracks

So, now that we have specified what we want, can Silverlight provide us with the means to build this? Eh, no.

Silverlight has some means to manipulate sounds, but these do not apply to video sound tracks. If you want to use these facilities, you first have to extract the sound track from the video. That’s not what we want. We want the user to select the video to be shown, and then play it. We might even want the user to be able to play arbitrary streaming video. Easy and elegantly.

At first that is some kind of a disappointment. Why don’t they (Microsoft) just provide what I need???

But if you think about it: the intelligent solution is not to port all the Microsoft software to Silverlight – that would turn Silverlight into a monstrum – but to make all software available to Silverlight. And that is about (not all software is available) what has been done by the introduction of COM Interop (earlier) and PInvoke (version 5) mechanisms.

The possibilities available in Silverlight to manipulate a video soundtrack are the volume and the balance. Balance is not equal to Pan, although the difference is a bit subtle. The balance is the distribution of the stereo channel volume over the speakers. Pan is the distribution of the soundtrack channel volumes over the output stereo channels. So, what we can have is an increase in volume when approaching the video screen, including a stepwise increase when coming through the Moon perimeter, and some play with the balance.

Poor man’s sound effects

To see how far this could be pushed I devised some poor man’s sound effects.

Sound intensity I (volume) decreases as a function of distance d like:

I(d) is proportional to 1/ (1 + d^2)

Which is used to modify an initial intensity, resulting in:

I(d) = I(0) / (1 + d^2)

By using a high power (7, in this case), you get steep changes in volume close by the video screen. This provides at least some reference to a Doppler effect. Thus we get:

I(d) = I(0) / (1 + d^7), with I(0) = 1, and d typically in [0, 3000].

For intensities outside the confines of the moon, we subtract 10%:

I(d)outside = 0.9 * I(0) / (1 + d^7)

Values below 0.1 are clipped. The lower bound provides a faint trace of the sound track that is supposed to lure the user towards the video screen.

You can also swing the balance a bit to emulate panning the sound to match the positions of the sound pillars. In this case the balance swings between ± 0.5. By keeping the swing limited I hope to fool the listener in hearing a faint panning effect :-).

Alternatives

To be honest, the result is not very satisfying, especially with regards to the emulation of panning, and I am not going to accept this ‘unpremium’ sound effect situation. Well, I will for now, but I will drastically change my technological scope, see below.

As indicated above, the situation initiates a need to take the route via either COM or PInvoke to available technologies that do provide enough control over the sound track of the video. What are our options?

XNA Game Studio

We could create COM wrappers for XNA dll-s. And hope to access the relevant XNA functionality this way. But Hey! XNA cannot apply its sound effects to video sound tracks either. This will get us nowhere.

Silverlight MediaStreamSource

We could implement Silverlight’s abstract MediaStreamSource class. That will be a solution for .wmv file video’s (ASF format). The MediaStreamSource class allows for separate access to the video and the sound streams in a file. This however, feels like rewriting already existing software. Windows, DirectX, etc. already contain various software facilities to quickly access media files containing both sound and video, so we like to use those existing facilities.

Examples of COM and PInvoke

SharpDX and SlimDX

There exist Managed wrappers for DirectX technologies. In particular SlimDX and SharpDX. There may be others, of course, but these I found.

SlimDX is an open source .Net library that wraps DirectX. It makes intelligent use of C++ by loading an activating the native DirectX COM components, and wrapping it by .Net C++ code. The SlimDX library itself is a .Net assembly (that cannot be referenced by Silverlight directly). When I first heard of SlimDX, I thought it might be rebuilt against the Silverlight runtime, thereby solving the problem we’re facing definitely. However, since it is written in .Net C++, rebuilding it as a Silverlight application is a very extensive job. Nevertheless, When rebuilt as a COM server, we may be able to use it from Silverlight after all. It would bring the advantage that marshalling to / from the CLR already has been taken care of.

SharpDX is a direct competitor to SlimDx, but the source code is C#, and the Library is to a large extent generated from the DirectX api’s (wow!). It is also faster than SlimDX. A major advantage is also that it built cleanly on my PC, and SlimDX does not – it wants me to install VS2008 first. The advantage of SharpDX over SlimDX is that it seems possible to port it to Silverlight

Both SlimDX and SharpDX do not provide access to libraries that process video files. Too bad. Both are valuable sources of knowledge on how to access native software from Managed code.

NESL

NESL stands for Native Extensions of Silverlight. A open source collection of COM-servers that wrap Windows 7 functionality. Interesting about NESL is not only what it is, but that it also has an elegant mechanism to install required native software on the user’s machine (requires Administrator rights, though). Of course, Silverlight 5 introduces PInvoke, thereby providing direct access to the Windows system. Nevertheless, NESL provide good knowledge on the aforementioned install mechanism.

Windows 8

In Windows 8, The use of Xaml with C++ is a standard option. This makes DirectX (etc.) directly available for use in Xaml based applications; Only issue is that rendered 3D-graphics are to be re-rendered on a Xaml control like a BitmapImage, using a WriteableBitmap and a blit operation.

So, Conclusion

There is much to learn from NESL, SharpDX and SlimDX. SharpDX may be portable to Silverlight, providing a much better alternative than the current partial XNA port. In principle what is required is to provide native DirectX technology as COM servers in Silverlight apps. This can also be done on a per application basis; build a DirectX application, Wrap it in COM, use it in Silverlight, Require the user to have DirectX 11 and or Windows 7(+).

But this is not all. We also see the advent of Google’s NaCl (Native Client) which allows HTML5 applications to use C/C++ code in a sandbox. Microsoft will presumably follow suit in order to provide comparable performance, and sustain the market dominance of IE. But wait! It used to be possible to reference COM components from HTML pages. Does this still work in HTML5?

In this blog post , a lot of options have been discussed, and a policy has been set out: to create COM-Components, or PInvoke dll-s that can be executed by Silverlight applications to use the full power of the DirectX and Windows Media libraries, that can also be elegantly installed within the context of the Silverlight application.

I expect and hope for seamless integration of a DirectX surface in Xaml and the possibility to execute native code from an HTML5 application in order to draw the results on the HTML5 canvas, both as parts of Windows 8.

Sample Application

A Sample application exhibiting the poor man’s sound effects can be found here. Scroll down to ‘The Moon Poor Man’s Sound Effects’.

Silverlight 5 – The moon interior world

While looking at the moon as presented in the sample application, I thought: “wouldn’t it be fun to be able to get into it and then find a friendly landscape to explore”. What we need for this is a so called Sky Sphere, and a moving camera.

Sky Sphere

A Sky Sphere consists of a cube which faces are textured with a coherent set of 6 images. For instance, the 6 connected images resulting from taking a photograph into each of the 6 orthogonal directions, starting at the horizontal direction facing north. Within this cube is a sphere, and the inside of this sphere is textured with the cube textures by means of extrapolation of positions on the sphere onto positions on the cube. Then, within the cube you see the seamless projection of the cube’s images. This technique is used to provide an environment; it is also called environment mapping. The idea is to move the sphere and cube with the camera so the sky sphere always presents a distant environment. In the current version we have not yet set the step to always keeping the environment distant. At this point we want to be able to travel through the moon, to, into, through, out and away from it. The XNA Sky Sphere Community Sample served as a starting point for development.

Camera

A camera is in fact a component that provides you with a View matrix. There are a number of cameras available through XNA community samples. I explored:

  1. The ArcBall camera
  2. The Orbit camera
  3. The Chase camera

The ArcBall camera

The advantage of the ArcBall camera is that it has a large number of properties and many degrees of freedom. It therefore has a broad spectrum of applicability. On the other hand, the code is complex, and in any particular application it holds a lot of code you don’t need. Also it is hard to keep it leveled straight. Somehow I always managed to get the view somewhat tilted, but never managed to get out of the tilt. So I looked a bit further.

The Orbit camera

The Particles XNA community sample contains code for a typical orbit camera. An orbit camera can move around its LookAt, even zoom into it. This camera lacks the capability to move around freely, that is, to change, or control, the LookAt. So, we will need not one, but two camera’s, one of which will be this orbit camera.

The Chase camera

The Chase Camera is another XNA community sample. In the sample the target is a ship. In our project we do not have a visible target. The target is just the LookAt. Typically, the target, or LookAt, is what you control with the keyboard or game controller, the camera just follows the target. The target can also be controlled using the mouse. Instructions for controlling the chase camera can be found in the section “Sample Application” below.

What you control defines the major difference between a chase camera and an orbit camera. In the former case you control the target, and in the latter the camera (while the target remains stationary). One could easily create an integrated camera that switches between orbiting and chasing. For now, I’ve adapted the chase camera to fit the needs of this application, as described above.

Creating the Silverlight 5 version of phase 2

To port software from XNA to Silverlight 5 does (still) require real work. The first task is to split up the Effect file into separate shaders. One file for each vertex shader and one file for each pixel shader. Since we have an inside and an outside to draw, we end up four shader files that have to be loaded separately. The same holds for the .dds file that holds the image set for the environment map. The set had to be split into 6 separate file that have to be loaded individually.

At first I couldn’t find a way to load the Silverlight 5 textureCube into the shaders. But at some time I found that the TextureCube is a subclass of the Texture class, hence can be assigned to the textures collection of the GraphicsDevice which will then load it into the shaders at runtime.

The UI thread and the Composition thread of the DrawingSurface

We want to control the pitch (vertical angle), yaw (horizontal angle) and thrust by means of keyboard (and mouse, but see below). So, mouse and keyboard events set a variable that indicates that a change in angle is required. The mouse and keyboard events happen on the UI thread. The Draw event happens on the Composition thread. In order to prevent synchronization problems, we want to prevent (re)setting the yaw and pith variables on this thread.

In order to prevent both synchronization problems and loss of performance as a consequence of using locks or wait states, we set e.g. the pitch variable in the KeyDown event handler (for the Up and Down keys), and reset it in the KeyUp event handler. The Draw event handler only reads (samples) this variable.

In the RC (release candidate) version of Silverlight 5, the DrawingSurface and e.g. the Rectangle do not raise keyboard events. The Textbox, however, does. On the other hand, the DrawingSurface and the Rectangle do raise mouse events, but the Textbox doesn’t. Also, these events do not seem to be routed events, that ‘bubble up’, yet. So we have to choose between mouse input and keyboard input over the DrawingSurface. For this application I have chosen keyboard input. We use a transparent Textbox, stretched out over the DrawingSurface to raise keyboard events.

Keyboard control has its problems:

  • Before the TextBox has the focus, we first have to click it. We use a startup message that has to be clicked away as a work around.
  • It might well be that the left and right arrow keys respond only after a while. This does not hold for the up and down arrow keys, or the space bar. Reason unknown.

Remark on the render system: at the edges of the drawing surface, the shape of the moon gets distorted – flatted out to an ellipsis. This turned out also to be the case for the XNA version; something to look into.

Sample application

A sample application can be found here, as the Phase 2 app. The keys for navigation are:

  • Space: forward
  • Left (Right, Up, Down) arrow: rotate to the left (right, up, down)
  • T key: turns the camera around (180 degrees)
  • R key: resets
  • A key: toggles physics

The sample application demos the moon interior and the chase camera. And that’s it.

Edit 29-12-2011: Since the Release of Silverlight 5 RTM some changes were made to the sample program. There is no startup message any more. On the other hand, the application only runs after local installation.

Silverlight 5 – Rotating Moon in 3D

Providing 3D user experiences over the web is an extraordinary technological feat. It is possible when we use the XNA integration with Silverlight 5 (beta). This blog post describes a first experiment: showing a rotating moon. In upcoming posts I hope to extend this experiment to include some further nice features.

Approach

The approach taken here is to first develop a model in XNA, for PC, and to then port it to Silverlight 5 when finished. This provides access to well known and well developed coding idiom, and the means to test and debug the code. It also provides the opportunity to separate off any problems that are specific to the integration of XNA and Silverlight.

Starting Point

The starting point was the “Textures and Colors” sample from the Microsoft App Hub site. This sample provides the essential techniques for importing a sphere and texturing it with an image. The sample provides much more, but we don’t need that right now.

The mapping of an image, which is usually rectangular or square, onto a sphere is not straightforward. See the screenshot from the “Textures and Colors” Sample below (left) which maps a “Cloud” texture onto a sphere, and an experiment in which a quilt design was mapped onto the sphere.

There are two obvious ways around this problem. One is to have an image that has been prepared to compensate for this effect. The other way is to map the image to the sphere while compensating for its shape.

When we apply a technique called Spherical Mapping, we get somewhat improved results, but still not satisfying. Spherical Mapping uses Normal vectors to correct the u and v coordinates that are used to sample a texture for a given xyz 3D coordinate. Coordinates are calculated as:

u = asin(Nx)/PI + 0.5

v = asin(Ny)/PI + 0.5

where NX is the x-component of a Normal vector. Results in our cases are as follows.

The blue and white quilt ball looks much better, the cloud texture map still looks awful. Luckily, I was able to find a cylindrical projection map of the moon on the internet which looks quite good.

At first, I thought that the terrain was looking odd because of deformations caused by mapping the image onto the sphere, but this is the actual south pole of the moon. And, by the way, this is a much better image than the one you get from Google Moon (Earth).

Creating the Silverlight 5 application

The idea is to use the new DrawingSurface control to render the 3D scene. Needed here is a way to import the sphere model into a Silverlight 5 Application, and to use vertex and pixel shaders to position and texture it with the image.

Some people have already created practices and extensions that can be used to get this done. Entry point to these pioneers is Aaron Oneal’s Blog. Notable contributions that have been made are the CodePlex Babylon Toolkit project by (predominantly) David Catuhe, and The model loading CodePlex project by Simon Ferquel. We will use aspects of those contributions where required.

I’m somewhat reluctant to jump at CodePlex products, since experience taught me that frequently there are some disadvantages attached. For instance, a feature I really would like to use is the ShaderBuildTask. This automatically builds the shaders when you build your project. However, for the ShaderBuildTask the shaders should be attributed as ‘content’, not as ‘resource’. If not, you’ll get a build error stating that the output already exists. The disadvantage is that you cannot use the resource addressing idiom (the “;component/” format). Then the shaders cannot be loaded since the code that loads the shaders is in another Visual Studio Project than the shaders. So, back to the scripts from the first Microsoft Silverlight 5 3D demo application. See what I mean?

Anyway. The model Importer by Simon Ferquel did prove its use.

Sample App

A Silverlight 5 sample application showing the moon in slow rotation can be run at my App Shop. This is a separate corner of the App Shop: an HTML 5 site, presenting Silverlight 5 Apps.