A few minutes ago, I finally succeeded at getting stencil buffers to work in the VectorStorm engine. (I’ll be pushing this, along with a few related changes, into the public repository later today)
Ignoring the modified GUI which is visible here (which I’ll talk about more in a day or two), the important point to note is the box in the top right corner; that’s going to contain a minimap.
The red area visible in it is actually just a small part of a very large red square which I’m drawing into the minimap. Stencil buffers are the way that I’m going to be constraining the minimap contents into that slightly oddly-shaped minimap area, rather than letting them extend out and overlap on top of everything else in the screen.
Conceptually, stencil buffers are a bit like regular render buffers, except that instead of storing a color for each pixel, they store a simple number value. In the old days, that number value could be either 0 or 1. Nowadays, though, they can typically go as high as 255. I’m not using those at the moment, though; just 0 or 1 for me. Anyhow, at the start of the frame, the whole screen’s stencil buffer value is cleared to zero, and then when we render things, those things can change the value in the stencil buffer wherever they draw. Later, we can use the contents of the stencil buffer to constrain which pixels other things can get rendered to. Stencil buffers used to be most commonly used to implement mirrors and shadows, back in the PS2-era. There are better, faster ways to get those effects on modern hardware. But stencil buffers occasionally still come in handy. Especially in this sort of “irregularly shaped GUI windows” situation.
In the VectorStorm implementation, I’ve exposed it this way:
Any vsRenderTarget may be told to contain a stencil buffer (set the ‘stencil’ variable on the Settings object you use to construct the vsRenderTarget). That stencil buffer is then automatically cleared to a value of 0 at the start of each frame. Materials also now have an optional “stencil” field, which sets how things rendered with that material will affect the stencil buffer of the render targets they’re rendered into. In the screenshot here, I have the minimap window set to render a value of ‘1’ into the stencil buffer everywhere where it draws to the screen. Finally, there are new vsDisplayList commands to turn on and off testing against the stencil buffer. For convenience, you can set vsScenes as being subject to the stencil buffer, so everything in that vsScene is automatically constrained inside what has been rendered to the stencil buffer. So inside such a constrained vsScene, I’m rendering a huge red square, but because of the stencil buffer, it’s only actually showing up where it intersects with the minimap window.
Next time, I’ll actually talk about this GUI. :)
