how to (re)calculate shadow_tol

Recently, during shader development, I had to have the shader recalculate state->shadow_tol. I asked on the mental ray list, and Halfdan Ingvarsson shared what he called an “officially sanctioned snippet” (if I remember the phrase correctly).

This is how it goes (more or less accurately :)):

miVector *a, *b, *c;
mi_tri_vectors( state, 'p', 0, &a, &b, &c );

double d_shadow_tol, tmp_d;

d_shadow_tol = mi_vector_dot_d(&state->normal, a);

tmp_d = mi_vector_dot_d(&state->normal, b);
if (d_shadow_tol < tmp_d)
       d_shadow_tol = tmp_d;

tmp_d = mi_vector_dot_d(&state->normal, c);
if (d_shadow_tol < tmp_d)
       d_shadow_tol = tmp_d;

miVector ray_dir;
mi_vector_sub(&ray_dir, &state->point, &state->org);

state->shadow_tol =
       d_shadow_tol - mi_vector_dot_d(&state->normal, &ray_dir);

As I can sometimes have a hard time reconstructing math behind code (and NOT shamed of it! :) as creationists fundamentalists say “unapologetically”), he was also kind enough to provide an explanation:

If you know the plane equation, this is fairly straightforward. It’s basically the use of the point/normal dot product that’s used to find out the distance from point to a plane.

Basically, it computes the largest deviation of each triangle vertex from the plane defined by the normal at the shading point. This then becomes the distance to “push” away from the surface when computing the shadow (the tolerance), so that the shadow ray can “peek” over the terminator, so to speak.

Theoretically, you should recalculate state->shadow_tol in your shaders if you change any ray-related struct member (org, point, dir, dist) or state->normal (bump maps, for instance).

As I’m rather fond of C++ (even within mental ray shaders, if possible) and shorter code, I created a small utility class to encapsulé it:

/**     Class for calculating shadow_tol
        (distance tolerance value for preventing self-shadows).
*/
class calcShadowTol
{
public:
    calcShadowTol( miState * const s ) { init(s); }
public:
    void     init( miState * const state );        ///< Set up prerequisites for calculation.
    double  value( miState * const state ) const;  ///< Calculate and return tolerance value.
    void   update( miState * const state ) const;  ///< Calculate and put result to state/shadow_tol.
private:
    double            _tol;
};

inline void calcShadowTol::init( miState * const state )
{
    assert(state);
    miVector *a, *b, *c;
    mi_tri_vectors(state, 'p', 0, &a, &b, &c);

    _tol = mi_vector_dot_d(&state->normal, a);
    double            tmp = mi_vector_dot_d(&state->normal, b);

    if ( _tol<tmp ) _tol=tmp;
    if ( _tol<(tmp=mi_vector_dot_d(&state->normal, c)) ) _tol=tmp;
}

inline double calcShadowTol::value( miState * const state ) const {
    assert(state);
    return _tol-( mi_vector_dot_d(&state->normal, &state->dir) * state->dist );
}

inline void calcShadowTol::update( miState * const state ) const {
    assert(state);
    state->shadow_tol = value(state);
}

Hola. Ahh, the beauty of C++, right? (Looks awful in the blog window, but that’s not the case I assure you :))

Then you just say something like

calcShadowTol tol(state);

at some point in the shader to initialize, then you can say

tol.update(state);

as many times as you like. This can be useful if you have to do the update multiple times (ie. for any multisampling, like in a home-brew soft shadow shader :)).

UPDATE (2008-08-16):

Warning: the above code doesn’t work (crashes in fact) for hair primitives (as mi_tri_vectors(‘p’) doesn’t work with hair). So if you want to use the above code in a light shader for instance, you’ll need to add some extra checking (for the result of mi_tri_vectors() is miFALSE, hair primitive it is).