I agree with Rico. It's hard to find issues in your own code, which makes it all the more important to check it thoroughly.
Still, at least try to be careful with the code style:
+static void RotateX(FLOAT *out, UINT order, FLOAT a, FLOAT *in)
I think it's better to only use lower case for the (private) function name. Something like "rotatex" or "rotate_x" or whatever.
I can agree with the rotate name, but not with the X. X in capital is there to mean the X-axis. In math, one never uses a lower case letter for an axis. So, it would become hard to understand what means x.
+ if ( order < 2 ) + return;
Please indent it correctly.
OK.
+ return; +}
Unneeded.
One always said to me that it is a good pratice that a function returns something.....
+FLOAT* WINAPI D3DXSHRotate(FLOAT *out, UINT order, CONST D3DXMATRIX *matrix, CONST FLOAT *in)
Fix the FLOAT*.
I don't understand.
+ return; +}
Again.
Also, the patch is full of trailing whitespaces, please fix those too.
OK
Nozomi