Mindstorms bricks can also be programmed using the graphical Robolab system. Robolab shares similar firmware with RobotC for both the RCX and NXT bricks.
'setMotorDirection' is a legacy feature required for Robolab compatibility. In Robolab, there's a series of common functions for manipulating motors that was the original RCX motor functions commands. Some of these functinos are not "natural" for the NXT and some emulation and conversion "glue" code was written to behave as close as possible to the legacy functions.
"setMotorDirection" is one of these legacy compatibility functions and it is not recommended for use in new RobotC programs.
To change motor direction you use either positive or negative power
The 'motorReflected' function is a nifty feature to chang the motor performance. Suppose you've got your robot built and you find the configuation is that one (or more) motors ends up moving in the logically opposite direction due to mechanical orientatino of the motor. On the RCX, this was easy to fix -- you simply rotated the black connector a quarter turn and this would reverse the way the pair of wires was connected to the RCX.
Connectors on the NXT are polarized and you can't use the RCX trick of physically flipping the way the wires are connected. So, the variable "bMotorReflection' is used to achieve the same effect in software.