Download the NXT Software Development Kit from the LEGO site. Read about the Fantom protocol and DLL. You'll need to make calls to this DLL from within MATLAB.
ROBOTC supports the Fantom.dll messaging protcol. This is the same communications protocol that is used by NXT-G and Microsoft Robotics Studio.
Try cross-posting on the nxtasy.org forum asking for someone who is already using MATLAB and is willing to share the DLL function calls with you.
Also do a similar post on the
www.lugnet.org robotics newsgroups.