Software Engineering Institute

Several VMware products include the ability to create and play movies of running virtual machines. The codec used in these movies is called VMnc, which is based on the VNC RFB protocol. The VMnc decoder is provided by the file vmnc.dll. The VMnc codec fails to properly handle video content with a specified height of less than 8 pixels. This flaw can lead to heap memory corruption. The vulnerable code in vmnc.dll may be reached via Windows applications that supports the DirectShow API.

By convincing a user to parse a specially crafted VMnc codec AVI file, a remote, unauthenticated attacker may be able to execute arbitrary code on a vulnerable system. This may occur as the result of several actions, including playing an AVI file with Windows Media Player, viewing a web page that uses the Windows Media Player ActiveX control or plug-in, or even simply by selecting an AVI file in Windows Explorer.

Apply an update

This issue is addressed in VMware Movie Decoder 6.5.3, Workstation 6.5.3, Player 6.5.3, and ACE 2.5.3. Details for obtaining these versions are available in VMware Security Advisory VMSA-2009-0012.

Remove the VMnc codec

If you are unable to apply an update, this vulnerability can be mitigated by removing the vmnc.dll file. Note that this will prevent a system from being able to play VMnc codec AVI files.