When you render in 3D Max make sure pre-multiplying is turned off.
Given the situation you want to render an object to be used for post-production within Photoshop, then…
- Be sure you set the background to complete black: Rendering > Environment > Background > Colour (click on the colour field and set
red, green and blue to 0)
- Set the antialising in your render settings to Blackman - this type of antialising is quite sharp and doesn’t blur the edges. In
V-Ray this is located in Renderer > Image sampler > Antialiasing
Filter.
- Set the file type in the common rendering dialog to tga, 32-bit, premultiplied and alpha split off, compress on.
- In Photoshop, load the TGA image, double click the layer to convert it from a background layer into a normal layer.
- Go to channels, CTRL-select the alpha channel, invert the selection, go back to layers and hit the delete button.
[...]
If you already have rendered your image and your alpha is accidentally
pre-multiplied, you can add a 6th step to the 5 I mentioned above. In
Photoshop go to Layer > Matting > Remove Black Matte. Voilà, you got
rid of the black ‘halo’ again.
Full article:
http://www.onoff.ch/2007/01/04/premultiplied-alpha-problems/