Usually the fresnel effect causes images to soften around the edge of the reflective surface. So, save a copy of the scene, and then delete all textures. Make one new texture with no channels activated except the color channel. Make the color channel 100% white and apply it to every object in the scene. Now, delete all light sources and create one new point light that is placed at the same location as your camera.
If you were animating this scene, you would want to group the light to the camera if the camera moved. This light source should have no fall-off and no shadows and be at full 100% intensity. Render this pass and call it “Fresnel.tif”. Now, to control the fresnel effect of the reflection, you should have four separate files from four rendering passes: Mask Pass, 100% Reflection Pass, 0% Reflection Pass, and Fresnel Pass.
Compositing these passes together can be done in Photoshop or any image editing program that allows for multiple layers. However, it is very straightforward in AfterEffects, and if you are doing animations, AfterEffects is the only way to go. Thus, we will look at the compositing process in AfterEffects. Within AfterEffects, import all four passes into a project (File>Import>Footage Files).
Once they are all imported, create a new composition (Composition>New Composition). Make sure to adjust the size of the composition so that it matches the size of your renderings and the final desired size. In this case, the renderings are 800 x 600 pixels large, and since it is a still, the length need not be any longer than one frame. In this first composition, we will want to begin affecting how the fresnel effect plays out.
Drag the 100Reflection.tif file from the Project window into either the Composition window or the Time Layout window. Then do the same with Fresnel.tif. At the bottom of the Time Layout window is a button labeled “Switches/Modes.” Click this button so you can see the Mode and Track Matte functions of this composition. Change the Mode of 100Reflection.tif to “Add,” and change the TrkMat setting to “Luma Inverted Matte”.
What this does is make the Fresnel.tif (or our fresnel pass) a matte to the 100% reflection pass. A Luma Inverted Matte defines pixels to be opaque when the luminance value is 0%. Notice in the composition window that the center of the ball is now dark and “fresnelled-out.” Now we need to mask out the effects of the fresnel pass from the rest of the image. To do this, create a new composition (Composition>New Composition) and place “Comp 1” from the Project window into the Composition or Time Layout window.
Now place MatteMask.tif the same way, and click the Switches/Modes button. This time, change the Mode of Comp 1 to “Add,” and change the TrkMat to “Luma Matte”. What this does is use the first composition (the one where the fresnel pass was applied to the 100% reflection pass) and mask out everything but where the ball is seen as defined by out MatteMask.tif rendering pass. Create a third composition (Composition>New Composition), and in this composition first place 0Reflection.tif in the Composition or Time Layout window and then place Comp 2.
This places the pure file with textures and lighting for all objects at the bottom, and then places the little bit of the sphere that has been altered through the layering we did in Comp 1 and Comp 2 over the top. Click the Switches/Modes button and set the Mode for Comp 2 to “Add” You can continue to refine the effect by applying a blur effect to the reflection. You will want to do this within Comp 2 and apply the effect to Comp 1.
To do this within the Time Layout window, click on the line Comp 1 (not the tab) and then go to Effects>Blur & Sharpen>Gaussian Blur. A new window should pop up in your work area that allows you to adjust the degree of this effect. Adjust to your liking, making sure to click on the Comp 3 tab within the Time Layout window to occasionally check the results. When you like what you see within the final Comp 3, go to Composition>Save Frame As>File…. This will then allow you to save the file as a Photoshop document that you can later save as a jpg, tiff, or whatever format you need.
If this were an animation, you would go to Composition>Make Movie. After-Effects would then take a while to render the adjustments you have made throughout the footage you have imported. You may think, “Well, if I have to render in AfterEffects anyway, why not just rework it in the 3D application?” In some cases, this may be a smarter way to go; however, in most cases, AfterEffects will render many frames in a minute rather than one frame in many minutes like most 3D applications do with large complex files.
In addition, in some situations like the one analyzed here, we are creating effects that cannot be achieved easily within the 3D application, so AfterEffects (or Photoshop) is the only way to do it. It is yet another argument for never getting too tied to any one program. Use the application or combination of applications that get the job done to create the visual effect you want.