For shot 3 in our project, gelato ingredients fly out of the computer and swirl around the character. To better integrate the CG ingredients, I created a setup to project shadows onto the character. I lack the skillset to create and animate proxy geometry to cast shadows on to, so I had to figure out a way to do it in Nuke.

I have already been experimenting with using some Cattery tools to generate normals of the live action for other effects. But a normal pass will not work for shadows that have to stretch and bend around what they’re being cast on. However, projections can be done with a position pass. Since I have the position of the camera, a position map for the image can be derived from a depth map.

I tried a few ML models to get a depth pass, MiDaS, DaVinci Resolve, and DepthAnythingV2. While they were all far from perfect, DepthAnythingV2 worked the best for this particular shot, although milage may vary with other shots. Rafael Silva’s version of the tool for Cattery allowed me to use it inside Nuke. Once I had the depth pass, I used a DepthToPosition node to convert it to world space position. I had to adjust the depth by multiplying it a bit in order to get the everything in the correct position.

Now that the depth pass has been set up and scaled for the scene, I created a camera in the same position where I figured the primary light source casting the shadows would be. I read in the geometry that would be casting the shadows and rendered it with a ScanlineRender through this new camera. I will be using the alpha of this render to project onto the position pass.

To do this position-based projection, I am using a tool by Hiram Gifford called ProjectionBuddy. I matched the rotation and scale of the projection to the second camera acting as the light source. There is some accuracy lost here due to the nature of position-based projections being limited to an orthographic perspective, but for the purposes of this shot, the discrepancies aren’t noticeable. I suppose this would only be completely accurate with a directional light emitting parallel rays.

Once the alpha of the ScalineRender has been projected, it can be used as a mask for a grade or any other method of composting shadows. Here is the result as well as a boosted version to show the effect better.