VFX Back to Basics Series: 8. What is compositing?

This is part of a series on the basic elements of Visual Effects. Each post will talk about a certain element which is one of the basic bricks used for building VFX shots.

In this eighth and final post in the Back to Basics Series I will talk about compositing.

A CGI image with a split trough the middle which shows color channels in the top part and the alpha channel in the bottom part. Alpha channels are the key to compositing.

Where all the previous posts talked about generating elements, this one will talk about combining these elements into a final image. For people who don't know the word compositing I always compare it to "Photoshop with moving images". We layer up different images and combine them into one with the illusion that the final image looks as if everything was filmed at the same time.

Back in the old days

Although digital compositing is a very powerful tool in filmmaking, compositing itself did start out as an analog process with the use of the optical printer. The first optical printers were created in the 1920's and were improved up till the 1980's. At first the effects were quite simple like a fade in or fade out but the effects became increasingly complex with the addition of things like matte paintings and blue screen effects. All these elements were combined by exposing the film several times.

This also explains the necessity of using mattes. If the film would be exposed twice it would create a double exposure unless the area where the second element needs to come is masked out. For blue or green screen shots these mattes were made with high contrast film and the use of a color filter to separate out the background color. In some other cases mattes were painted by hand which could result in chattering edges.

Films like 2001: A Space Odyssey and Star Wars had a staggering amount of success with this optical workflow.

The digital era

At the end of the 1980's digital compositing started to take over. It has two major improvements over its optical counterpart:

• In the optical process it is necessary to create the several layers like the mattes by copying it from one film strip to another. Each copy degrades the image and adds extra noise to the result. In the digital realm a copy is 100% the same as the original (unless you resample or recompress them but that is usually not done).
• The second problem is that film going trough a machine can drift a bit which could result in chattering mattes or halos. If done correctly digital compositing does never suffer from this.

When we need to composite for film a digital intermediate is created by scanning the film. When shot on digital cameras the source is already digital and does not need scanning.

Popular compositing packages today are Nuke, Digital Fusion and After Effects. The first two are node based packages while After Effects is a layer based package and works in a similar way as Photoshop does. Node based systems are represented by a flow chart where every node applies an operator to tree. Life action and CGI compositing do benefit more from this method while motion graphics are usually done in layer based packages.

A Nuke node based network.

Layered based compositing in After Effects.

Color and Alpha channels

Let's have a look at how compositing works in the digital realm. A digital image consists out of pixels which are represented by a combination of the three primary colors red, green and blue. This combination can create a broad amount of other colors available in the visible spectrum. Every color is stored in a channel.

In order to combine different images, it is necessary to have a fourth channel which is called the alpha or matte channel. It contains the transparency information of each pixel and this dictates how the result looks like when one image is put atop another image. It works just like its analog counterpart.

There are two ways to show an image with an alpha channel. The premultiplied one where the value of the alpha channel is already multiplied by the color values or the unpremultiplied one where the color values have their original values. This is very important to understand as failing to grasp this concept will possibly make composites look horrible with weird matte lines as a result. Check what your software package expects you to work with.

CGI images are usually premultiplied automatically when rendered.

A magnified CGI image. Look at the anti-aliased edges. This image is premultiplied.

The corresponding alpha channel image of the above image which has the same anti-aliased edges.

When unpremultiplied the edges become aliased.

The Extra's

Composting packages do not only put one image over another but have a vast set of tools available:

• Rotoscope tools: With these it is possible to create mattes to isolate parts of the image. These include also operations to modify edges or to paint over existing imagery.
• Retiming tools: All kinds of operators to manipulate the speed of the sequence.
• Color correction tools: Operators to analyze and change the color of the image. Very useful to match foreground and background elements.
• Filters: All kind of filters like blur, motion blur, noise, denoise and many more.
• Keying tools: These are useful for matte extraction from images with blue or green screen but also other methods are available like luma keying.
• Layering tools: from simple operators to put one image over another to operators for premultiplication.
• Transformation tools: Operators such as translate, rotate, scale, resize and distortions.
• Tracking tools: Tools to track objects in a scene or to stabilize a shot.

Although compositing is a 2D process, nowadays most packages support 3D environments. This adds extra functionality as it is possible to have proper parallax, use projections and makes the integration of particle systems really useful.

Another rather recent development is stereo conversion tools. It allows regular images to be converted to stereographic images. Note that I use the term stereographic instead of more common and commercial term 3D. Stereographic images are not truly 3D as the audience can not look behind objects by moving their head. I therefor prefer the term stereographic.

This concludes the eighth part of the VFX Back to Basics series.

Make sure to subscribe (at the top) or follow me on Twitter (check the link on the right) if you want to stay informed on the release of new posts.

In this series:

1. What is rotoscoping?

2. What is 3D modeling?

3. What is tracking and match moving?

4. What is 3D animation?

5. What is shading and texturing?

6. What is lighting and rendering?
7. What is matte painting?

VFX Back to Basics Series: 3. What is tracking and Match Moving?

This is part of a series on the basic elements of Visual Effects. Each post will talk about a certain element which is one of the basic bricks used for building VFX shots.

In this third post I will talk about tracking and match moving.

some of the tracking markers which I use on set. Eric Alba designed these markers.

We can divide tracking into two categories. There is 2D tracking mainly used for adjusting live action in the comp and 3D tracking for adding 3D generated imagery to a shot. The last category is also known as match moving. I'll start with 2D tracking as it is the base for the 3D kind.

Very often compositors need to follow certain objects or shapes in a shot. A good example is a billboard with some commercial message which the client likes to have replaced by something of his own. If the camera doesn't move then it is very easy to place the new image over the billboard and we can call it a job well done. Most of the time though directors do like to move the camera around to make those money shots. A solution would be to move the new image manually frame by frame to keep up with the movement. This is very time consuming and not too pleasant for the VFX artist. Instead we use a tracker and let the computer do the tedious work for us at a much higher speed. Another common use for tracking is to stabilize the backplate. Sometimes the movement of the camera is unwanted and instead of trackers following the image, the image tries to follow the stationary points. This is slightly oversimplified but brings the point across. Have a look at the image below and I'll explain how a tracker works.

A typical 2D tracking point with the tracking center and two boxes for the search pattern.

The main components are a tracking point, right in the middle, and two boxes around this point. The central tracking point is the coordinate of the tracker on your image. The smaller box is is the region which the computer tries to match when comparing frames and the bigger box is the search area in which the computer will make its search when trying to find the match for the smaller box. Choosing the location of your tracker is rather important as the computer needs a pattern which it can recognize. Placing the tracker on a solid color for example is bad as there are several positions on the image which will generate a match. Use corners of objects or high contrast pixels which can be provided by carefully placed tracking markers (see image at the start of the article).

Example of a bad tracking point. The pattern in the sand is not contrasty enough and the tracking point will probably start sliding when the computer tries to find a solution.

Once the computer has figured out the motion path of the marker it is possible to attach an image to that marker. The image will now follow the motion of the backplate. If the movement is only a translation then a single tracker is enough. If the backplate also rotates you need a second marker which combined with the first marker will find you the proper rotation. Stabilizing a backplate which was shot handheld is a good example where you need at least two points. If there is also a scale or perspective change then it is best to use four points. Adding more points usually gives better results.

Let's move to the 3D realm again and I'll show you how we can use the above technique to our advantage to extract three dimensional information out of a two dimensional image.

The world around us is a three dimensional space but when we shoot film or pictures that world is converted to a flat image. In my previous post I was talking about how convenient it is to use 3D modeling for Visual Effects but how can we place those objects in the scene if our source is only a two dimensional image? Luckily some smart mathematicians came up with Photogrammetry. This fancy word means that we can deduct the measurements of the objects in the photo when we know at least some measurements on a couple of objects in the scene that we are photographing. With these measurements we can rebuild the whole scene in a 3D space.

While filming with a moving camera there is an added benefit. When seeing the same objects from two different points of view and two frames from a shot with moving camera provide exactly that, we can use the power of triangulation. This concept is also called parallax. The more depth there is in the scene the better this will work as objects on different distances will appear to move different distances. An extreme example of parallax is watching the moon while sitting in a moving train. The trees close to the track whiz by but the moon doesn't seem to move at all.

A typical automatic track with lots of tracking points chosen by the computer.

3D tracking is done by the help of specialized software. Matchmover, Boujou, PFTrack and even NukeX can do the job quite well. These programs can track a shot automatically but if the result are not satisfying then it is possible to add tracking points manually. Same rule applies here too, more tracking points give a more accurate result as all the little mistakes get averaged out. The tracking points do have a quality value added to them which is compared to the average value. When they exceed a certain error margin they will be dropped from the results. It is also possible to remove an automatic tracking point manually if you think it gives bad results.

The ultimate match move is to track actors and their movements such as facial expressions. By putting a grid on the actors face it is possible to follow the facial features and then use them as a base for a CGI face which will replace the original face of the actor.

This concludes the third part of the VFX Back to Basics series.
Make sure to subscribe (at the top) or follow me on Twitter (check the link on the right) if you want to stay informed on the release of new posts.

In this series:
1. What is rotoscoping?
2. What is 3D modeling?
4. What is 3D animation?
5. What is shading and texturing?
6. What is lighting and rendering?
7. What is matte painting?
8. What is compositing?

Useful links:
Eric Alba's website.