I have absolutely no idea what was I (or it) was doing earlier, but the Roto Brush tool seems to be working absolutely fine now. How bizarre!
My second attempt was infinitely more successful. I ran into a few problems with the background — it's a very similar colour to the cow so it was difficult to keep the selection consistent and close to her body. It's pretty shoddy and the outline wobbles all over the place but I wasn't too terribly meticulous with my selection — I figured that seeing as half of the cow would be hidden behind the window it wouldn't matter too much if it was less than perfect.
After watching a few more tutorials and having another go with it I feel much more comfortable with how it works. When you start painting your first selection you create a sort of "key frame" that After Effects uses as a point of reference for the following frames. It then creates a "span" which extends 20 frames ahead of and 20 frames behind (assuming there are that many frames ahead of/behind your reference frame). This span uses the information from the reference frame to calculate the movement of the selection as the selected object moves around. You can scrub through and view After Effects' automatically generated selection and for the most part, it's pretty accurate. Minor adjustments can be made simply by re-painting your selection on each frame.
The Roto Brush span lives just above the main timeline — the key/reference point is marked with a yellow square and the frame span is shown by a grey rectangle and series of arrows extending in either direction away from the key frame. My reference point was at the very beginning of the clip, so my span only extends forward. The default is 20 frames but you can extend it as long as is required. In the case of large differences between frames (in a clip with a lot of movement) it is often better to draw a new reference point rather than allowing After Effects to calculate the selection automatically.
Much like any other tool, the Roto Brush has a number of different parameters than can be used to really refine your selection. "Propagation" is quite complex and isn't something I delved in to, but it relates to how After Effects calculates the selection in each frame by tweaking things like pixel search radius.
The most interesting section for me were all the "refine matte" options. Much like Photoshop's "refine edges" function, it allows you to tweak the selection by smoothing, feathering and reducing the "wobbling" of your selection edges. Decontaminate colours does its best to remove traces of colour from the background seeping into your selection edges. For example if you had an object on a red background it would attempt to neutralize the leftover red tones from the edges of your object.
In the images above you can see how using decontaminate colours helped to trim excess pixels and neutralise the green tones from the edges of my cow — most notably on the neck.
I took the rotoscoped footage and scaled it down slightly before placing it on the background. I'm not too happy with the position of the cow — originally I wanted it more to the right of the shot but when I tried this the back end of the cow was obscured by the curtain. Seeing as the tail is where most of the movement is this resulted in it just looking like a static image, so I had to shunt it to the left a little so the tail would still be visible. I think I may try adding a second cow slightly off in the distance to the right of the window to bring the scene back into balance.
Fortunately the biggest problem area — the belly of the cow — is hidden by the window, meaning I don't need to worry too much about fixing that selection. Having said that there are some obvious problem areas I need to go back and refine — the neck and head of the cow and the tail. The tail was quite difficult to select owing to all the blurring in each frame but I think if I go back with a smaller brush I may be able to clean it up a little.
I also need to fix the unsteady camera. I don't own a tripod and had to balance my camera on a rickety wooden fence so there's a bit of jump in the footage. I think if I keyframe the position slightly to re-align everything in each frame it should be alright.