The Production of a short movie:
The Cineca experience in the making of “APA” from a technical perspective
The production of a digital movie, even a short one, is a team work.
Our team was composed by about 20 people, living in different italian town,
and cooperating trought web access.
Team subdivision
role | tasks |
artists |
modeling, texturing, rigging, animating, lighting, compositing. |
technicals |
online repositories setup, renderfarm setup, technical supervision of asset production, render wrangler. |
directors |
The director is primarily responsible for the storytelling, creative decisions and acting of the film.
The art director provides the visual concept and supervises the artists in order to guarantee
the quality and the coherence of the visuals. |
consultants |
Historian, Archaeologist philologists get base documents and data and supervise production work
and asses the scientific correctness of the produced assets according
to their respective scientific competence. |
producer |
tracks production progress and costs, allocates facilities and human resource. |
Pre-production
The director provides the Screenplay: a document which narrate the story Scene by scene, detailing the Locations and the actions, expression, and dialogues of the Characters.
From this document we derive the Breakdown which further split the scene in Shots
and for each shot details the position and movement of the camera. The text of the dialogue is nailed down and provide the shot time length. Further indication in this document can be the time of day, the type of the lighting, the music and the environment sound.For each shot one or more images are created in order to have a first visual representation of the movie: the Storyboard.
The previous aforementioned documents will drive the actors while recording each character speech line.
Finally, by putting together the Storyboard and the recorded speech we create the Animatic,
which is a draft version of the movie with a very poor visual but with the correct timing. This document allows the Director to review many aspect of the movie, such as the rhythm of the narration, and to take strategic decisions in advance.
When the Animatic is approved, the sequence of scenes and shots is fixed.
Pre-production technical steps:
Online resource setup. We used three different online-repositories
- unversioned repository: used to hold all static data and reference materials.
- versioned repository (svn): used to hold all the movie assets (detailed later)
- manually versioned repository: used to hold the frames generated by the renderfarm
and the intermediate versions of the movie.
The difference among the last two repositories is that the svn preserves the history of all data, even the deleted data, while the other keeps only the latest two or three version of the data. The comparison among different renderings of the same shot can be really useful but a versioned repository that maintains the whole history is nearly impractical for the hugeness of produced data.
Other technical step are the Render farm Web Interface and the Production Blog
Post Production
The post production starts when all the movie frames are ready and involves the creation of
Music and Sounds eventually some Compositing,
Color Correction and the Final Editing.
The result is the final movie. We dont detail these aspect further here.
Production
The production of a movie (even a short one) is a huge work.
To keep the time of its development within reasonable limits many people must be involved, often coming from different companies, and the production should enable them to work in parallel, not hindering each other. So it is important to understand which stages of the work are inherently serial and which can go in parallel. Hence it is possible to deliver a first production plan and to foresee a time-frame for hiring the different teams.
As an example, the following picture shows the pipeline for the creation of a character.
At stage 1 we are in a loop where the character model keeps changing until all the animation tests are passed.
On the other side, work done at stage 2 and 3 will be ruined if the model changes,
therefore stage 2 and 3 can go in parallel, but must strictly follow stage 1.
On a different scale, the creation of different characters can proceed in parallel,
but the animation of a shot must wait until all the characters involved, as well as the scenery, are ready.
In our short movie we managed to have the creation of the scenery and the character animation to work in parallel.
This was possible by first creating a very low resolution model for each scenery,
then animating the characters inside these environments,
and later replacing the sceneries with the final version.
Of course, every time a high-resolution model doesnt match exactly the low-poly version we have a technical problem,
but we managed to solve all of them.
The following picture illustrates the workflow of our production,
which involved four teams and a technical support, working with a team-parallelism of 2 or 3.
Within each team different artists work in parallel as well: how do they cooperate?
In a digital movie all the different activities produce digital contents, often referred to as Assets.
Assets can be transmitted trough the Internet, allowing people to do workgroup even if from different geographical locations. The first thing to know about Asset is that they usually are binary files, and must therefore be managed in mutual-exclusion. When two artists are working on the same asset at the same time there is not an easy/automatic way to merge the work of the two; what will happen is that the artist who commits last will overwrite the work of the first one.
By hosting the Assets in a repository that is not versioned, the previous scenario would be problematic if not catastrophic, depending on the amount of the work lost. So it is common practice to use at least a versioned-repository like SVN, that will always allow to recover an Asset in any of its previous statuses. Even with this aid, manually merging the work of A and B can be problematic so, in the end, it is better to prevent the aforementioned scenario by adopting a mutual-exclusion politic.
Asset Files and Repository organization
Given the need to access the asset in mutual exclusion, another way to improve the Artist parallelism is to have the assets divided in a large number of smaller files instead of few and big ones. This also improves the use of the versioned repository. To achieve this aim we largely exploited the Blender capability to create links to external files.
The following picture shows the general structure of a shot file, which embed the camera and character animation, the lighting, and the compositioning network, but links the models of character and locations. Other files may embed models or recursively link to submodels; at the end of the chain there are the textures files.
As a side note, materials definition in Blender cant be linked, so in our work materials are never shared among different asset.
The production repository is divided in scene and library. In the scene branch there are all the shots files organized by scene. In the Library branch we have: the locations, the characters, props, and all the textures. Shot-files always link assets in the Library, files in the library always link files in their subfolders.
All the links are specified as relative-path
This overall organization is inherited from the project Big Bucks Bunny,
and we must here acknowledge the Blender Foundation for having released their production files in public domain.
Using external links, Artists are required to have on their own machines the Asset being manufactured but also all of its dependencies. Here comes into play another nice feature of the versioning-system, that is the ability to simply recreate on the user machine a copy of the whole repository (checkout) or a part of it.
By using the politics to specify the external reference with relative path, the Artist is free to checkout the repository where he likes, and everything works smoothly.
At the beginning of the production, the repository was small and every partner checked-out all the repository, toward the end of the work it was more frequent to check-out only the required subset.
It must be highlighted that the repository should contain all and only the required files,
non needed files in the repository make confusion, make harder to resolve technical problems, and waste disk-space on eachpartners machine. To give an idea, our production repository consists in more than 10.000 files and it is 15GB in size.
The versioning-system gives some tools to aid in keeping the repository clean but, in general, adding new files must be supervised.
Asset Life Cycle
Form now on we change the definition of the term Asset, previously defined as a single file.
Now by Asset we mean a repository folder and the whole of its sub-folders in order to reflect
that usually when an artist takes in charge a model he has also the ownership of all of its textures.
Asset can be associated with many Tasks: units of work to be done, but only one task
per asset can be active at a given time, for the mutual exclusion.
The supervisor evaluates Task results by inspecting the Asset.
The life cycle of Tasks and Assets are strictly correlated, so from the production perspective,
we use the term Asset and Task as synonyms
The following diagram shows the life cycle of an Asset.
- Blocked means that Artists cannot work on it because of missing references, dependency on a different asset, and so on.
- Task Pending means that there is work to do and it is ready to start.
- Working means that someone is working on it and, thus, no one else should touch it.
- Revision means that the work has been finished and must be checked.
- Rejected means that there are bugs to be communicated and fixed.
- Ready is self explaining
- Closed means approved by the directors and that no one should ever touch it again.
This mechanism was implemented using a Google Doc called AssetStatus shared among all the partners,
and the picture shows a portion of it: rows are Assets, cells are Asset Tasks and their status is color coded.
The overall color represent the production status and from the green dominant we can see that the picture was taken near the ending of the work.
The column "locked by" reports who is actually working on this Asset and there are more columns collecting
the directors review and the rendering notes.
Production Work-flow Detail
The next picture shows the whole machine running.
Artists read reference materials and write in the Asset repository,
Technicals do the technical supervision,
run the RenderFarm and produce intermediate version of the movie,
Directors review the current movie and generate new tasks.
In the Artist Loop,
Get an asset means update ones copy of the repository,
mark it working means go in the AssetStatus and update the relevant cells.
Return the Asset means do a commit,
mark it revision means again, update the AssetStatus accordingly.
In the Tech.Loop there are three activities:
run the RenderFarm (RenderWrangler) which means submit jobs, monitor jobs, understand why sometime jobs do not complete, fix and restart them, create shots movie from the rendered frames.
Monitor tasks completion and dependencies, eventually changing Asset status from Blocked to Task pending.
Last is technical supervision, which corresponds to the work on the compiler in a software project, and means: check Asset for bugs, possibly in advance in order to not waste RenderingFarm time, if a bug is discovered and can be fixed, do fix it, otherwise signal it.
Directors follow the work on a day by day base, but mostly the new tasks are generated in Review Meeting, where if possible all the partners are physically present.
Examples of some technical errors
- Missing Asset
eg: an Artist creates a new texture and forgets to add it to the repository.
- Bad Asset Path
ie: an Artist links an Asset to another one using an absolute path.
- Disappeared Asset:
an Artist removes an asset by mistake, or a link is lost ie. the Artist does not have a complete copy of the repository, opens a file which doesnt find all of its linked objects, when this file is saved the unresolved references are lost.
(knowledge sharing problem)
- Disappeared Asset 2:
a scene layer that was supposed to be on, has been turned off.
Solved by encoding which layers should be rendered in a different way.
- Black rendering::
disappeared light, of lights layer turned off (see previous)
- Moving Asset::
(rare) an asset that should be still becomes randomly animated. Reason: often the animator leaves the “auto register keyframe” button enabled, and some modellers don’t know about it. (knowledge sharing problem)
- Flying Buildings:
someone has moved the terrain, the terrain model or the vertical scale of the terrain has been changed.
- Bad Light Level:
scene or material is too dark or too light. Reason: without a calibrated monitor it is difficult to achieve the correct light level.
- Bad rendering of billboard with alpha channel:
not all the partners know how to correctly set up a material for transparency (knowledge sharing problem)
- Can't commit:
someone else is working on the same asset
- Can't update::
your local copy of an Asset is more recent than the one on the repository;
you should do committ before updating or throwing your Asset.
- Artists Fight 1:
The same Model is used in two different scenes (not shots, shots have similar lights) and two artists are adjusting the model material to make it work in their scene. They are working serially but one undoes the work of the other
- Artists Fight 2:
Similar to the previous, a unique giant quad is the floor for a town.
Artist 1, working on a building door, moves the town floor to make it match the door, Artist 2, working on the other side of the town, is doing the same thing.
- Giant Commit:
(rare) Asset files are normally less than 10MB, nothing prevents to committ a file of 300MB, and the day after all the people spend half an hour downloading it.
- RenderFarm Crash:
(rare) the Blender used for the RenderFarm is custom build, and it is not identical to others. For example it misses fonts: scenes using fonts will crash.
- RenderTime Too Long:
(frequent) ie. The time to render a Shot changes from 40 minutes to 10 days. Rendering tme is very sensitive to Lights settings. Lighters should do testing on their local machine before committ.
- It is not good enough! Really? I thought it was!:
(rare) (communication problem)
Conclusions
Asset management plays a central role in the development of a digital movie,
it is known that all the major companies in computer animation are using a proprietary asset manager developed in-house.
At this time we don't know about the existence of an open-source asset-manager,
therefore, following the setup of “Big Bucks Bunny”, we based our work on a SVN-repository.
Appendix: Project Metrics
at the end of the movie the production repository has :
380 | blender files, for about 6GB |
8500 | textures file, for about 7GB (some texture are animated) |
1000 | other files, (animation baking, etc.) for about 1GB |
5000 | committ operation (overall) |
3600 | committ operation on Blender files |
avg | num of committ per blender file = 9.85 |
avg num of committ per blender file = 9.85
Files with the highest number of committ | num committ |
APA_rig.blend | 69 |
shot_01_01.blend | 64 |
Museo.blend | 56 |
shot_02_01.blend | 45 |
shot_04_09.blend | 45 |
BolognaMulini.blend | 43 |
shot_09_11.blend | 39 |
shot_09_13.blend | 39 |
Torri/torri.blend | 34 |
shot_06_11.blend | 34 |
The renderfarm repository has:
450.000 frames ( low-res, hi-res, left, right, and their latest versions -- all JPG compressed)
which sum up to 141GB
The final movie has 24.000 x 2 frames summing up to 32 GB.
In may mailbox I have 2200 mail related to this project,
and I have probably done a greater amount of phone calls.