Automated film direction for computer-generated discourse: planning, scheduling, and execution

Film directors make numerous decisions about how to record and edit content that aggregate into a very large logistical challenge. These challenges can be addressed by integrating AI problem solving systems into the film directing process. We present a computer-generated cinematic storytelling system that combines narrative planning with camera control to create a coherent cinematic narrative scene. Planning and scheduling are combined into a single problem solving task that is solved with our novel planning algorithm. The modifications to the task allows our system, called CINEPSE, to generate stories in the form of cinematic sequences. CINEPSE uses a film directing knowledge representation to transform a virtual game environment into a film directing planning problem. It interprets the output of the planner and executes a sequence of animation and camera instructions that produce a video output. Our evaluation shows that viewers rank the cinematic editing of our system's output comparably with an off-the-shelf camera sequencing tool. As a proof of concept, we adapted an existing game level into a film directing environment. We also used the film directing language to create a cinematic output and compared how viewers form memories of a cinematic version of events versus a noncinematic version. We conclude by considering future avenues of research, including strategies to expand our predictive model of comprehension using the combined story and discourse model.