Luís Arandas (https://orcid.org/0000-0001-5413-2832) University of Porto – INESC-TEC, Porto, Portugal
Author’s contact information: firstname.lastname@example.org
Miguel Carvalhais (https://orcid.org/0000-0002-4880-2542)
University of Porto – i2ADS, Porto, Portugal
Author’s contact information: email@example.com
Mick Grierson (https://orcid.org/0000-0002-6981-5414)
University of the Arts London – CCI, London, United Kingdom
Author’s contact information: firstname.lastname@example.org
INSAM Journal of Contemporary Music, Art and Technology Main Theme of the Issue: Technological Aspects of Contemporary Artistic and Scientific Research
Publisher: INSAM Institute for Contemporary Artistic Music, Sarajevo, Bosnia and Herzegovina
Section: THE MAIN THEME
Abstract: Language-guided generative models are increasingly used in audiovisual production. Image diffusion allows for the development of video sequences and some of its coordination can be established by text prompts. This research automates a video production pipeline leveraging CLIP-guidance with longform text inputs and a separate text-to-speech system. We introduce a method for producing frame-accurate video and audio summaries using a virtual timeline and document a set of video outputs with diverging parameters. Our approach was applied in the production of the film Irreplaceable Biography and contributes to a future where multimodal generative architectures are set as underlying mechanisms to establish visual sequences in time. We contribute to a practice where language modelling is part of a shared and learned representation which can support professional video production, specifically used as a vehicle throughout the composition process as potential videography in physical space.
Keywords: artificial filmmaking, deep generative models, language-guided diffusion, short film computing, audiovisual composition, multimodal sequencing.
The Journal is indexed in:
ERIH PLUS – European Reference Index for the Humanities