I want to share an experiment I did today that’s a testament to the incredible possibilities at the intersection of AI and creativity. It is still in its early stages, but I am excited about all the possibilities.
Here is my first attempt at turning a video into poetry using Generative AI, or as a matter of fact creating AI-generated voice-over for a video.
Sintel: A Nostalgic Choice
Let me take you back to my college days. Sintel was an open movie project released by Blender in my final year of engineering. All the 3d models, sets, and entire movie was available online to download and I could do as I please with all these wonderful assets. It opened a world of possibility for me as I learned animation and game development. So, when it came to choosing a video for this unique project, Sintel's beautiful trailer was a natural choice.
The Process
The process is straight forwards:
Since OpenAI can now interpret images I extracted frames from the trailer.
Next, I merged the frame with a prompt.
Transform the output response into voice using OpenAI's text-to-speech API.
Merge the voice-over with video from the trailer.
Converting Video to Frame to Prompt
Using OpenCV I extract one key frame for every second from Sintel Trailer, every 24th frame to be exact. Next, I encoded these frames in base64 - a language that AI understands. And then I merge these frames with a simple prompt asking GPT to create a poem inspired by the frames from the movie and the lyrical beauty of William Wordsworth.
Text to Speech
The poetry, once created, isn't just left as text. Using the new and updated voice-to-text API I transform the poem into a voice-over.
And finally I merged the audio with the existing video from the Sintel trailer using MoviePy to create a new version of the movie that's infused with poetic narration.
Here is the poem created by GPT:
In the realm where the cold winds blow, Did a tale of a girl and her dragon grow, Sintel, fair with spirit bold, Ventures forth through the wisp of snow. 'Neath the crests of mountains high, Past the silence of the sky, Bound by love that will not die, For a dragon she'll defy. Through the ruins time forgot, Echoes of a friendship caught, 'Gainst all odds, her battle fought, In the whispers of her thought. In the twilight of the day, Wings asunder, hearts at play, Sintel finds her fearless way, As the shadows turn to grey. Yet the bond that fate has spun, Catches fast the setting sun, When the end and start are one, What is lost, or truly won? Time, the thief, so swift in flight, Brings the dawn and steals the night, Sintel learns that love’s true might, Lies in letting go of plight. With a heart both brave and tender, Every step, an ember's sender, To the skies, she must surrender, Dragon's flight—her love's defender. Now beneath the desert vast, She moves onward, free at last, 'Neath the tree of life's repast, Sintel's tale—forever cast.
And it is really that straight forward.
GenAI is not just a tool for efficiency or automation. It's a collaborator that opens up new realms of creativity. The way it interpreted the frames of Sintel Trailer and wove them into poetry is nothing short of mesmerizing. I envision in the future AI will not replace artists but rather enhance their capabilities. AI provides new mediums and methods to express artistic visions. Just as Photoshop and Blender once revolutionized painting, AI is poised to transform creative arts, opening doors to uncharted territories of expression, making the once-impossible, possible.
Comments