From working for over 15 years at Vice as head of publishing, then as an independent writer, to getting a film deal for one of his stories, Thobey Campion saw one recurring challenge in media: turning text into visuals was “super inefficient and super slow.”
When generative AI came into the mainstream in 2022, and image generators like Midjourney and Stable Diffusion rose to popularity, Campion saw an opportunity to solve the story visualization challenge he’d been experiencing.
He started Lore Machine with Moises Sanabria, the company’s chief prompt officer, and Kutlu Mizrak, who is in charge of natural-language processing.
Lore Machine turns text stories into images, ensuring consistency in style for the different characters, locations, and recurring elements.
“To generate multiple images that feel like they’re all part of one story feels like a tug of war between a human and a machine that’s hell bent on doing its own thing,” Campion said of other AI image generators. “This consistency challenge became my holy grail.”
Users can upload their stories to the platform in any format, including poems, short stories, and screenplays. Then, Lore Machine’s natural-language-processing engine synthesizes the text to identify characters and their attributes, locations, lighting, props, and create summaries of different scenes.
After processing, the AI converts the textual work into a gallery of images, or a “storyboard,” with a double approach that combines large language models like ChatGPT and diffusion transformers like Stable Diffusion.
Users can currently choose between eight “styles” for their storyboard, like watercolor, manga, or line art.
Campion said the company plans to begin partnering with artists to create customized style presets for users to purchase off a marketplace. The artists would receive 100% of the profits from the sale of the presets.
The tool taps into OpenAI and Stable Diffusion, but its ultimate goal is to become “generator agnostic,” allowing users to choose between different text and image generators (e.g. Stable Diffusion vs Midjourney) depending on their preferences and their values.
For now, the platform is limited to generating images, but Campion said the company is building capabilities to allow the generation of animated content, as well as a “generative soundscape,” so that users can create a soundtrack to accompany their stories.
Once the storyboard has been generated, users can click into each image and re-generate it or modify it until they obtain a result that matches their needs.
Lore Machine is currently in beta, and being used by screenwriters, artists, novelists, graphic novelists, and animators, with a waitlist of over 12,000, the company said. So far, it’s had a successful run partnering with HP’s gaming division Omen, production studio Anonymous Content, and sci-fi writer Phil Gelatt to produce a manga using the platform.
Lore Machine’s team wants to help writers to come up with visuals that complement their stories both to reach a greater audience and be more involved in extensions of their work.
“In Hollywood, writers create the foundations for all entertainment and then get left out in the cold,” Campion said. “Our vision is that if writers could help shape the visual experience of their stories with Lore Machine, they could play a bigger role in a film’s production and the companion experiences.”
Campion said he expects Lore Machine to be publicly available in early 2024.
Here is a video of Lore Machine’s “spokesentity” giving a detailed description of how the AI works:
And here’s an example of a graphic novel based on Samuel T. Coleridge’s Rime of the Ancient Mariner made using Lore Machine.
Read the full article here