On February 15, 2024, OpenAI shared a research paper on X along with a few amazing AI-generated videos to introduce the world to OpenAI Sora. Although Sora wasn’t the first artificial intelligence video model, it was the first to demonstrate such high levels of photorealism, consistency, and duration. Though some of the videos were made using fan-submitted prompts, only OpenAI staff videos have been shared on X or TikTok thus far, despite the impressive quality of the output. Before the model is incorporated into a program like ChatGPT, no information has been provided regarding when it will be made available to the public or what restrictions will be imposed on its output.
What is OpenAI Sora?
Like Runway’s Gen-2, Pike Labs Pika 1.0, and Stable Video Diffusion from StabilityAI, Sora is a generative video model. It creates AI video content out of text, photos, or videos. The company explained that the name “sky,” which is derived from the Japanese word for “sky,” represents its “limitless creative potential.” A couple strolling through snow-covered Tokyo was depicted in one of the initial videos. OpenAI Sora seems to be far more capable than some of the models that came before it; it can produce clips up to one minute long with consistent character and motion.
How Does OpenAI Sora Work?
Sora is a diffusion model, just like text-to-image generative AI models like DALL·E 3, StableDiffusion, and Midjourney. This implies that machine learning is used to gradually change the images into something that resembles the prompt’s description from the beginning when each frame of the video is composed entirely of static noise. Sora can make Videos that last up to 60 seconds.
What is the Technology Behind Sora?
The models developed for OpenAI’s generative image platform DALL-E 3 were modified to create Sora, adding features for more precise control. As a diffusion transformer model, Sora combines the token-based generators that power ChatGPT with the kind of image generation model that underpins stable diffusion. A latent space video is created, “denoised,” or formed in three-dimensional patches, and subsequently passed through a video decompressor to produce an output that is standard and readable by humans.
What Data Was Sora Trained On?
According to OpenAI, it used videos that were in the public domain, freely accessible, and copyrighted—for which the company had already paid for the right—to train its model. It hasn’t stated how many videos were used in the training set and isn’t likely to do so in the future. Millions are thought to be involved. To further refine OpenAI Sora on real-world content, the company used a video-to-text engine to generate captions and labels from ingested video files. Some rumors and conjectures indicate OpenAI also utilized artificially created video content, like that produced with Unreal Engine 5, since this would provide it with knowledge about the physics of the worlds within the video clips it consumed.
What are the Limitations of Sora?
OpenAI points out several issues with Sora’s current release. Because Sora lacks an implicit grasp of physics, “real-world” physical laws might not always be followed. The model’s lack of understanding of cause and effect is one illustration of this. For instance, the basketball hoop explodes in the video below, but the net looks to be repaired afterward.
What About Content Restrictions and Privacy?
Red teamers and safety specialists were also present during training, tracking, labeling, and outlawing use cases involving false information, hate speech, and bias using adversarial testing. The produced videos also have metadata tags identifying them as AI-generated, and text classifiers that ensure usage policies are followed by prompts are included. Similar to DALL-E 3, OpenAI claims Sora will have several content limitations before release. This will include putting restrictions on producing photos of actual people. In addition, making videos with graphic violence, pornographic material, racist imagery, celebrity likenesses, or intellectual property (IP) belonging to other people—such as logos and merchandise—will be prohibited. With DALL-E 3, none of this is easily achievable, and the same limitations will hold.
How Can I Access Sora?
Sora is restricted to researchers on the “red team” at this time. That is, specialists are tasked with attempting to find flaws in the model. For example, OpenAI will try to create content that includes some of the risks listed in the previous section to address the issues before making Sora available to the general public. A public release date for Sora has not yet been announced by OpenAI, but it is most likely scheduled for 2024.
What are the Risks of Sora?
Without safeguards, Sora can produce offensive or unsavory content, such as videos that glorify or promote illicit activity, feature violence, gore, or sexually explicit content. It can also result in derogatory portrayals of specific racial or ethnic groups. In an instructive manner, a video alerting viewers to the risks associated with fireworks could easily turn graphic. Consider how different inappropriate content is for different users based on who is using it (think about a child versus an adult using Sora).
FAQ of OpenAI Sora
Q1. Is Sora open to the general public?
A. No. Only a small number of highly skilled testers can currently access Sora, and they will be checking the model for any issues.
Q2. When will be OpenAI Sora released?
A. The public launch date for Sora is still unknown. Preceding OpenAI releases suggest that at some point in 2024, a version of it might be made available to certain individuals.
Q3. Is Sora AI unrestricted?
A. Though OpenAI typically charges for its premium services, there is currently no information available regarding Sora’s pricing.
