AI can watch videos by mimicking a living brain

Scientists at Scripps Research have recently devised MovieNet, a transformative artificial intelligence (AI) model capable of understanding moving images with the subtlety of the human brain. 

Unlike traditional AI models that excel at analyzing static images, MovieNet is designed to recognize and interpret complex, changing scenes over time. 

The innovation, detailed in a study published in Proceedings of the National Academy of Sciences, holds significant promise for applications ranging from medical diagnostics to autonomous vehicles.

“The brain doesn’t just see still frames; it creates an ongoing visual narrative,” said senior author Hollis Cline, the director of the Dorris Neuroscience Center at Scripps Research. 

“Static image recognition has come a long way, but the brain’s capacity to process flowing scenes – like watching a movie – requires a much more sophisticated form of pattern recognition. By studying how neurons capture these sequences, we’ve been able to apply similar principles to AI.”

Drawing inspiration from the brain

Cline and first author Masaki Hiramoto, a staff scientist at Scripps Research, based their work on how the brain processes real-world visual sequences. The research focused on tadpoles, whose optic tectum – the brain’s visual processing region – efficiently detects and responds to moving stimuli. 

These neurons assemble fragments of visual information into coherent sequences, mimicking how humans perceive flowing scenes in real life.

“Tadpoles have a very good visual system, plus we know that they can detect and respond to moving stimuli efficiently,” Hiramoto explained.

The researchers identified neurons in tadpoles’ brains that detect features such as shifts in brightness and changes in object rotation. These neurons process visual data in 100 to 600-millisecond clips, combining light and shadow patterns to create a continuous narrative. 

Cline and Hiramoto trained MovieNet to emulate this neurological process, encoding dynamic video clips as a series of recognizable cues.

AI watching video clips

To evaluate MovieNet, the researchers presented the model with video clips of tadpoles swimming in various conditions. 

The model achieved an accuracy of 82.3% in distinguishing normal swimming behaviors from abnormal ones – outperforming human observers by 18% and surpassing the performance of leading AI models like Google’s GoogLeNet, which managed only 72% accuracy.

“This is where we saw real potential,” Cline noted, emphasizing the significance of MovieNet’s ability to handle dynamic data. 

Unlike conventional AI models, MovieNet efficiently processes and compresses information, enabling it to deliver high accuracy with reduced data and computational demands.

A greener approach to AI

One of MovieNet’s standout features is its energy efficiency. Conventional AI models require immense computational resources, contributing to a significant environmental footprint. MovieNet, by contrast, reduces energy demands by simplifying data into essential sequences without sacrificing performance.

“By mimicking the brain, we’ve managed to make our AI far less demanding, paving the way for models that aren’t just powerful but sustainable,” Cline said. 

This efficiency positions MovieNet as an eco-friendly alternative, paving the way for scaling AI in industries where high costs have been a barrier.

Transformative potential in medicine

MovieNet’s ability to interpret subtle changes over time has profound implications for medicine. The model could assist in early detection of health conditions like neurodegenerative diseases and irregular heart rhythms. 

For example, small motor changes associated with Parkinson’s disease – often imperceptible to the human eye – could be flagged by the AI, allowing clinicians to intervene earlier.

In drug discovery, MovieNet’s dynamic analysis could lead to more precise screening techniques. Traditional methods rely on static snapshots, which can miss critical changes over time. 

By tracking cellular responses to chemical exposure, MovieNet can provide deeper insights into how drugs interact with biological systems.

“Current methods miss critical changes because they can only analyze images captured at intervals,” Hiramoto remarked. “Observing cells over time means that MovieNet can track the subtlest changes during drug testing.”

A technological leap for AI

MovieNet’s innovation goes beyond accuracy; it bridges gaps in existing AI technology by enabling nuanced analysis of dynamic scenes. 

Its ability to identify and interpret real-time changes in visual data sets a new standard for AI, making it a vital tool for applications requiring continuous monitoring and precise recognition.

For example, in autonomous vehicles, the AI could enhance safety by detecting and responding to changes in road conditions or pedestrian behavior. Similarly, in medical imaging, it could improve the detection of subtle anomalies that might signal early disease stages.

AI models that think like living brains

Cline and Hiramoto plan to enhance MovieNet’s adaptability, expanding its capabilities across various environments and applications. 

This includes refining the model to handle more complex scenarios and exploring its use in other fields, such as environmental monitoring and wildlife observation.

“Taking inspiration from biology will continue to be a fertile area for advancing AI,” Cline said. “By designing models that think like living organisms, we can achieve levels of efficiency that simply aren’t possible with conventional approaches.”

The research team envisions a future where biologically inspired AI like MovieNet revolutionizes technology across sectors. 

By replicating the brain’s sophisticated processing abilities, MovieNet not only advances our understanding of AI but also opens doors to innovations that could redefine industries and improve lives.

—–

Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.

Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.

—–