• Screenshot from Grand Theft Auto IV (Eurogamer)Source: Eurogamer
Getting computers to recognise other cars is surprisingly difficult, but a realistic video game can help.
Timothy Revell

New Scientist
27 Oct 2016 - 12:44 PM  UPDATED 27 Oct 2016 - 12:44 PM

Getting computers to recognise other cars is surprisingly difficult. Earlier this year, the first fatal autonomous car crash happened when a Tesla Model S failed to distinguish a white truck against a brightly lit sky.

Now a study has shown that self-driving cars can be taught the rules of the road by studying virtual traffic on video games such as Grand Theft Auto V (GTA V).

Although firms like Google and Uber are teaching their software by physically driving millions of miles in the real world, they also train their algorithms using pre-recorded footage of traffic. But there’s a catch: computers need hundreds of thousands of laboriously labelled images, showing where vehicles begin and end, to make them expert vehicle recognisers. That takes people a lot of time and effort.

“One evening, after a long day of hand-labelling images, I was playing Grand Theft Auto V,” says Matthew Johnson-Roberson at the University of Michigan in Ann Arbor. “I thought, ‘this is just so realistic that it would be a perfect simulation of the real world’.” 

Should self-driving cars kill one person or many?
With driverless cars already being tested on many roads, ethicists are trying to figure out how the vehicles should behave in a crash.

Picking out cars in a video game is a similar task to doing it in reality, with the advantage that everything comes pre-labelled because it has been generated by the game’s software. The team trained an algorithm solely using GTA V and tested it against the same algorithm trained on real-world images. The GTA-trained one performed just as well at spotting cars in a pre-labelled data set (arxiv.org/abs/1610.01983). The video-game version needed around 100 times more training images to reach the same standard – but given that 500,000 images can be generated from the game overnight, that is not a problem.

This isn’t the first time a research group has used video games to train AI, says German Ros at the Autonomous University of Barcelona in Spain. “It’s part of a bigger movement of using simulations to train artificial intelligence, which is beginning to take off.”

Finding the right training data is difficult. “We see AI being trained on images from similar locations, at similar times of day, under similar weather conditions, and then tested under similar conditions,” says Ros. This means that it’s hard to tell whether the computers can genuinely recognise cars, or whether they have just memorised that particular data set.

Using video games can help because they often show a variety of vehicles and conditions – but the problem is still there. “There’s nothing in GTA V that looks like a city in Japan, for example,” says Ros.

We have to show that driverless cars are safer than human-driven ones. In the real world, there is a car fatality for every 100 million miles driven. Racking up that kind of distance with a prototype is not easy. “The first step could be to make sure every system has been tested using video-game-style simulations before it hits the road,” says Johnson-Roberson. 

Autonomous future
Self-driving cars from Jaguar Land Rover set to hit the streets
Many in the car industry see driverless cars as the future of motoring, and a major shift for the industry.
Here's how self-driving cars will completely change your life
Let the car find a park itself. Drive if you're blind, or just blind drunk.

This article was originally published in New Scientist© All Rights reserved. Distributed by Tribune Content Agency.