Story image

How AI can transform doodles into photorealistic landscapes

21 Mar 2019

A new deep learning model developed by NVIDIA Research turns rough doodles into photorealistic masterpieces with ease. 

The tool leverages generative adversarial networks, or GANs, to convert segmentation maps into lifelike images.

The interactive app using the model, in a lighthearted nod to the post-Impressionist painter, has been christened GauGAN.

GauGAN could offer a powerful tool for creating virtual worlds to everyone from architects and urban planners to landscape designers and game developers. 

With an AI that understands how the real world looks, these professionals could better prototype ideas and make rapid changes to a synthetic scene.

NVIDIA VP of applied deep learning research Bryan Catanzaro says, “It’s much easier to brainstorm designs with simple sketches, and this technology is able to convert sketches into highly realistic images.

Catanzaro likens the technology behind GauGAN to a “smart paintbrush” that can fill in the details inside rough segmentation maps, the high-level outlines that show the location of objects in a scene.

GauGAN allows users to draw their own segmentation maps and manipulate the scene, labelling each segment with labels like sand, sky, sea or snow.

Trained on a million images, the deep learning model then fills in the landscape with show-stopping results: Draw in a pond, and nearby elements like trees and rocks will appear as reflections in the water. 

Swap a segment label from “grass” to “snow” and the entire image changes to a winter scene, with a formerly leafy tree turning barren.

“It’s like a colouring book picture that describes where a tree is, where the sun is, where the sky is,” Catanzaro said. 

“And then the neural network is able to fill in all of the detail and texture, and the reflections, shadows and colours, based on what it has learned about real images.”

Despite lacking an understanding of the physical world, GANs can produce convincing results because of their structure as a cooperating pair of networks: a generator and a discriminator. 

The generator creates images that it presents to the discriminator. Trained on real images, the discriminator coaches the generator with pixel-by-pixel feedback on how to improve the realism of its synthetic images.

After training on real images, the discriminator knows that real ponds and lakes contain reflections, so the generator learns to create a convincing imitation.

The tool also allows users to add a style filter, changing a generated image to adapt the style of a particular painter or change a daytime scene to sunset.

While the GauGAN app focuses on natural elements like land, sea and sky, the underlying neural network is capable of filling in other landscape features, including buildings, roads and people.

This Iron Man drone wants you to fly like a superhero
Iron Man must be one of the most popular superheroes of 2019 – because this year he has been transformed into a robot, and now he’s styled as his very own drone.
HP back on board with Emirates Team NZ
HP 3D print technology will supposedly help the team innovate at speed.
Do you use the Peel Smart Remote app? Delete or update it now
Peel Smart Remote could leak your personal pictures, information, and documents to an unknown server.
How printing solutions can help save the planet
Y Soft has identified five key ways organisations can become more economical and reduce their environmental impact.
Information is power when choosing electricity plans
While 90% of respondents knew their average power bill over the summertime, more than half didn’t know their kilowatt hour (kWh) price, or their monthly usage amount. 
A quick look at Haier & LG's take on the smart home revolution
Haier is going all-in on the smart home revolution – and it believes your home should be smart all the way from the laundry to the bedroom.
An 8K TV for $80,000? Samsung has you covered
8K televisions have landed in New Zealand, but they come with eye-watering price tags of up to $80,000.
Scammers targeting more countries in sextortion scam - ESET
The attacker in the email claims they have hacked the intended victim's device, and have recorded the person while watching pornographic content.