Pokémon Go maker Niantic’s pet sim Peridot might be the future of AR

Niantic is the creator of Pokémon GoIt was a pleasure to be invited by’s London office to view its mobile pet sim Peridot. I saw a presentation, tried the game — which is released on iOS and Android on May 9 — and got to hang out with some of the developers, including the computer scientist heading the small research team that’s building the machine learning tech behind Peridot’s potentially groundbreaking new augmented reality technology.

There is no better demo to demonstrate the full potential of PeridotAfter a couple of days, I got a home version with a tester to play. I played it for less than 30 minutes. I have two young kids (ages 4 and 6), who were immediately charmed to see the cute creature of Niantic’s invention, called a Peridot, hatch on my phone screen and run around my house. The kids were fascinated by the cute Peridot creature that Niantic created. They petted, fed, and even tossed a small ball at it which bounced off of the walls.

Niantic

Checking the baby Dot’s desires, I saw that it wanted to eat a dandelion, which needs to be foraged from grassy areas. Our backyard is paved, but there’s grass across the street, it’s a pleasant spring evening, and there’s a little time to kill before bath time — let’s go! As we set off, the children squealed when they saw the Peridot run ahead down the drive. It was impressive that they could differentiate between grasses, trees, and pavers so their creature could pick up different food.

I noticed that my Peridot wanted to forage from a Habitat, which is what this game calls the map-based local points of interest shared by all Niantic’s AR games (in Pokémon Go, they’re called PokéStops). I saw this was only a short stroll away and I hadn’t visited it before (I’m not a Pokémon Go player) — why not? We went there. I live in London. Can you guess what the object of interest is? It’s a Victorian stinkpipe.), which looks like a lamppost without a lamp and is in fact a sort of giant straw that was designed to release noxious gasses from the sewers below, well above the heads of the good Queen’s subjects. I’d never heard of these things or noticed this one before. After a fun time and some unexpected exercise (as well as learning a bit about the local history), we returned home. Niantic couldn’t have scripted it better.

PeridotThis is pretty much a typical pet-sim, similar to something like Nintendogs, crossed with Niantic’s vast mapping data resource and a new generation of AR tech. Interacting with your pet will earn you growth points and help it grow from a baby into a teen or adult. At the adult stage, the creature will want to be “released” at a habitat where it can breed with other Peridots (the animals are genderless) and sire a new baby for you to look after. Ziah Fgel, the producer, says it was originally intended for players to give their Peridots up permanently, but they were too upset. So you can keep them. Niantic has built enough variables into the creatures’ DNA that each one is genetically unique, and every pairing will create a new, equally unique baby.

Niantic

Imagine the social features Niantic built into this game genre; you can take your pets for walks and forage in different areas, or engage with local players to breed as an endgame. Imagine the constant but gentle trickle of goals, currency, rewards, and progress that is built into this free-to play live service game, along with the customization options (yes, pets can wear hats).

Niantic also has a second mission for gamers. PeridotIt is to redefine the AR experience and how we think of it. Your pet’s camera (or your smartphone) should be able, at some level, to see the world around it and react to it. This is the work being undertaken by Niantic’s R&D team, which is based over six floors of a narrow office building crammed into London’s Covent Garden area, and is led by Niantic’s chief research scientist Gabe Bostow.

AI work begins with accurate 3D mapping and object occlusion so that pets can be realistically hidden behind objects like chairs and trees. (Bostow seems frustrated that the design team requested a shadow showing the Peridot’s position behind the object be put back into the game, lest players get too stressed out about losing their charges.) It continues with “semantic segmentation,” as Bostow puts it, of your phone’s video feed of the environment, using machine learning to train the code to tell the difference between grass and water, to recognize a TV screen so your Peridot can chide you about screen time, and to identify classes of object such as pets, people, and plates of food. The neural networks required for this weren’t possible on mobile phones even five years ago, Niantic says. Peridot certainly exercises your phone’s CPU and GPU, if its effect on battery life is anything to go by.

Niantic

Peridot started life as a tech demo, and although the pet sim genre is a natural fit for Niantic’s efforts, you can tell. It still seems more like a skunkworks of a tech firm hungry for advancement than the labor of love of a video game developer. Bostow’s team is sharing its findings with the academic community, while the code will be available to third parties to license via Niantic’s Lightship suite of development tools. You can imagine the benefits the tech might have for robotics or other assistive technologies; Bostow likes to picture a future where he can hold his phone up to see the location of pipework behind walls and under floors when he’s doing some home improvement. There are probably more sinister uses for this combination of machine learning of environments with Niantic’s industrial-scale geopositional data farming, too, but don’t think about that — look at how cute these little critters are!

In practice, the tech is far from perfect: It can’t spot food that reliably, can see water outdoors better than in, and has trouble with windows. But there’s an undeniable magic to watching your Peridot scamper around your home, jump up onto tables, and notice your cat. It feels a little bit alive, and it will only feel more alive as the game is released and players begin to train Niantic’s neural nets en masse. Fogel notes that the theme of a growing game and its introduction to the outside world are a perfect match for the technology, which will help it improve over time its knowledge of the universe and its behavior.

I think it’s impressive. But it’s not my reaction that matters, or that tells you whether PeridotIt will definitely be a hit. My kids were delighted by it within seconds, and my 4-year-old was wandering around the house calling for “pennydot” within minutes. Also, everyone knows about Victorian stinkpipes now.