AI learns to predict average household income from Satellite Images

Meet Penny, the AI who "knows" how much you earn by looking at your home from space!

In my work at the European Space Agency, I come across many cool new research projects on using satellite images to create new and innovative services. In an earlier post, I shared how satellitess can be used to help predict the GDP growth of a region or country. Today I bring you a quite similar use-case of satellite imagery, however relying on a completely different technique.

Penny, an AI developed by Stamen Design and Researchers at Carnegie Mellon University, using satellite Data provided by Digital Globe. Image source

It works rather simply. You enter an address or click somewhere on the map of New York or St. Louis where the app currently works, and Penny tells you how certain she is that the area is a low, low-medium, medium, medium-high, or high-income neighborhood. While the application is still only an early prototype, it gives you an idea of where this is heading.

So how was Penny made?

A lot of data exists on average household income and self-reported well-being in western regions. In the US, surveys are regularly carried out in order to study what variables help improve a city or regions economic status and development. These data, more often than not, are publically available and can be accessed on site such as the US Census Bureau. An American company called Stamen Design, together with researchers at Carnegie Mellon University, came up with the clever idea of training an AI to predict these numbers using satellite images of regions.

At the top is a satellite image of Manhatten by DigitalGlobe, the image on the bottom show different zones separated by average income levels. Image source

The AI was fed with images of different American regions together with public data on average household income. Based on this information, it found patterns between the appearances and structures of buildings and infrastructures on one hand and average income levels on the other. For instance, living next to a hospital, railway station, or football field was highly correlated with a low-income neighborhood. Larger houses with big gardens, on the other hand, was a clear sign of high median income. Same with famous landmarks such as living close to the Empire state building. Some samples of images that the AI correlated with low, medium or high income can be seen below.

Sample images of what Penny considers to suggest a low, medium or high average income in a region. Image source

I find this really cool and fascinating on quite a few levels. At first, I was not really that sure whether anyone would really want to use this application, and that it was mostly just a fun little experiment. I guess some might be curious to see what people earn in different parts of the country, but this can be accessed in other ways just as easily. However, what is really cool is the idea that an artificial intelligence is literally looking all around our world, all at once, and "understanding" quite a lot about our lives and the state of our well-being and economic status. Since these images are all processed in the cloud, you can just imagine Penny as this cloud-based intelligence that is examining our lives and trying to figure out what we're up to. I can only begin to fathom what can be done when other secondary datasets are supplemented, or when other applications integrate this type of intelligence.

So what do you think Steemers? Is this cool or frightening? Should there be limits to what type of information we feed to an AI? If so, what? Looking forward to reading your opinions in the comments!

@fredrikaa