David Pogue on Why Big Data will Change Everything
We’re well into the first phase of the Big Data era - the collection age. Data is gushing from the Internet, from our cars, from our phones, from our purchases. It’s being picked up by sensors, RFID tags, cameras, microphones, GPS sensors. And, as nervous, privacy-concerned consumers are well aware, all of that data is being collected as fast as we can generate it.
I’ve been writing about consumer tech for a long time. So long, in fact, that I can start to see the megatrends.
One of the most striking big-picture insights is that we’re not as far along technologically as we like to think. In fact, we’re just getting off the ground.
We think a phone that’s a quarter-inch thick is sleek and sexy? Oh, my friends—that’s not sleek and sexy. Ten years from now, it’s going to look like a Commodore 64.
We think it’s amazing that certain cars can parallel park by themselves? That’s nothing. In 25 years, we’ll plug in a GPS address and then read while the car drives itself.
And then there’s Big Data.
This field is so young, we’ve only just coined that term. It refers, of course, to collections of data “so large and complex that it becomes difficult to process using… traditional data processing applications,” according to Wikipedia, the source of all knowledge.
Former Google CEO Eric Schmidt’s 2010 quote still ricochets around the Internet: “There were 5 exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days.” (An exabyte is a billion gigabytes.) There’s some dispute over his numbers and his sources, but you get the point: We’re churning out data at an incredible rate, and it’s speeding up.
It’s already hard to contemplate how vast that pile of data is becoming. The industry is already straining to store it, back it up, and transfer it. (When a company ships you a hard drive by FedEx because it’s faster than uploading the data on it, you know the Internet infrastructure is already maxed out.)
Still, we’re getting better at it. We’re well into the first phase of the Big Data era - the collection age. Data is gushing from the Internet, from our cars, from our phones, from our purchases. It’s being picked up by sensors, RFID tags, cameras, microphones, GPS sensors. And, as nervous, privacy-concerned consumers are well aware, all of that data is being collected as fast as we can generate it.
But here’s the thing: We haven’t yet entered the second phase of the Big Data era: using the data.
OK, you could argue that we’re at the dawn of this new era, but I’d argue that it’s 5:30 am at the latest.
Here and there, you can see clever engineers harnessing big data already. You know how Google Maps color-codes the roads to indicate the speed of traffic? It performs that magic by analyzing big data. All those tens of millions of Android phones, Google’s software, are constantly transmitting their movements to Google (anonymously). Google’s algorithms crunch that enormous pool of real-time data, calculate the speed of the cars carrying those phones, and comes up with the traffic speed.
But traffic speed is low-hanging fruit. There are so many things we wish we knew with greater certainty. What the weather will be. What the stock market’s going to do. What makes a video become viral. How cancer works.
The next company to succeed on a Google scale won’t be a company that collects data or finds data; it’ll be the company that helps us analyze, process, and understand it.
Once we get a handle on big data, disease is a sitting duck. The keys to understanding the genome—understanding the workings of disease, and the body, and healing—are buried somewhere in those mountains of data.
Once we start to analyze the data, our understanding of genetics will change, too. And sports. And air traffic. And energy distribution. And climate change. And marketing. And earthquakes. And global security. And agriculture. And romance.
Data engineers, start your engines. It’s 5:31 am of the Big Data Era.