How ESPN Taught the Pentagon to Handle a Deluge of Drone Data

For many years now, huge—and hugely expensive—sports arenas have struggled to compete with the common sofa. The couch-sitter's view of a game broadcast on a high-definition TV beats the tar out of the view from a stadium seat. Digital recorders allow you to skip commercials and rewind key plays. And broadcasts provide a litany of statistics, commentary, and points of view (slow-motion, anyone?) that live venues simply can't deliver.

But in October 2010, when the Orlando Magic squared off against the New Orleans Hornets in the Magic's brand new Amway Center, fans at the game had the best of both worlds: They could watch the game live or see it on any of the 1100-plus screens situated around the arena. That's a little bit of heaven: the thrill of a live NBA game, plus more information and camera angles than any home viewer could ever hope to enjoy.

Today, more sports arenas have followed the same path. The modern stadium is a phantasmagoria of viewing technology—massive monitors hang above the floor and decorate seating tiers like bunting. Fans feast on data: replays, statistics, and live action from multiple angles. The job of managing all that information takes place in a control room that brings to mind a mission-command bunker, where a team of seasoned technicians pores over the video captured by dozens of cameras. The Amway Center uses a broadcast-video-management system developed by Harris Corporation, a Florida-based communications firm. As the video streams in, technicians tag it, time-stamp it, and embed it with information about the players and the action.

PM recently visited the control room, high inside the arena, with Magic spokesman Joel Glass. He explained why televised action plays an increasingly important role in live sports. "We're living in a world where it may be easier for people to stay at home and watch the game on their HDTV," Glass says. "In essence, we've brought that experience into the arena."

Less than an hour's drive away, in Melbourne, on Florida's Space Coast, Harris engineers demonstrate another version of this same broadcast technology in a more subdued office-park setting. This one is for the U.S. military and intelligence communities. Here, U.S. government intelligence analysts use the Harris video-management system to tag video not of professional athletes, but rather of, say, insurgents planting roadside bombs. The Full-Motion Video Asset Management Engine, or FAME, demonstrates just one way that the military is drawing on commercial broadcasting to help it fight in distant regions such as Afghanistan.

Lt. Gen. Larry D. James, the Air Force deputy chief of staff for intelligence, surveillance and reconnaissance, says looking to the sports world makes sense. When it comes to collecting and analyzing data, sports broadcasters are far ahead of the military. More crucially, for someone like James, who controls the Air Force's fleet of sensor aircraft—including a rapidly increasing number of UAVs—the goal is essentially the same: While sportscasters want to collect and catalog video on a specific player or a winning shot, the military wants the same capability to follow insurgents and bombings.

>

Last year, a group of Air Force officials, including James's deputy, visited the ESPN studios to see how the network handles its voluminous video intake. "They can find every clip of Brett Favre that they've had for X amount of time, whatever it is; they've developed the ability to do that sort of tagging and finding. For us, those are many of the same problems," James says.

At first blush, flying over Afghanistan looking for insurgents may sound nothing like observing a sporting event. But from a technical standpoint, the latter is actually more complicated, says John Delay, who has worked in commercial broadcast technology for more than 30 years and today is Harris's director of strategy. As many as 40 cameras may be recording a game at once, and broadcasters have to choose and sort footage of the action in real time. "We take that for granted from our broadcast consumption of video," Delay says. "Every day we watch sporting events or televisions: They are the equivalent of what you have in a Predator [drone] mission in a broadcast space."

FAME, like its civilian counterpart, is an intricate cataloging system that allows the video coming in to be tagged with a variety of metadata, such as time, location, type of event, and other markers, which allow the users to search and retrieve the video. In effect, metadata helps turn data into intelligence by making it searchable. FAME also records telestration, effectively turning intelligence analysts into uniformed John Maddens, marking up the action on a battlefield with Xs and Os.

The reason the military is increasingly turning to the commercial broadcasting world is simple: The military's collection of full-motion video, along with other sensor data, is exploding beyond what anyone could ever have imagined just a decade ago. The number of aircraft flying intelligence, surveillance, and reconnaissance missions over Afghanistan today is already double that of Iraq, according to Michael Vickers, the undersecretary of defense for intelligence.

It's not just the amount of data that has changed over the past 10 years, but the type of data. Before the 9/11 attacks, the Defense Department was focused on using overhead imagery to keep an eye on governments and militaries, individual terrorists and insurgents. Analysts would look through imagery for evidence of Iraqi forces massing near a border with Kuwait or for signs of new missile silos in the Soviet Union. When the United States went into Afghanistan, in 2001, and then Iraq, in 2003, analysts were suddenly charged with trying to spot much smaller targets.

Now, according to an Air Force intelligence analyst who spoke on condition of anonymity, analyzing imagery captured by drones is like a cross between police work and social science. The focus is on understanding "patterns of life," and deviations from those patterns. For example, if a normally busy bridge suddenly empties, that might mean the local population knows a bomb is planted there. "You're now getting into a culture study," says the analyst, who has been analyzing imagery since 1993. "[You're] looking at people's lives and realizing that while we're dealing with an enemy, that enemy is embedded within a civilian population, and there have to be ways to differentiate between somebody who is trying to eke out a living—and somebody who means to do us harm."

Not only is the focus different, so is the entire system of analyzing and distributing imagery and intelligence. These days, analysts scrutinizing imagery can carry on live chats with people on the ground in Afghanistan, a monumental change from years past. "In the olden days, [compiling information] meant sending the film back to the lab to go get prints made while we wrote up a report, stuck it inside a big ziplock bag, and mailed it to somebody," says the analyst.

As the wars in Iraq and Afghanistan began to rely increasingly on UAVs like the Predator to hunt high-value targets such as senior al-Qaida leaders and to battle insurgents planting roadside bombs, commanders suddenly wanted to collect more imagery—everywhere and all the time. In 2008, after then U.S. Defense Secretary Robert Gates blasted the Air Force for not providing enough surveillance aircraft, the flood of imagery became a deluge.

Today, officials routinely refer to the "tsunami of data," which is apt: The Global Hawk unmanned aerial vehicle takes 11,000 still images a month. The Air Force collects 1200 hours of video a day, while Blue Devil, another new sensor system, and Gorgon Stare are generating 53 terabytes of data a day—the equivalent 53,000 full-length movies. "If you think your iPad could download that much, have at it," James says.

After years of working with this flood of data, the Pentagon is now learning how to put it to good use. Consider this hypothetical scenario: An intelligence analyst at Langley Air Force Base, Va., reviews video shot by a Predator drone flying over Afghanistan. The analyst sees a red car that he or she thinks might have been used in a bomb attack, and then tags that video while also searching for references to red cars and bomb attacks in the region, trying to find a match. Data about the red car might also be cross-referenced with intercepted cellphone communications, in an attempt to ID who's in the vehicle.

At Harris's headquarters, Tariq Bakir, a motion-imagery scientist who works on FAME, demonstrates the system by showing how an analyst might look for clues to an improvised explosive device, or IED, attack. "You put in a keyword; for example, we put in the word 'IED,' here," he says, pointing to a form on a computer screen. "That searches for all video content and any piece of metadata that could be associated with it. This metadata could be a PDF report; it could be a cellphone picture of an explosion; it could be a TV broadcast."

FAME also stores live chats between the intelligence analysts and troops downrange, which provides key information about what's going on in the video. The analyst can even search by date range, location, and type of file, whether audio, video, or still image. "It's almost like Google," Bakir says.

>

While FAME is helping the Pentagon and the intelligence communities manage their video, the national security world is struggling to catch up with sports and commercial broadcasting in other ways. The military still follows analog conventions, for example, storing videos as discrete files, the way you would still images. Complicating matters further, the military services have separate data sets that can't be cross-referenced.

According to Harris, the Air Force has asked for the company's help migrating its video archives into a searchable database. If that occurs, relevant bits of information could be snatched and compiled from petabytes of data collected over more than a decade. "There are a lot of secrets locked up in that data," Delay says.

This is just another instance of the military following the commercial world's lead. ESPN has already spent hundreds of millions of dollars to tag and archive its videos, and Harris has also worked with ABC to transform the network's archives into a searchable, digital format. "We migrated every tape—and everything else—ABC had in its archives. Why? When I want to do a story on Andy Rooney, and there's a video clip from 1970 that's relevant, I want to get at it," Delay says.

The goals of the military and broadcasters may be vastly different, but Lt. Gen. James is unequivocal in his belief that all this video is helping the Pentagon. "Let's take the Predators: To have full-motion video of what's going on around you, in many instances, is priceless for the war fighter," he says. "To have those eyes overhead and to know what's going on is absolutely essential."

One thing is certain, however: Broadcasting technology in some cases may be all-seeing, but it is not all-knowing. In late November 2011, Afghan president Hamid Karzai blasted NATO Forces for an airstrike on a village in Kandahar province that killed civilians, including six children. Reconnaissance aircraft had reportedly been tracking suspected insurgents before the bombs rained down. Such incidents are not uncommon and underscore the difficulty of ascertaining from an image on a TV monitor exactly what's happening on the ground.

The Air Force intelligence analyst acknowledges the limitations of drone footage, and says that information collected the old-fashioned way must also be used. To correctly interpret an image, the analyst says, "Sometimes they have to go out and physically interrogate a subject."

Harris's Delay says the military has gained a clear advantage by following the sports-broadcasting model for collecting and analyzing video. "The biggest benefit that video has brought to the intelligence community is that you can observe and fight a battle remotely," says Delay.

It could be then that the most important contribution of full-motion video and modern broadcast technology is not necessarily in winning wars, but simply in changing how they are fought. The new tech allows the military to conduct battles the way many fans now watch sports: from the comfort of home.