8 January 2021

The Autonomous-Car Chaos of the 2004 Darpa Grand Challenge

by Alex Davies.

WHEN THE INQUISITION required him to drop his study of what the Roman Catholic Church insisted was not a heliocentric solar system, Galileo Galilei turned his energy to the less controversial question of how to stick a telescope onto a helmet. The king of Spain had offered a hefty reward to anyone who could solve the stubborn mystery of how to determine a ship’s longitude while at sea: 6,000 ducats up front and another 2,000 per year for life. Galileo thought his headgear, with the telescope fixed over one eye and making its wearer look like a misaligned unicorn, would net him the reward.

Determining latitude is easy for any sailor who can pick out the North Star, but finding longitude escaped the citizens of the 17th century, because it required a precise knowledge of time. That’s based on a simple principle: Say you set your clock before sailing west from Greenwich. Say when the sun hits its apex, the clock reads five hours past noon. Because you know the earth rotates 15 degrees per hour—completing the 360 in 24 hours—you know you’re 75 degrees west of London. Easy peasy. That only works, though, if you have a clock that can keep accurate time, which nobody sailing the high seas then did. Clocks were complex mechanical devices, ill-suited to sea voyages. The rolling oceans messed with the pendulums that kept time on land. Salt air messed with everything. So explorers increasingly interested in crossing the oceans had a dangerously limited understanding of where they were.
Galileo’s solution lay with the incessant eclipses of what he called Jupiter’s four moons (and what astronomers now call Jupiter’s Galilean moons, having since discovered 75 more). In 1612, Galileo realized he could use the movement of the moons, which he charted, as a sort of astronomical clock. Over the next century, this became a common way of determining longitude on land. But it didn’t work on the ocean. The telescopes of Galileo’s age had tiny fields of view, and finding and tracking things that are at least 365 million miles away was virtually impossible on a ship bouncing along the waves. Galileo thought that by wearing the telescope on his face, a navigator could counteract that pitching and rolling more easily, and keep his focus on Jupiter’s moons.

The Spanish were unimpressed, and no one ever claimed the king’s ducats. Then, in 1714, British Parliament passed the Longitude Act, offering £20,000 for a solution. After decades of trial and error, working-class clockmaker John Harrison presented the marine chronometer, a time-keeping device that could withstand the rigors of the seafaring life and keep its practitioners on the maps they were filling in as they went along.

That successful iteration of the longitude prize was what Tony Tether, the director of the Defense Advanced Research Projects Agency, had in mind when he concocted the 2004 Grand Challenge: a way to motivate the Harrisons of the world as well as the Galileis.

DARPA WAS THE secretive arm of the Pentagon that had birthed the internet, GPS, and the stealth bomber. In the early 2000s, it turned its gaze to autonomous vehicles that could keep US soldiers out of harm’s way. Frustrated by the slow progress by the defense contractors and university labs that had spent decades on the problem, Tether tried something new: a race for autonomous vehicles, through the Mojave Desert, open to anyone. The $1 million prize and the audacity of the challenge drew in all sorts of hopefuls. Esteemed roboticists, university and high school students, hobbyist tinkerers, BattleBots veterans. What they lacked in experience with autonomous machines they made up for in enthusiasm. They spent the year leading up to race day sacrificing sleep, social contact, and in one case, $50,000 in Jeopardy! winnings to the Grand Challenge. A qualifying round slimmed an original field of 86 teams to 15 finalists.

At 4 in the morning on March 13, 2004, race director Jose Negron handed each team a CD‑ROM detailing the course. This list of 2,586 latitude and longitude coordinates, each about a hundred yards from the one before it, would lead the robots from the Slash X Cafe in Barstow, California, to the finish line in Primm, Nevada, 142 miles away.

While most teams simply uploaded the coordinates into their vehicle’s navigation system, a dozen members of Carnegie Mellon’s Red Team sat in a trailer and went to work on the task they had spent months training for. The route their software spat out said it would take their robotified Humvee, Sandstorm, 13 hours to reach the finish line. But Darpa wanted the winner to finish in 10 hours or less, and Team lead Red Whittaker—a cantankerous ex-Marine and one of the most talented roboticists on the planet—wasn’t about to settle. Neither was his team. Sitting shoulder to shoulder in front of a bank of computers, each student studied the waypoints for the section of route they'd been assigned.

The team compared them with their database of satellite maps, evaluating the difficulty of the terrain and calculating where the Humvee would need to run slow and, more importantly, where it could gun it. They entered a specific speed for every bit of the course and loaded all the data into the computers whirring away in the huge electronics box. Their planning was aggressive, but this was a race. If Sandstorm could pull it off, it would be hard to beat.

At 6:30 am on Saturday, March 13, 2004, with the sun just starting to peek over the hills to the east, the Grand Challenge was finally starting.

Safety monitors and reporters were in place along the route. Hundreds of people filled the bleachers. A helicopter whirred overhead, carrying cameras and the electronic kill switch that could stop any robot in its tracks. Darpa crew members sat in the chase vehicles that would follow each entrant through the course, with the same gear. A crew of biologists had run a final sweep for the endangered desert tortoise that called the Mojave home, and the first six vehicles sat in a semicircle, each in its own starting gate. They would enter the course at five-minute intervals, followed by the rest of the field. The Red Team, based on its impressive performance in qualifying, would go first. Chris Urmson, one of Whittaker’s lieutenants, prepared Sandstorm for its final journey, turning on its various systems, waiting for the green “it’s all good” lamp to signal that the emergency brake was primed, and flipping a silver switch from manual to auto.

The national anthem played, and Tony Tether took his seat next to a four-star Army general who had showed up without warning. Then the race announcer went to work. “The command from the tower is to move,” he called out. The flag dropped. Yellow light flashing and siren sounding, Carnegie Mellon’s Humvee rolled out of the gate. “Ladies and gentlemen, Sandstorm! Autonomous vehicles, traversing the desert with the goal of keeping our young military personnel out of harm’s way,” the announcer went on. “Booyah!”

Jose Negron and course designer Sal Fish had made the opening moves of the course deliberately simple. The GPS coordinates would take each vehicle out of the starting gate, through a couple of easy turns, and past the grandstand. Then it was through a couple of cattle gates and onto Powerline Road, a dirt path barely distinguishable from the brown land around it. As long as a vehicle could stick to the road and handle any rocks or tumbleweeds it encountered, the first few miles would be easy going.

As Sandstorm turned the first corner and started to disappear into the open desert, its laser scanner detected a clear path ahead, and its detailed map data said it was time to hit the throttle. Sandstorm’s wheels spun, kicked up dust, and carried it away at more than 30 mph.

For Chris Urmson and his teammates, the moment resonated. This was the first time they had let their robot out of sight and out of their control. There was nothing more they could do. No more testing, no more fixing. Sandstorm would complete the course, or it wouldn’t. And its first big test was just a few miles ahead. The trick to getting up and over Daggett Ridge was mastering the switchbacks, hairpin turns so tight that following a GPS path alone could easily send a vehicle tumbling off the path and hundreds of feet down. So could a misaligned sensor or any number of software glitches. If you could clear that hurdle, however, it was back to flat ground and mostly clear roads, nearly smooth sailing all the way to Primm.

Over the next 20 minutes, three more vehicles left the gate, following Sandstorm’s trail. It looked like Tether was going to get a proper race, even after some very shaky showings in the qualifying round. Then the problems started.

Sixth off the line was Axion Racing, a group of friends from San Diego, funded by an investor in a company importing bottled water from Micronesia. Over the previous year, software lead Melanie Dumas, an engineer who’d once written off the Grand Challenge as impossible and not worth trying, had seen her skepticism and reluctance transform into swelling optimism.

She had seen her team’s Jeep drive in this kind of terrain, and drive well. She even thought that, with a bit of luck, it might outrun Carnegie Mellon’s Sandstorm. When the flag waved, the Jeep pulled out of the chute and made the first turn smoothly. But as it approached the first narrow gate, it turned again. All the way around. There was no obvious reason for the about-face. Maybe the sensors had deemed the opening too tight. Perhaps something else had acted up. It didn’t matter. As the Jeep drove back to the starting line, sending its chase vehicle backward like a linebacker, Darpa hit its emergency shutoff. Axion’s Grand Challenge was over in a matter of seconds. Dumas was devastated.

Next up was the University of Louisiana’s six-wheeled Cajunbot. It smacked a wall on the way out of the chute, knocking itself out of contention. It was followed by Ensco’s bathtub of a bot. As the flag waved, it stood frozen for a few seconds, rolled forward, stopped, then started again. It drifted to the left, where the edge of the road sloped upward, tilting to one side before moving back to flat ground. Then it went left again, this time too far. It flipped over and landed on its side, 1,000 feet into a 142-mile course. The whole run lasted 1 minute and 6 seconds.

A group of students from Palos Verdes High School had spent the night before the race scrambling to fix the steering controls for their vehicle. At the last moment, they settled on a solution they hoped would work, with no time to test it. Their prayer went unanswered. Their entry, Doom Buggy, never turned at all. It rolled out in a straight line and, after 50 yards, hit a concrete barrier.

SciAutonics I, led by an engineer who’d worked on Germany’s autonomous driving efforts in the 1980s, saw its ATV wander off the trail, never to return. (The SciAutononics II made it about seven miles before getting stuck on an embankment.) The University of Florida’s Cimar strayed off course half a mile in and got tangled in a wire fence. Terramax, the 14-ton, lime-green, six-wheeled military truck, went 1.2 miles before getting stuck between a pair of small bushes that its sensors mistook for immovable obstacles. Tired of watching it lurch back and forth like a driver trying to escape an impossibly tight parallel parking space, Tony Tether ordered the kill.

More than one person had compared the field of homemade robots to something out of a Mad Max movie. Now they had the carnage to match.

Ignorant of the slaughter happening at the Slash X, Sandstorm carried on ahead, driving smoothly down the dirt road. But the race officials in the chase vehicle soon noticed troubling behavior. Nine minutes into its run, the Humvee approached a pair of fence posts about 12 feet apart, at a road crossing. Its laser sensor picked up the obstacles, and Sandstorm should have been able to slide through with a foot to spare on either side. But it hung to the left and smacked head on into one post, which didn’t stand a chance against a Humvee moving at speed. Neither did its compatriot at the other side of the intersection, which met the same fate a few seconds later.

Ninety seconds after the double whammy, Sandstorm stopped, possibly because of an inadvertent signal from Darpa’s emergency stop system. It got moving again after five seconds, but when it strayed into yet another fence post 44 yards up the road, it didn’t have the speed to blast through it. The Humvee wasn’t programmed to give up, though, and it pushed against the barrier for nearly two minutes before the post gave way. Sandstorm went the next 13 minutes without incident, then smacked into a boulder on the left side of the road. Its recovery software sent the robot too far back to the right, then corrected.

This “off-nominal behavior,” as Chris Urmson and his teammates put it in a postmortem report, was not disqualifying. Darpa had no rules against destroying a bit of property on the way to Primm. But some flaw in the Red Team’s system was leading their vehicle astray, and there was no fixing it as the robot headed for the narrow road that would lead it up and over Daggett Ridge.

The robot worked its way through the first few turns of the road without incident. Then it headed into one hairpin a little bit wide. As it turned the corner, it cut toward the inside, going just a touch too far and pushing its left wheels off the road. Sandstorm kept going, straddling the embankment and lifting itself just enough that its wheels couldn’t get the traction to move forward. The computers didn’t know the vehicle was beached, so they stayed on the throttle. For three and a half minutes, Sandstorm spun its wheels, until the rubber melted off the front tires. Watching the smoke pour off the robot, a Darpa crew member hit the emergency stop. The wheels were going so fast, the clamping of the brakes snapped the half shafts that connected the transaxle to each wheel. That last bit of damage didn’t matter, though.

Just 7.4 miles into the Grand Challenge, Sandstorm was dead. Back at the Slash X, Chris Urmson and his teammates had no idea how Sandstorm was doing until radio reports started to trickle in. As with most disasters, the first bits of news were hard to piece together: The robot was on fire, or its tire had come off, or it was stuck on a rock. It soon became clear, though, that the robot was finished. All that work— the freezing nights at their abandoned steel mill test site in Pittsburgh, the brutally hot days in the Nevada desert where they turned the Grand Challenge into a 24-hour affair, the endless practicing with the mapping software—and the team had made it just 5 percent of the way to the goal they had always refused to consider out of reach. They had come in as the favorites, the powerhouse to be reckoned with. And they had come up short. Some cried. Others let the exhaustion finally, fully hit them. At least they’d get to go home now.

By 9 am, Tony Tether knew he wouldn’t get to grin for the cameras as he handed someone an oversize check. The other vehicles that had looked good at the start met their own fates within a few miles: blocked by a rock, stuck on an embankment, stalled going up a hill. But one vehicle had yet to leave the start line: Anthony Levandowski’s Ghostrider.

Two days shy of his 24th birthday, Levandowski was the foil to Red Whittaker and Chris Urmson’s effort. Without backing from UC Berkeley, where he was pursuing a master's degree, Levandowski had pulled together a scrappy team and created a robot nobody else had been crazy or stupid enough to attempt: an autonomous motorcycle. A two-wheeler could be more nimble than a hefty Humvee, Levandowski had thought. It would definitely attract plenty of attention for its creator.

Ghostrider had performed miserably in the qualifying round, but Tether and Negron saw the PR value—it encapsulated the open nature of their challenge—and invited it to the final. It couldn’t win, in any case: It didn’t carry enough gas to cover 142 miles. But Levandowski didn’t have to reach Primm to come away a winner. If the motorcycle could go a few miles, it would prove that he had the engineering chops to back up the gutsiest idea Darpa’s Grand Challenge had engendered, and prove a lot of naysayers wrong: the Berkeley faculty who had chuckled at his idea; the many, many potential sponsors who had turned him down; the established robotics community as a whole, whose recipe for autonomous driving Levandowski had ignored. He was the underdog, the kid with the crazy idea, the leader of the team that made the world’s first self-driving motorcycle. Ghostrider would show the world what he could create.

Levandowski rolled his robot cycle into the starting gate, set it to run, and passed it off to a Darpa official who would hold it upright until it was launched. The flag dropped. Ghostrider started to roll forward.

The Darpa man took his hand off.

The motorcycle fell to the ground.

Levandowski ran to his machine, flapping his long arms in distress. In the heat of the competition, and after two days without sleep, he had forgotten to flip the switch to turn on the stabilizer system his team had spent a year developing. Darpa’s rules were clear: Once the vehicle started the race, it couldn’t accept any human help. Ghostrider wasn’t about to raise itself from the ground. It was done, dead. The bike had crashed somewhere between 600 and 800 times over the previous year. None hurt like this one, because this was the one time it was supposed to work, when everyone was watching. “Good try, guys,” the race announcer said. “Anthony, good effort.”

Tony Tether got into the helicopter watching over the race and headed to Primm, where a gaggle of reporters was waiting to see the first robots cross the finish line. One asked him how it was going. “Well, it’s over,” Tether said. “You know, the farthest car went 7.4 miles, caught fire, blah blah blah. It’s over.”

The Darpa Grand Challenge was indeed over, and the headlines would mock it as an overhyped science project. “Foiled: Darpa Bots All Fall Down,” WIRED sassed. “Nobody won. Nobody even came close,” CNN wrote. “The reality of the event did not come close to meeting the hype surrounding it,” said tech site the Register. On that count, Popular Science blamed the feds: “If Darpa was plainly guilty of anything, it was not managing inflated expectations. Instead of billing this inaugural Grand Challenge as a not-ready-for-prime-time field test to calibrate what was needed for future efforts, race manager Negron, in the months leading up to the checkered flag, continued to predict a victor.”

Darpa didn’t have that victor, but an embarrassed Tether chose to focus on the positive. Watching these kids in the desert, competing but coming together, trading stories and strategies, he didn’t worry about the steel carcasses and burned-up tires they had left behind. His great race had proven the motivation to create autonomous vehicles was out there, and it had found the talent with the drive to make it happen. He wasn’t about to give up. A reporter asked him what he would do now.

We’ll do it again,” Tether said. “And this time, the prize will be $2 million.”

No comments: