ONTD Political

Mitt Romney's Campaign Meltdown. It Was Technology's Fault!

9:31 pm - 11/10/2012
The Sin of Not Beta Testing. You Fail Romney!

The Romney Campaign’s Ground Game Fiasco
They were the gang who couldn’t shoot straight. Romney’s ground-game operation was a disaster—from technology that didn’t work to field operatives who didn’t understand their tasks. The result: Obama won.

When Republican fundraisers solicited the party’s big donors on behalf of Mitt Romney this year, the centerpiece of the pitch was a state-of-the-art campaign to identify the party’s likely voters and make sure they came to the polls on Election Day. Political pros call these county-by-county, block-by-block campaigns the ground game. And while most of the media attention focuses on candidate speeches, debates, and ad buys, it’s the ground game where elections are won and lost.

On Tuesday, Republicans lost the election on the ground. As the Republican Party picks up the pieces from overwhelming defeat, the first fingers are being pointed at a GOP ground game that insiders describe to The Daily Beast as nothing short of a fiasco.

The story starts in 2008. The Romney campaign sought to counter Team Obama’s highly touted, high-tech voter-targeting system, nicknamed Narwhal after the Arctic sea mammal. Narwhal provided the Obama campaign with reams of specific data on voters—finding single women in conservative counties, for instance, or families with children who have disabilities.

In 2012, the Romney campaign unveiled its own killer app and called it Project Orca—the fierce great whale that is the natural predator of the Narwhal. The only problem: Boston’s Orca turned out to be toothless.

The system was different from Narwhal. It was designed to allow Romney poll watchers, in real time, to identify likely Romney supporters who still had not shown up at polling stations on Election Day. By uploading the names of people who had voted, the computers back in Boston could figure out who still needed to be targeted and turned out.

At least that’s the way it was supposed to work. But on Tuesday, it became clear that the deployment of Orca was doing more harm than good. “I think it’s fair to say that pretty much everything about the system that was supposed to work actually failed,” said one campaign official who witnessed the breakdown from the Romney war room on the floor of Boston’s TD Garden.

The Romney high command had cloaked the system in secrecy to maintain what it hoped would be a true competitive turnout advantage. But by limiting the number of people with access to Orca, the campaign was not able to train its field operatives to use it or do the necessary beta-testing to work out the kinks that typically plague new software.

“It did not work perfectly,” said Rich Beeson, the Romney campaign’s political director, in an interview. He acknowledged that Orca crashed in the morning on Election Day. At first the campaign thought the system had been hacked, he said. Passwords and user names for the 34,000 volunteers using the program had to be reset.

But Beeson also said Orca was able to provide voting data on 91 percent of the precincts and accounted for turning out some 14.3 million voters. “At the end of the day I can look any donor in the eye and say we used our resources effectively,” he said. “This is the first time we have attempted to do anything on this scale. By no means was it an abject failure.”

Others who worked on the ground for the campaign disagree, however. In many instances, the voter lists that were loaded on the smart phones of field operatives didn’t match the precincts where they’d been sent, campaign officials said. In addition, there were massive credentialing problems, so Romney poll watchers were not permitted to operate at many precincts. In many rural precincts, poor cellphone coverage made it difficult or impossible for Romney forces to transmit information. Finally, because poll watchers tend to be older, tech-averse volunteers and because there was so little training, many of them simply couldn’t master the technical aspects of the task.

“We were sold on Mitt as this brilliant manager and turnaround artist,” said John Ekdahl, one of those poll watchers in Florida who used Orca. “But it was a snake-oil kind of program. I say this as a Web developer. This was throwing money at a product that just didn’t work.” Ekdahl first published his critique of Orca on the conservative website Ace of Spades. Other poll watchers who asked not to be named had similar complaints.

The mood grew increasingly grim on Tuesday as Romney officials realized that their supposed state-of-the-art answer to the Chicago’s turnout juggernaut was a bust. Walking down the central aisle of the Romney war room you didn’t hear the humming of a well-oiled turnout machine, one campaign official recalled. You heard the panicky tones of operatives flooded with calls from the field about technical snafus and mass confusion.

As campaign officials monitored central computers in Boston, instead of taking in the metrics of a proficient ground game, they saw depressing evidence of a gang that couldn’t shoot straight—anxious messages from operatives who were at the wrong polling place, couldn’t work their smart phones, or were barred from a precinct because they lacked the proper credentials. “It was amateur hour,” lamented one Romney official.

There were other problems for Romney’s ground game in the battleground states. The Obama for America team, for example, had field operations in states like Ohio stay behind after the 2008 election and slowly but surely pick up steam as Election Day 2012 approached. The Republicans closed their field offices after the 2010 midterms.

“We were never going to have the same size and staff as the Obama campaign,” Beeson said. He added that the Republican National Committee was several million dollars in debt by the time Michael Steele left the job, putting resource constraints on the party in terms of operating field offices in states like Ohio.

Finally, the Republicans were never able to match the Obama campaign’s ability to use data from purchase histories, voting registration, and campaign contacts to tailor specific messages to specific voters in a process known as micro-targeting. Republicans first pioneered the use of this kind of data in the 2004 election cycle but mainly used the data to target television ads, direct mail, and robocalls, according to Republican strategists. The Democrats were able to use this kind of data in deploying armies of volunteer door-knockers and others who targeted their voters over time.

Beeson acknowledged that his party will be seeking to learn from the Democrats in terms of micro-targeting. “They have taken this to an organic, micro, micro level,” he said. “We will be looking at how they did it.”

At the end of the day, Beeson said he thought Romney ran a good campaign. It’s just that Obama ran a better one. “With the time and resources we had, the campaign was run very effectively,” he said. “We just did not see them being able to turn out the numbers they turned out.”

Inside Team Romney's whale of an IT meltdown
Orca, the Romney campaign's "killer" app, skips beta and pays the price.

It was supposed to be a "killer app," but a system deployed to volunteers by Mitt Romney's presidential campaign may have done more harm to Romney's chances on Election Day—largely because of a failure to follow basic best practices for IT projects.

Called "Orca," the effort was supposed to give the Romney campaign its own analytics on what was happening at polling places and to help the campaign direct get-out-the-vote efforts in the key battleground states of Ohio, Florida, Pennsylvania, Iowa, and Colorado.

Instead, volunteers couldn't get the system to work from the field in many states—in some cases because they had been given the wrong login information. The system crashed repeatedly. At one point, the network connection to the Romney campaign's headquarters went down because Internet provider Comcast reportedly thought the traffic was caused by a denial of service attack.

As one Orca user described it to Ars, the entire episode was a "huge clusterfuck." Here's how it happened.

Develop in haste, repent at leisure

The Romney campaign put a lot of stock in Orca, giving PBS NewsHour an advance look at the operation on November 5. But according to volunteers who saw and used the system, it was hardly a model of stability, having been developed in just seven months on a lightning schedule following the Republican primary elections. Orca had been conceived by two men—Romney's Director of Voter Contact Dan Centinello and the campaign's Political Director Rich Beeson. It was named in honor of the killer whale as an allusion to the Obama campaign's own voter identification program, code-named Narwhal; orcas are the top predator of narwhals, Romney campaign staffers explained, and they were preparing to outshine the Democratic voter turnout effort.

As Romney's Communications Director Gail Gitcho put it in the PBS piece, "The Obama campaign likes to brag about their ground operation, but it's nothing compared to this."

To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm. The goal was to put a mobile application in the hands of 37,000 volunteers in swing states, who would station themselves at the polls and track the arrival of known Romney supporters. The information would be monitored by more than 800 volunteers back at Romney's Boston Garden campaign headquarters via a Web-based management console, and it would be used to push out more calls throughout the day to pro-Romney voters who hadn't yet shown up at the polls. A backup voice response system would allow local poll volunteers to call in information from the field if they couldn't access the Web.

But Orca turned out to be toothless, thanks to a series of deployment blunders and network and system failures. While the system was stress-tested using automated testing tools, users received little or no advance training on the system. Crucially, there was no dry run to test how Orca would perform over the public Internet.

Part of the issue was Orca's architecture. While 11 backend database servers had been provisioned for the system—probably running on virtual machines—the "mobile" piece of Orca was a Web application supported by a single Web server and a single application server. Rather than a set of servers in the cloud, "I believe all the servers were in Boston at the Garden or a data center nearby," wrote Hans Dittuobo, a Romney volunteer at Boston Garden, to Ars by e-mail.

Throughout the day, the Orca Web page was repeatedly inaccessible. It remains unclear whether the issue was server load or a lack of available bandwidth, but the result was the same: Orca had not been tested under real-world conditions and repeatedly failed when it was needed the most.

All tell, no show

Before Election Day, volunteer training at Boston headquarters amounted to a series of 90-minute conference calls with Centinello. Users had no hands-on with the Orca application itself, which wasn't turned on until 6:00 AM on Election Day.

"We asked if our laptops needed to be WiFi capable," Dittuobo told Ars. "Dan Centinello went into how the Garden had just finished expansion of its wireless network and that yes, WiFi was required. I was concerned about hacking, jamming the signal, etc...Then we were told that we would not be using WiFi but using Ethernet connections."

Field volunteers also got briefed via conference calls, and they too had no hands-on with the application in advance of Election Day. There was a great deal of confusion among some volunteers in the days leading up to the election as they searched Android and Apple app stores for the Orca application, not knowing it was a Web app.

John Ekdahl, Jr., a Web developer and Romney volunteer, recounted on the Ace of Spades HQ blog that these preparatory calls were "more of the slick marketing speech type than helpful training sessions. I had some serious questions—things like 'Has this been stress tested?', 'Is there redundancy in place?', and 'What steps have been taken to combat a coordinated DDOS attack or the like?', among others. These types of questions were brushed aside (truth be told, they never took one of my questions). They assured us that the system had been relentlessly tested and would be a tremendous success."

In a final training call on November 3, field volunteers were told to expect "packets" shortly containing the information they needed to use Orca. Those packets, which showed up in some volunteers' e-mail inboxes as late as November 5, turned out to be PDF files—huge PDF files which contained instructions on how to use the app and voter rolls for the voting precincts each volunteer would be working. After discovering the PDFs in his e-mail inbox at 10:00 PM on Election Eve, Ekdahl said that "I sat down and cursed, as I would have to print 60+ pages of instructions and voter rolls on my home printer. They expected 75 to 80-year old veteran volunteers to print out 60+ pages on their home computers? The night before election day?"

Invalid passwords, crashing servers

When the Romney campaign finally brought up Orca, the "killer whale" was not ready to perform. Some field volunteers couldn't even report to their posts, because the campaign hadn't told them they first needed to pick up poll watcher credentials from one of Romney's local "victory centers." Others couldn't connect to the Orca site because they entered the URL for the site without the https:// prefix; instead of being redirected to the secure site, they were confronted with a blank page, Ekdahl said.

And for many of those who managed to get to their polling places and who called up the website on their phones, there was another, insurmountable hurdle—their passwords didn't work and attempts to reset passwords through the site also failed. As for the voice-powered backup system, it failed too as many poll watchers received the wrong personal identification numbers needed to access the system. Joel Pollak of Briebart reported that hundreds of volunteers in Colorado and North Carolina couldn't use either the Web-based or the voice-based Orca systems;  it wasn't until 6:00 PM on Election Day that the team running Orca admitted they had issued the wrong PIN codes and passwords to everyone in those states, and they reset them. Even then, some volunteers still couldn’t login.

In Boston, things weren't much better. Some of the VoIP phones set up for volunteers were misconfigured. And as volunteers tried to help people in the field get into the system, they ran into similar problems themselves. "I tried to login to the field website," Dittuobo told me, "but none of the user names and passwords worked, though the person next to me could get in. We had zero access to Iowa, Colorado, North Carolina, and Pennsylvania. Seems like the only state that was working was Florida."

As the Web traffic from volunteers attempting to connect to Orca mounted, the system crashed repeatedly because of bandwidth constraints. At one point the network connection to the campaign's data center went down—apparently because the ISP shut it off. "They told us Comcast thought it was a denial of service attack and shut it down," Dittuobu recounted. "(Centinello) was giddy about it," he added—presumably because he thought that so much traffic was sign of heavy system use.

Flying blind

As the day wore on and information still failed to flow in from the field, the Romney campaign was flying blind. Instead of using Orca's vaunted analytics to steer their course, Centinello and the rest of Romney's team had no solid data on how to target late voters, other than what they heard from the media. Meanwhile, volunteers like Ekdahl could do nothing but vote themselves and go home.

This sort of failure is why there's a trend in application testing (particularly in the development of  public-facing applications) away from focusing on testing application infrastructure performance and toward focusing on user experience. Automated testing rigs can tell if software components are up to the task of handling expected loads, but they can't show what the system's performance will look like to the end user. And whatever testing environment Romney's campaign team and IT consultants used, it wasn't one that mimicked the conditions of Election Day. As a result, Orca's launch on Election Day was essentially a beta test of the software—not something most IT organizations would do in such a high-stakes environment.

IT projects are easy scapegoats for organizational failures. There's no way to know if Romney could have made up the margins in Ohio if Orca had worked. But the catastrophic failure of the system, purchased at large expense, squandered the campaign's most valuable resource—people—and was symptomatic of a much bigger leadership problem.

"The end result," Ekdahl wrote, "was that 30,000+ of the most active and fired-up volunteers were wandering around confused and frustrated when they could have been doing anything else to help. The bitter irony of this entire endeavor was that a supposedly small government candidate gutted the local structure of [get out the vote] efforts in favor of a centralized, faceless organization in a far off place (in this case, their Boston headquarters). Wrap your head around that."

Republican campaigners will undoubtedly try to wrap their heads around it for some time to come.

Romney's Get Out The Vote Plan Was a Complete Disaster

As Republicans continue to deconstruct the failure of the Romney campaign, volunteers have revealed that there were serious problems with the candidate's whiz-bang tech solution for getting people to the polls. Prior to election Day, Mitt Romney's team was already bragging about Project Orca, a web app that would connect tens of thousands of volunteer poll watchers in the field with the home office in Boston. It was all part of a massive coordination effort to get Republicans out of the house and into the voting booth.

Poll watching is a time-honored tradition in politics, where volunteers from each campaign literally stand at poll stations with lists of registered supporters and check off the names of people as they come into vote. As the day goes on, they can see who hasn't voted yet and then other volunteers can give them a call or physically go find them and bring them to the polls. Normally, that's done with simple paper and pencils, but Orca was designed to digitize and streamline the process, while also centralizing the effort with headquarters, allowing campaign leaders to re-direct resources on the fly to the areas they were needed most.

On the day after the election, complaints started pouring in from the volunteers themselves indicating that not only did Project Orca not improve the process, it may have actually hindered it. John Ekdahl, a volunteer who writes at Ace Of Spades HQ, outlined the various glitches and breakdowns of the system. Instead of handing out voter lists at local offices, volunteers were emailed 60+ page PDF files and told to print them out at home the night before the election. They weren't given official poll watcher certificates or told that those were required to enter most polling places. The "app" wasn't really an app at all, it was a secure website, creating confusion for volunteers trying to find it in the iTunes store. It also didn't auto-forward users who didn't know to add an S to the http:// protocol in the app's URL (which most browsers don't ask you to type anymore), leaving numerous user lost on a broken webpage.

Volunteers in other parts of the country shared similar complaints. The emailed packets came late or not at all. PINs that were required to login and download the voter lists didn't work and couldn't be reset. Calls and emails to the help desk went unanswered, and the entire system may have just completely crashed in the middle of election day. Frustrated volunteers struggled to get answers that never came, leaving most of them to fend for themselves or simply give up, wasting an entire day without bringing a single new voter to the voting booth.

The whole thing is vaguely reminiscent of the droid army  in Star Wars: Episode I (or The Avengers if Marvel is more your thing) in which a fearsome hyper-coordinated fighting force is controlled from afar by a centralized brain trust ... that gets blown up by a little kid in a toy plane. It's also left a lot of GOP volunteers wondering if the Romney campaign, in an effort to micromanage the all-important ground game, may cost itself the election. By pushing Orca into the most important swing states they drew resources away from more battle-tested methods and left local campaign offices flying blind. No wants to admit that the plan built to put the Romney campaign over the top may have actually led to its demise, but it's hard to ignore the damage done by the broken project. Briebart.com can do the math:

"If each of the 37,000 volunteers that had been devoted to Orca had instead brought 20 voters to the polls in those states over the course of the day, Romney would have won the election."
Page 1 of 2
<<[1] [2] >>
tilmon 11th-Nov-2012 05:43 am (UTC)
To sum up, maybe Republicans should reconsider the value of having experience as a community organizer. Though, honestly, I hope they go right on being ignorant and thinking that social skills are beneath them.
tabaqui 11th-Nov-2012 07:05 pm (UTC)
HA. Yes.
yeats 11th-Nov-2012 05:59 am (UTC)
as an obama volunteer in ohio on election day, this makes me laugh so hard. basically, our strategy was:

1. go to houses that we knew -- based on weeks of calls -- supported the president. at noon.
2. ask if they'd voted.
2a. if they had, strike their name off the list and thank them
2b. if they hadn't, tell them where to go and offer them a phone number to get a ride to the polls.
3. come back at 4:30 to the houses that hadn't voted and ask again, with the same offer of a ride.
4. ???
5. win the fucking election.
thenakedcat 11th-Nov-2012 07:54 am (UTC)
Yup. My neighborhood in Colorado is in deep red country so we didn't have enough volunteers to make a second pass, but the strategy was pretty much the same:

1. Knock the door. If no one answers, leave a hanging flier on the doorknob with the precinct address, voting times, ID requirements, voter helpline #, etc.
2. If the name on the list answers the door, ask if they've voted.
3. If they have voted, thank them for their commitment to democracy and cross them off the list.
4. If they haven't voted, make sure they have the correct info and offer to arrange for any help they might need getting to the polls.
5. If the name on the list doesn't match the person who answers the door, ask if THAT person is registered and has all the information/help they need to vote.
6. Repeat, repeat, repeat until the polls close.
7. Acquire alcoholic beverages and turn on the news.
8. Bask in the glow of victory.
hinoema 11th-Nov-2012 06:46 am (UTC)
I bet I can guess exactly why this happened. They developed this like it was a commercial product for profit, not a campaign tool that needed to be open and accessible. Their capitalist instincts killed them.
bushy_brow 11th-Nov-2012 09:07 am (UTC)
Sounds like that's EXACTLY what happened. Excuse me while I point and laugh.

romp 11th-Nov-2012 07:01 am (UTC)
Interesting. Is this a Republican thing? I remember Reagan with Star Wars and Bush with Homeland Security--both of those hemorrhaged money for businesses (and gov't departments) willing to play along.
shorofsky 11th-Nov-2012 07:19 am (UTC)
I just LOLed at "the Romney high command"...
jenny_jenkins 11th-Nov-2012 07:31 am (UTC)
At least that’s the way it was supposed to work. But on Tuesday, it became clear that the deployment of Orca was doing more harm than good. “I think it’s fair to say that pretty much everything about the system that was supposed to work actually failed,” said one campaign official who witnessed the breakdown from the Romney war room on the floor of Boston’s TD Garden.

As someone who has worked these campaigns for BC (and Canada's) NDP I can't believe (though I'm hugely amused) they would screw this up. You don't need a war-room. You don't need some app. You need volunteers fed on cheap pizza - some of them (with - GASP! - correct credentials) travelling to and from the polls collecting data, another group manning the phones, a group processing the data, and another group with transportation to pick up voters who need a ride.

This isn't rocket science. This isn't something an app can do for you. You need people to back it up. People to process what is sent in. People to drive people to polls. People to telephone (repeatedly, throughout the day) likely supporters to harass them, gently to vote. It can't be a machine or a recording - the personal touch is important.

Pathetic. Hilarious. I loved reading this.
jenny_jenkins 11th-Nov-2012 07:41 am (UTC)
I'm so amused I'm leaving a second comment.

Meanwhile, volunteers like Ekdahl could do nothing but vote themselves and go home.

Ummm...rule #1. Your campaign volunteers vote BEFORE ELECTION DAY. You need them from 6 am until the vote count ends. How could they not know this. RULE NUMBER ONE: Vote early if you are volunteering.

Also, I'm so amused by this obsession they had with the precision of Obama's targeting.

I have no idea if Narwhal can be credited with that, but it's really very simple. You get your elderly volunteers to do areas of the constituency they are familiar with. When I helped on Carole James's first ever MLA campaign (she became leader of the NDP ultimately) it was so basic. You had people work in their own neighbourhoods. Some of these people had lived there 50 years. Not only did they know if Bob next door had voted, they knew how he had voted. And if Bob was a Liberal/Social-Cred party voter mischievously filching a ride from us, they knew better than to send a car to take him to the polls ;)

It's so bloody basic. And the only preparation involved was printing off huge sheets of paper with the voter numbers in each poll, organizing the scrutineers and handing out some maps.

60 pages + ?????

ANY campaign run so stupidly deserves to lose on principle.
bestdaywelived 11th-Nov-2012 03:54 pm (UTC)
Some states don't have early voting. Mine doesn't. That being said, these people are really, really stupid and probably could have early voted and just didn't.
blackjedii 11th-Nov-2012 10:00 am (UTC)
He added that the Republican National Committee was several million dollars in debt by the time Michael Steele left the job, putting resource constraints on the party in terms of operating field offices in states like Ohio.

Republicans in debt?

Scuze me you must have missed the memo about this being the party of Fiscal Responsibility (TM) ok ok.

As a former IT student... hah. You never ever EVER not beta test something, and then test it again, and then give it to someone who can break stuff, and then test it again...

But there is a lot of talk about how the campaign itself may have been fleeced. Not surprised, TBH...
365reasonswhy 11th-Nov-2012 08:22 pm (UTC)
Aren't these the people you'd want in charge though? Campaigning as fiscally responsible, honest and forthright yet up to their ears debt and getting scammed by their own?
pennylane101 11th-Nov-2012 10:33 am (UTC)
to be fair it WAS technology's fault. mitt romney's software hasn't been updated since 1950.
wrestlingdog 11th-Nov-2012 02:20 pm (UTC)
hilsongirl 11th-Nov-2012 12:07 pm (UTC)
Mitt's campaign is just a big ass list of what Not to do
sleeky 11th-Nov-2012 12:13 pm (UTC)
And they had a crappy tumblr.
wrestlingdog 11th-Nov-2012 02:19 pm (UTC)
So much schadenfreude.
aviv_b 11th-Nov-2012 02:43 pm (UTC)
They should have outsourced their ground game to China. After all, the people doing the work don't have to be Americans, they just have to know what the hell they are doing.
shortsweetcynic 11th-Nov-2012 03:10 pm (UTC)
so one of the comments on the atlantic wire story is pretty much MTE:

A chilling preview of what a Romney government would have been like--cheap-ass tech used bureaucratically that would result in widespread chaos and failure and with no accountability.
papilio_luna 11th-Nov-2012 03:29 pm (UTC)
I have actually been sort of curious about the Obama GOTV strategy because my house quite conspicuously was never targeted and I'm legitimately curious about why. I live in a very Blue neighborhood in a very Blue city, voted Obama last time, and gave money last time and this time. I would often come home in the evening to find a door hanger on my neighbors' doors but none on mine. I never got a mailing (though boyhowdy did I get email!), never got a canvasser knocking on my door (I'd see them around the neighborhood but they never came to my door).

I'm not complaining at all--I've got a new baby and 2 loud dogs so the last thing I need is random people ringing my door bell--I'm just wondering if that was on purpose. Like: don't spend any resources on that house, they are fully committed already? Is that how it worked?
bestdaywelived 11th-Nov-2012 03:57 pm (UTC)
I don't know. I'm a registered Green, and I received tons of conservative mail and only a few pieces from liberals, who I actually vote for. When the canvasser came to my neighborhood, he was looking for my fiance and not me.

Are you registered Dem?
Page 1 of 2
<<[1] [2] >>
This page was loaded Apr 20th 2018, 12:50 pm GMT.