In Which Robert Waldmann boldly goes where Nate Silver says number crunching is pointless and wins a big downer has since arrived to a different conclusion.

Update: The last bit of my original post was an amazing howler noted by Tom Bozzo (thanks).
Nate Silver has now analysed the numbers and came to a conclusion very different from mine. I attempted to extrapolate assuming that all precincts are the same and that Coleman challenges are very different from Franken challenges, but are symmetric -- roughly that challenges by a campaign are likely to be frivolous and equally likely to be frivolous. I conclude that, under those assumptions, the probability of a Franken win is very low. I *hope* that the precincts assumption is false.

In total contrast, Silver analyzes the data available so far without either assumption. In fact, he argues that it might not matter much which campaign issues the challenge (I assumed all were claiming that a ballot counted as a vote for the other guy was no vote). He does *not* assume this. Instead he estimates the effect of challenges by Franken and Coleman on Franken's gain by precinct. Then forecasts by taking fitted values for zero challenges.

So Silver is using way more data than I did (precinct level data) and making much different and generally weaker assumptions.

Now it is very important to note that Silver assumes that there are important differences in the behavior of local elections judges. I really assumed that the differences across precincts would be in the behavior of the campaigns as they are challenging more and more with time.

Silver's key assumption is that the best prediction is the fitted value for no challenges. How might that be true ? It could happen if some local elections judges are incompetent and, therefore, challenged a lot. If the final decision is like that of competent judges, then Silver's analysis is valid. However, it is possible that there are challenges because voters in some precincts cast incomprehensible ballots. If it is about the voters, not the judges, I don't see any reason why the central board will behave systematically different from the local judges. I think that Silver is also, implicitly, assuming that the Franken campaign behaves similarly in each precinct, that is a large number of Franken challenges indicates odd behavior by the local judge, not the Franken campaign representatives. I can imagine another possibility which explains the odd result that Franken gains less in precincts where his campaign challenges more. If the results are going against Franken, his representatives get pissed and become more combative. This means bad performance for Franken (just due to chance) implies a high number of Franken challenges. Clearly, I have to assume that the Coleman campaign isn't like that, that they always push as hard as they can say. If that which varies is the rate of frivolous challenges by the Franken campaign, then Silver's analysis is invalid.

In the end, I think to justify Silver's analysis, one has to assume that, compared to the final judgement, local election judges are biased against Franken and that Coleman's campaign is more frivolous explaining the roughly equal number of total challenges. Variability in local judges bias against Franken (relative to the final judgement) explains variability in challenges across precincts. In particular, local judges in Hennepin County (including St Paul) must be very very biased against Franken. Hmmmm well guess we have to wait and see.

Can Franken catch Coleman in the recount ? The most recent news is very bad for Franken. He is now definitely not on pace to pass Coleman before ballot challenges are judged.

Silver argues that the high number of ballot challenges means that there is no way to put odds on the final outcome. I'm afraid I disagree.

The Strib site gives data on challenges too (I used data as of 00:12 EST November 23). Coleman's campaign has only challenged 28 more ballots than Franken's. As an extreme hypothesis, suppose all challenges are ruled frivolous. Then we get Franken will gain 63 not 35 votes out of those precincts for which preliminary results are reported. With 68% of votes recounted, I don't see how one can avoid placing the probability of Sen Franken near 50%. (no I don't see how 206-35 = 180 either, I guess that the strib had the post recanvass Coleman lead at 215 for some reason).

Most strikingly have been almost exactly equal numbers of challenges by the campaigns in Hennepin county so far 254 and 252 thus the ratio of Franken challenges to Coleman votes is significantly higher than the ratio of Coleman challenges to Franken votes.

Now the Coleman challenges might be more frivolous on average. However, it is hard to give a story about first time voters or whatever that implies that there are systematically more Franken ballots that were missed by the scanners yet systematically more Coleman ballots that can be legitimately challenged.

If the distribution frivolity of the challenges by the two campaigns is similar, further upswings in challenge rates are clearly wholly frivolous challenges ,and if the 2 thirds recounted were typical of the whole state (which they aren't but I don't see any patters to who gains where) then one can calculate odds.

So far there are 1982 challenges. Change + challenges equals number of votes where the optical scanner recorded no vote which are at least debated now. so 635 new possible votes detected. I think the assumption that gives probabilities of victory closest to 50% is that these are two seperate sets of ballots (all challenges are of ballots counted by the scanners).

Forecasts assuming that the remaining third have the same distributions as the currently counted 2 thirds. 318 new votes detected on ballots where the scanners reported no vote and a total of 2973 challenges (not counting the extra challenges from increasingly furious frivolity beyond the current level of frivolity which I assume will be rejected out of hand).

Expected Franken gains at end of recount 95 (just 1.5 times gains so far). Maximum possible variance of number of challenges rejected about 743 (for each challenge has a 50 50 chance and I assume that all challenges are between for Coleman and no vote or between vote for Franken or no vote). Maximum possible variance of "newly detected vote is for Franken minus newly detected vote is for Coleman" less than 318 (this is if there are no newly detected votes for 3rd candidates). so total variance of change 1061 so standard error about 10 and Franken is over ten standard errors short of the Senate.

Update: OOOOOOOOOOPPPS standard error around 33 so three standard errors. How embarrassing. Thanks to Tom Bozzo in comments.

Franken's hope depends on systematic differences between re-counted and non recounted precincts and systematic differences in the degree of frivolity of challenges. No pattern is apparent in vote changes by county. The rate of challenges is similar by campaign.

Bummer.