What has Nate Silver done for us lately ?

This isn't really about Nate Silver.  It is about poll aggregators in general and just as much about the Huffington Post pollster, Real Clear Politics and Talking points Memo as about Silver, Wang Linzer and others who try to estimate outcome probabilities and not just average polls.

I think that aggregators have proven that it is better to look at their aggregates than to look at individual polls and try to see patterns or calculate averages from memory.  I think this means that their judgement to not include a poll would be damaging to the pollster.  I conclude that, if they work together, they can demand and obtain transparency.

I have a real problem with what I perceive to be a lack of transparency of pollsters. I refer really to the methodological explanations they have on their web sites.  I do not have the impression that I could reproduce their numbers from the transcripts of their interviews based on their explanations.  I do not think this is acceptable.

My sense is that this is not just my ignorance -- that pollster methodology is partly secret.  Here I stress that I might be wrong -- it might be that the full protocol is not secret and I am just ignorant.

My understanding is that pollsters keep their methodology partly secret claiming it is a trade secret -- that since they are the best pollster, all other pollsters would imitate their methods if they weren't kept secret.  If they do make this claim, it is BS.  They make different choices and each is, as noted, convinced they are right. My guess is that they don't describe the assumptions they make, because if they did then people would understand that the assumptions are assumptions and make a difference.  I think that, most unfortunately, most poll consumers are willing to accept a black box, buy a pig in a poke (and mix metaphors).

I think this is not a healthy attitude and that poll consumers including journalists should have a rule that no special sauce is allowed.  I have no hope that journalists will adopt this rule.  But I really hope that poll aggregaters do.

I note that academic journals have this rule.  They demand the raw data and an explanation of calculations such that they can be replicated by editors (doesn't mean they do replicate them and doesn't mean they don't ). This is actually a fairly new development.  Editors of academic journals can impose such rules as their decisions are very important -- basically research doesn't count without their approval.

I think a conference of leading poll aggregators would be useful and that a decision that polls with secret methodologies are to be ignored would make a huge difference.  But I don't think their power to impose transparency on pollsters will last.  Right now they have high status, because they totally out predicted the non quantitative go with their gut pundits.  I'm pretty sure that in 6 months the fact that they nailed 2012 will be old news which can be ignored.