Gallup Bait and Switch:

The Gallup poll has been criticized a lot recently. Gallup's defence is to point out their excellent record in predicting past elections. Notably the evidence they show is a comparison of their final poll with the actual election result. see below.

Year	Candidates	Final Gallup Survey	Election Result	Gallup Deviation
		%	%	%
2000	Bush	48.0	47.9	+0.1
	Gore	46.0	48.4	-2.4
	Nader	4.0	2.7	+1.3

Now it is, in fact possible to evaluate a polling agency using polls other than the final poll. It is true that deviations of older polls from the election result are partly due to people changing their minds, but this is equally a problem for all pollsters. Ruy Teixera has noted that the Gallup poll fluctuates a lot, even compared to other polls. This means that older Gallup polls are poor predictions for some reason which does not affect all pollsters.

The explanation is very simple. I assume it is well known in the field. Unlike other pollsters, Gallup does a bait and switch (in reverse chronological order). Knowing that only the final polls are used to evaluate pollsters, they put much more effort into the very last polls.

Look at some data on polls in 2000 from polling report here and here.

for example

Gallup/CNN/USA Today Poll
						.
"If the elections for Congress were being held today, which party's candidate would you vote for in your congressional district [rotate]: the Democratic Party's candidate or the Republican Party's candidate?" If "Undecided": "As of today, do you lean more to [rotate] the Democratic Party's candidate or to the Republican Party's candidate?"
		Democratic Candidate	Republican Candidate	Other/ Undecided
		%	%	%	N	MoE
	11/5-6/00 #	49	45	6	2,350LV	+/- 2
	11/4-5/00 #	48	47	5	2,386LV	+/- 2
	10/26-28/00 #	45	49	6	1,858LV	+/- 3
	10/23-25/00 #	44	47	9	LV
	10/16-18/00 #	45	50	5	706LV	+/- 4
	10/13-15/00 #	47	46	7	756LV	+/- 4
	9/4-6/00 #	46	46	8	777LV	+/- 4

Notice the rather dramatic increase in the sample size in the last 3 polls. If there were no change in opinion, this would create a pattern of the poll moving around then stabilizing near the election result. It would look as if people were finally making up their mind, but it is really that Gallup is finally doing a serious survey.

Matthew Yglesias has (as usual) something very interesting to say. He might even be more cynical than I am about polls.

The trouble is that, for media polls, like Gallup, NYT/CBS, ABC/WaPo, etc., the incentives are all wrong. The polls are conducted in order to sell newspapers or attract viewers to television programs. As a result methods, like the Gallup LV model, that seem to exaggerate the effect of opinion swings, are actually preferable to accurate methods since they build the drama. In general, no one has any real incentive to get things right and everyone does have various incentives polling in other directions (if nothing else, there's always incentive to minimize costs).

Part of the trouble, I think, is that people only use the actual election results to evaluate pollsters. However, after the actual election, almost no one is interested in polls anymore, so polling and polling controversies are forgotten till the next election. Therefore it is important to be able to evaluate polls before the election.

Watch this space.