Pollster.com reports on interesting results from Pew

Pollster quotes pew

Pollsters are continuing to monitor changes in telephone use by the U.S. public, since most surveys are still conducted using only landline telephones. Growing numbers of Americans are reachable only by cell phone, and an even larger number who have both a landline and a cell phone may be "functionally cell-only" because of their phone use habits. The latest Pew Research Center national survey, conducted June 18-29 with a sample of 2,004 adults including 503 on a cell phone, finds that the overall estimate of voter presidential preference is modestly affected by whether or not the cell phone respondents are included. Obama holds a 48% to 40% lead in the sample that includes cell phones, and a 46% to 41% advantage in the landline sample. Estimates of congressional vote are the same in the landline and combined samples.

The numbers noted above are based on interviews with registered voters. When they narrow the universe to more likely voters, however, the difference disappears:

Narrowing the analysis to voters who are certain about their vote choice, there is almost no difference between the landline and combined samples: Obama has a 38%-28% advantage in the combined sample, while the margin is 38%-30% in the landline sample.

I analyse the bit on registered voters.

Am I interpreting the phrase "the sample that includes cell phones" correctly to mean the sample of 2,004 only 503 of whom were contacted by cell phone ? If so, backing out from the rounded results presented by Pew suggests that 54% of those contacted by cell phone support Obama and 37% support McCain from whole sample average equals landline + (cell phone - landline)503/2,004.

Now the variance of obama-mcain in the landline sample is less than 1/1,501 and the variance in the cell phone sample is less than 1/503(some people are undecided so for Obama and for McCain aren't perfectly negatively correlated). Bit more rounding up and the variance of the difference in the two differences is less than 4/1,503 so the standard error of the difference is less than
0.2/root(15) which is less than 5%.

The difference in the difference (using rounded numbers) is about 17% or 3.4 times the overestimated standard error. Now clearly one shouldn't round to two figures then do the calculations, but that is a very large ratio and makes me think that the results by cell phone are strongly statistically significantly more favorablet to Obama.

As far as I can tell from the snippet you quote, the 503 are not cell phone only customers or functional cell phone only customers. I should read the report (should have before posting this) but it seems that they are just people contacted by cell phone. Thus if one were to assume that the only difference between results with people contacted by cell phone and people contacted by land line is due to the fairly small estimated number of people who can't be contacted by a land line, such contactable only by cell phone people would have to be even more overwhelmingly for Obama than the 503 people contacted by cell phone in the sample.

I'd say that Pew is determined to conclude that they don't have to worry about cell phones which are a hassle.

I actually think the different result with "likely voters" which is, I would guess more than calculate given rounding, not statistically significant tends to re-enforce the view that using land lines causes pollsters to miss a group of the population which is very pro-Obama. The reason is that "cell phone only" like "not likely voter" stands for young and (maybe) poor.

I think an even larger study of cell phone vs landline should be done once. Call a cell phone number -- ask if the owner has a land line, ask what is the chance that the owner would be reached by landline at the same hour on a weekday, on a weekend, ask what is the probability that the owner would be reached by cell phone at that
hour on a weekday (weekend).

The ask age, race, gender, party affiliation, and likely voter questions. This would make it possible to calculate correction factors to apply to results from landline only polls. In particular, the party affiliation proportions in the population may be changing but don't bounce around as much as candidate support. Getting a correct party affiliation estimate once is therefore useful for a long time.