Quite a few commenters here on Alan Abramowitz' study of the negative impact of conservative ideology on Republican incumbent senators' general election campaigns from 2000 to 2008 wondered if he might do a counterpart analysis of Democrats over the same period.
From your lips to Alan's computer: he's done the counterpart analysis, and found that ideology does not seem to have had much impact on Democratic senatorial incumbents from 2000-2008; being more liberal didn't hurt them, all other things being equal. The numbers are clear, but the reasons for the very different partisan results are not at all obvious.
I'd guess that voters want different things from the President and their senator. Senators want the same thing (to be elected President). In particular, the Senate votes on taxes and spending, but, in practice, lets the President handle foreign policy without their advice and consent. I'd guess the median voter (in about 40 out of 50 states) is a nationalist semi-hawk who wants to soak the rich, that is, a populist.
This is my proposed explanation for everything in US politics. The median US voter is strongly egalitarian and well to the left of the median congress person on bread and butter economic issues.
The evidence (always the same) is that fact that at least 60% of people in the USA want to increase taxes paid by rich people (start at pollingrepot.com and search for taxes but the answer also appears in questions on HCR and social security). As hard as the median poll respondent is on the rich, she is harder on corporations. The wild enthusiasm for increasing taxes on corporations convinces me that my country-people have not studied tax incidence (OK I knew that already).
So one guess is that Senators are running for President and the first rule of running for President is don't be George McGovern. Note that's about war and peace and not taxes.
Another theory is that polticians balance pleasing the public and pleasing lobbyists. This makes their voting record more favorable to concentrated interests than the public wants. On the rare occasions when the public notices, incumbents lose. Conservative Republicans who really think that what's good for the concentrated interest is good for the public are the only ones who go so far as to be noticed. This last is an explanation of how the median Congress person can be well to the right of the median voter.
OK a third theory (and maybe a fourth). The problem isn't why liberal Democratic senators get re-elected. The problem is that, given that's what people want, why aren't more elected in the first place ? Here I'd guess part of it is that Americans are ideological conservatives and operational liberals. So if a non-incumbent candidate can be labeled a liberal, he or she is in trouble. Once voters see what being a liberal senator means in practice (voting just like almost all of the other Democrats) he isn't. It's hard for a socialist to be elected to the US senate. It's easy for a senator with Sanders's voting record to be re-elected.
A variant on this is that the DSCC, political consultants, Democratic strategists, the DLC, Major contributors and, above all Rahm Emanuele believe that liberals can't win statewide elections, so they have trouble winning the nomination unless they are incumbent. Then they do fine in the general election. The belief in the party elite that the voters don't like liberals can keep the Senate delegation well to the right of the political sweet spot.
And yes the phrase "Democratic strategist" in that list was a dig at Ed Kilgore personally. He is a hero palladin of the truth, because he reported a fact which tends to undermine the case he has been making for years.