I will write a second comment. I will try to respond to your questions. I wasn't an economist in the 70s so (as usual) I don't know what I am talking about.
"So why didn’t this happen? Why did we have a revolution which overturned an existing methodology and temporarily banished Keynesian theory, rather than an adaptation and augmentation of what was then mainstream?"
"Was the attraction of overturning orthodoxy too strong, as it is for a minority of heterodox economists today? "
I think this attraction was very strong. There is something else. A couple of Economists from U Minnesota mentioned Harvard for no clear reason -- I think there was and is an element of Midwestern pride (neither of the economists is originally from the USA). In particular a distasteful aspect of appeals to common sense or judgment is that they are and must be assertions of intellectual authority. Math has the appealing feature that a proof is a proof and does not rely on the authority of the person stating it.
" Did an ideological imperative of dismissing Keynesian ideas play a role?" I think it definitely did. Friedman, Lucas, Prescott and Barro are very ideological. The models change but the policy proposals remain the same. Views on methodology change (Sargent asserted that Lucas and Prescott were all in favor of testing models as null hypothesis until he rejected their favored models).
"To what extent was the hostile reaction of many in the macroeconomic establishment to eminently sensible ideas like rational expectations responsible? "
I think the extremely hostile reaction played an important role (of course, I don't think that rational expectations is a sensible idea).
"Was the attraction of a methodology where at least you could be sure you were consistent too enticing, ? "
Here I would say that mathematics was enticing. Writing papers that look like Physics except the variables have different meanings is enticing.
"perhaps encouraged by increasing segmentation between theoretical and empirical macro? "
I'm quite sure this came later. In the 70s and early 80s new Classicals developed new empirical tools and did a lot of empirical work. I think theoretical macro separated from empirical macro when the data kept saying that new Classical models were not good approximations to reality.
So far, I have made no progress towards understanding how a few departments were so influential. Here I think that hard work and the passion due to fanaticism were important. The general rightward political shift also must have mattered. I think that the loony left managed to cast discredit on Republican Keynesians somehow. But I really don't understand how and why it happened.
I have two other thoughts.
If macro theory (or non theory or ad hoc models or whatever) were about as good as it is would ever be in 1973, PhD candidates and junior faculty would still have to present something new. In fields which aren't progressing a new and crazy original contribution is preferable to an unoriginal contribution. Here I think the trouble is that some questions in macro are too simple and have for decades been answered well enough to serve. I think it is very easy to fit consumption and investment without theory -- 1960s equations fit well out of sample.
If aggregates depend on expectations which are not rational nor adaptive nor anything which can feasibly be modelled, then the problems of stabilizing output and even of forecasting can't be solved. If so the only reasonble thing to do is to try something and if it fails badly try something completely different.
I think many of the questions are too easy and have been answered (but young researchers can't afford to accept this) and the unanswered questions are too hard (I fear unanswerable).
Now my usual boring pointless comment
Here I am again more certain than ever that this comment serves no useful purpose. You support an eclectic approach and not the utter rejection of all of the past 40 years of theoretical macroeconomics. I guess this must seem obviously reasonable to you and to basically everyone. However, while you have very often stated this view, I don't recall much in the way of argument, evidence or even examples.
"Microfounded models could have shown the kind of errors that can arise in more empirically based models when theory is ignored or only applied piecemeal, "
for example ? To be sure such errors can arise, one should point to examples of such errors arising. The classic example is the 1960s era Keynesian ad hoc assumption that the Phillips curve is a structural relationship. This error never happened. It may well be that going to the data armed only with common sense and verbal arguments leads to errors which would have been avoided if one used formal theory. But to claim that such errors occured, one should cite them with authors, dates, titles, Journal titles volume and page numbers.
"However I also think the new ideas that came with that revolution were progressive. I have defended rational expectations, I think intertemporal theory is the right place to start in thinking about consumption, and exploring the implications of time inconsistency is very important to macro policy, as well as many other areas of economics. I also think, along with nearly all macroeconomists, that the microfoundations approach to macro (DSGE models) is a progressive research strategy."
I note that the word "progressive" appears twice. I think it is very useful, too useful. To argue that something is progressive, one need not point to achievements. The word is consistent with the claim that the approach will in the future yield valuable fruits. I recall reading in 1982 Sargent expressing the hope that realistic micro founded models which could be taken seriously and confronted with the data would be developed in the following 30 years. I say time is up.
I also note "to start". Now the claim that something is the right place to start can't be disproven. The claim is that something good will happen. I know of no evidence against the view "(6) Changes in expectations of the relation between the present and the future level of income. — We must catalogue this factor for the sake of formal completeness. But, whilst it may affect considerably a particular individual’s propensity to consume, it is likely to average out for the community as a whole. Moreover, it is a matter about which there is, as a rule, too much uncertainty for it to exert much influence." Keynes 1936 chapter 8 section II.
I have 2 3 final questions.
1. What evidence could possibly convince you that intertemporal theory is not the right place to start in thinking about consumption ?
2. What evidence could possibly convince you that DSGE isn't a prgressive research program ?
3. What could the data do which they have failed to do in the past 40 years ?