banks came to them with a swelling array of rocket-science securities and paid them to provide a rating. Because these structured securities were complex, the fees for figuring out the rating were high, and it turned into a lucrative source of business.
The conflict of interest is obvious: banks wanted high ratings and the agencies wanted lots of deal flow. That meant they were motivated to push the envelope in order to insure a steady stream of business. After all, if you get a little too picky about things, clients will just head across the street to see if a different agency might treat them a little better.
If only the ratings agencies had waited for financial firms to come through their doors bearing rocket science securities the conflict would have been less severe.
The ratings agencies decided to consult too (remember how well that worked out for Arthur D Anderson). So they charged large fees to help financial firms design financial instruments . This was a new practice and the blatant conflict of interest was obvious. It was so obvious that reckless SEC head Chris Cox was alarmed (you know the guy who decided to relax capital controls on investment banks). Nothing was done because of our friend Chuck Schumer (see your post below).
This created a huge problem totally aside from conflict of interest -- that is a problem which would have been huge if all ratings agency employees and shareholders were saints who were indifferent to money. Assume it was so. Even under that assumption, the agencies used the same models to design assets which they thought were safe and to evaluate whether those assets were safe.
These models turned out to be very bad approximations to reality. That happens (I'm an economist and boy do I know that). However, there was no second opinion. If sincere financiers had designed securities whose riskiness was then assessed by ratings agencies, there would have been two groups of modelers and a better chance that they wouldn't both be grossly wrong in the same direction.
On the conflict of interest, I don't think it is rating shopping. That would be easy to solve. Just define major ratings agency and say that if one wishes to claim that an instrument has been rated then one must have it rated by all of them (I think you can even name the major ratings agencies without it being a bill of attainder since this is good for them). No more shopping.
Getting a secret rating from just one would not be a tempting proposition ("it doesn't work if you keep it a secret --your not supposed to keep it a secret" -- Dr Strangelove).
However, I think the problem mostly remains. If all three of the ratings agencies had been tougher on new rocket science assets, the whole huge industry would never have existed and all of them would have been poorer. A reasonable rule would be that no asset gets AAA unless an asset which is identical except for maturity dates paid on time and in full in each of the past 3 recessions -- that is no AAA for new stuff for decades -- no exceptions. Obviously with or without agency shopping, they wouldn't have done that.
So I don't really have a solution.
OK I have an idea. A big bank (Goldman Sachs) writes a fixed premium CDS. These aren't auctioned. GS says we will sell you a CDS on this asset for this price. They have good reason to do research obviously -- their money is where their mouth is. Regulations refer to the price of the CDS not to the rating. I mean the requirement for transparent CDS pricing with a requirement that CDS are offered to the public at a price (like an IPO) could make ratings agencies obsolete. I think it would work in Nash equilibrium.