osprey - funny you should mention cannibalism
In addition to the form you mentioned (big companies with deteriorating pipelines swallowing up smaller companies with more promising pipelines), there is another one -- companies deciding it pays more to simply sell more compounds into the same space, just varying them by a little, and focusing on marketing and advertising to stimulate demand. Easier to raid a known pot of gold from the next-door neighbor than go down panning on the river.
More importantly, this is where more and more R&D money is being diverted. Given the way these companies have shrunk their true R&D investment, it's no wonder their pipelines are shrinking.
I'd like to put an asterisk on your statement that "drugs have become harder to find" -- and say that drugs have been harder to find using the same 30 year old discovery and development techniques, and harder to find in the same focus areas that have been "mined" for the past 50 years.
Part of the problem here, is that these companies don't want to step out and do what they once did.
We've got major problems coming down the pike on bacterial infections and with basic anti-biotics. It's sickening to see the corruption of these companies grown fat, lazy and afraid, actually doing their best to cover up these serious public health issues and challenges because they'd rather just knock off each other's latest relatively inconsequential allergy medicines.