I think the main reason they failed was the price. When you could get the fastest, most loaded Corvette for $20k less, who's gonna go with the XLR? Obviously not enough. Caddy was so all-fired determined to butt heads with Mercedes, they priced themselves out of the market.
Also if Caddy tried to imply the XLR was a better as well as hotter Corvette, that was a big mistake. This car is a boulevard cruiser, not a dragster. Anyone who bought an XLR and expect it to whip the Vettes, was in for a big surprise. This probably lead to a dampening of enthusiasm and word-of-mouth.
And the quality is nowhere Mercedes-like. I love my 06, but she ain't no Benz. The quality of the furnishing is much too cheap. Way too much flimsy plastic.
Having said all that, I enjoy the heck out of ours, and she will be with us for the rest of my days. I still have folks ask, "I heard Cadillac was coming up with a new 2 seater. Is that it?" When I tell them it's 16 years old, they're amazed. Still a fresh and exciting looking car.