We study the link between the probability of default implied by Credit Default Swaps (CDS) spreads and the final prices of the defaulted bonds as established at the CDS settlement auctions. We observe that the post-default recovery rates at the observed spreads imply markets were often 'surprised' by the credit event. We find that the prices of the bonds that are deliverable at the auctions imply probabilities of default that are systematically different than the default probabilities estimated prior to the event of default using standard methodologies. We discuss the implications for CDS pricing models. We analyze the discrepancy between the actual and theoretical CDS spreads and we find it is significantly associated both to the CDS market microstructure at the time of the settlement auction and to the general macroeconomic background. We discuss the potential for strategic bidding behavior at the CDS settlement auctions.