> Under these conditions, large scale refactorings or framework upgrades can cause massive test failures
An even bigger problem is that large scale refactorings can result in the tests themselves no longer being correct (which isn't the same thing as “not passing”), if you test at too fine a level of granularity. However, when you wrote these tests, you couldn't have possibly foreseen a large scale refactoring 18 months into the future, so how do you tell in advance what the right level of granularity for your tests is?
An even bigger problem is that large scale refactorings can result in the tests themselves no longer being correct (which isn't the same thing as “not passing”), if you test at too fine a level of granularity. However, when you wrote these tests, you couldn't have possibly foreseen a large scale refactoring 18 months into the future, so how do you tell in advance what the right level of granularity for your tests is?