But it seems that to "do it right," the writer must be able to pick the right anecdotes and draw the right conclusions from them. This is probably close to impossible if he's not an expert in the area. And if the writer's an expert, then he should be able to provide other (non-anecdotal) evidence to support his claims as well.

I think that's why many of these books are so frustrating to read -- they seem to rely almost completely on anecdotes, without any sort of underlying data to look at long-term/overarching trends/themes. Thus, instead of saying, "here are various data, including these representative anecdotes about some of them, and so we can see that my theory is true," these writers usually say "this is my theory, and look how well these carefully chosen anecdotes match it!"