#1 will never work well for the simple fact that when people know that they are being asked to pick something without actually going through the mental process of evaluating options and picking a goal, they will not pick in a way that correlates with a larger population that doesn't know it's being tested.

The best way to test something is to try it on people that actually want something and voluntarily are picking something and going towards a goal... and don't know they're being tested. The proof is in the pudding: if the copy works better than a control, it should produce a sale/signup/whatever.

Asking a bunch of people who are working for pennies if they want to buy some product that they have no clue about and are not in the market for, based on a copy they don't even understand well, is a waste of money and would even do you some damage since your customers might be more sophisticated.

Now, if you positioned it for simple, general websites, it might work better. But then again, those sites are not high paying customers.

PS: Look into Google Optimizer: http://www.google.com/websiteoptimizer

Also, you have no way to check answers. MTurk workers might just pick random to increase their earnings. Random results will just create noise. It's even worse if you have just two options.

I suppose a principal question is whether the results tend to extrapolate and generalize.

no, statistics doesnt work like that (sadly).

I kind of like it. I've worked in contexts where the "decision makers" write terrible verbose copy. It would be nice to quickly see "but here is what the data shows"

For that matter, forget mechanical turk. Get some really bad ass copywriters working freelance for you. Give me a text form where I paste in my A/B copy and an optional explanation... I hit submit and within 30 seconds a professional copywriter has chosen the better copy (extra $ for an explantion etc). Charge $2 per review and make it really simple. I've been in enough "copyfights" that would have easily been worth $2 to resolve. "mechanical turk for copy"

If you go through the trouble to hire professional copywriters (even freelance), you could offer addons as well; like $0.0x/word for professional spell/grammar checking (and other services).

Re #1 - drop monthly and increase per-use fees (to $10-20) and your idea will find its audience.

Monthly fees drag the customers in a long-term relationship that they most likely don't want to be entering. Setting the per-use fee that low serves no purpose. If you want to go after customers with no money, you will get them. Instead however you could go for those who can pay, and you would still be able to acquire them, since the idea works for them as well.

Who would pay $10 or $20 per-use fees ... you?

#1 and #2 are more or less the same thing. I'd phrase the service as A/B testing, and then give #1 and #2 as some of things users can do with it

You need a way to narrow down, among all the people using Mechanical Turk, people who actually fit the user's target demographic. You need this right from the get go.

e.g. What people in India think of my craigslist add for something in North Carolina is totally irrelevant.

Also, I agree with another poster, who basically stated that you should make a generic comparison service, and let people compare anything, whether they be two domain names, two versions of a site, or whatever.

Is this a product you would actually use - the generic comparison service? Assuming we had proper demographic targeting.