Hacker News new | past | comments | ask | show | jobs | submit login

"who are you to tell Google they can't index a third party domain?"

No tweets link to third-party domains, since Twitter wraps all links in their own URL shortener (t.co). Even if you use an independent URL shortener, it will be wrapped by t.co.

(I suppose you could argue that there is no way for Google to determine this algorithmically, since "twitter.com" != "t.co", so it should go ahead with the crawl, but there's the question of how Twitter would respond to that.)




They could create a robots.txt for t.co and set it to noindex.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: