Does the HTTP standard or robots.txt specification mandate the collapse of conse... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

rlpb on Sept 11, 2014 | parent | context | favorite | on: Google search indexes itself

Does the HTTP standard or robots.txt specification mandate the collapse of consecutive slashes, though? I agree that it is common, but if it is server-side implementation detail, then a correct implementation of robots.txt should not collapse them, as they might mean different things to a particular server.

kentonv on Sept 11, 2014 [–]

I agree. If there's a bug here, it's in the server which collapses slashes seen in request paths, not in the indexer's interpretation of robots.txt.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact