That sounds plausible. I have heard a similar story about the ruins of the Roman baths at Bath, Somerset in medieval times: the locals thought that they were built by vanished giants.