That may be true, but one of the key ideals in science is that the findings are reproduceable. If I can't reproduce it, it's helpful for the science to be fairly specific, in case the findings depended on details that were thought to be unimportant, but we later find out were critical. The most interesting science comes in unexpected results.
I feel it's like a piece of closed source software releasing "the source code" and yet no one can make it compile. You don't have to have a clean codebase, it can be a bit tricky to build, but it has to be able to be compiled before anyone can have any confidence in your claims. If you don't provide that, it's much less useful.
The separate issues of for profit labs and what support they should expect in this context are tough issues, but we shouldn't undermine the science to thwart them.
Thanks for giving an excellent example. NuPRL is exactly like this, but no one in their right mind would say the group that put out NuPRL hasn't done good science for 20+ years.
To clarify, the NuPRL project is like this in what way? They discuss cs topics and algorithms but no source code exists? or it exists but they don't provide it? Or (unlikely) they provide source code no one can compile?
I don't want to mix up the side issue of "should code-based research always provide source code" with the metaphors being discussed. I definitely think useful CS research can happen without source code needing to exist, which I hope is the NuPRL project's case.
I do feel code based research should provide source code that was created as part of the research when it is material to their claims, including 'we ran a simulation' and similar findings. If you already have that level of detail, you should include it. If you make claims based on data, you need to provide that data. If you make claims based on source code, you need to provide that source code.
Difficult to obtain and get running (I'm told by everyone I've asked). So definitely not the first thing; mostly those last two things.
And yeah, I think that high-quality source code is definitely a pretty impressive feat when researchers pull in off. And I also agree source code should be provided. But beyond providing a VM, I think it's really dangerous to sort of fixate on building software that's easy to setup/use. There's a pretty significant time cost there in some cases, and it's worth asking whether that's what we want to be spending our Science dollars on.
I feel it's like a piece of closed source software releasing "the source code" and yet no one can make it compile. You don't have to have a clean codebase, it can be a bit tricky to build, but it has to be able to be compiled before anyone can have any confidence in your claims. If you don't provide that, it's much less useful.
The separate issues of for profit labs and what support they should expect in this context are tough issues, but we shouldn't undermine the science to thwart them.