This is the largest practical issue I've had with developing a Scala project.
One major cause of the problem seems to be that Scala is stuck with the "precompiled bytecode in JARs" model of software distribution; while this may be appropriate for Java (where core developers go through great pains to maintain compatibility between versions) it seems like a poor fit for Scala. Not only do the core library APIs change between versions, but the conventions used by the compiler to "link" Scala code together are also altered, for good reason.
An alternative distribution mechanism that's more resistant to these kinds of issues might make sense. Even just distributing source code and build files (like, say, Ruby) could be a better solution. Building dependencies might be slow at first, but centralized caching could address this. Another solution might be an intermediate representation (after type-checking and other compiler phases, but before "linking") that gives the Scala compiler the knowledge that's missing from compiled class files.
Even namespace-mangling or class-loader magic could be helpful here. Loading multiple versions of a single dependency should in theory be possible, since OSGi manages it.
Sadly, there seems to be a pretty large investment by the Scala community in the whole Java/Maven/JAR ecosystem. It'd take a concerted effort by many people to create a more robust solution.
Thanks for this comment, as not yet having got into Scala myself, I had to read between the lines to figure out what the problem was here. I thought "Ok, you don't have binary compatibility, so recompile your dependencies. Sure, it may extend your build time somewhat, but it shouldn't make it impossible to use dependencies or test pre-release builds..."
Oh, you don't _have_ the source? Well, there's your problem.
A good reminder of what some of us take for granted in our development environments, I guess.
You have the source, but the Maven model used by the whole Java ecosystem for distribution/management of packages doesn't work like that. And replacing it with something else isn't a small feat.
> You have the source, but the Maven model used by the whole Java ecosystem for distribution/management of packages doesn't work like that.
Oh, I don't know about that. Clojure does this; shipping source is default, and shipping bytecode is only done in cases of certain Java interop forms.
But Maven has a notion of "classifiers" that seem to be a good fit for this. I've seen them used to specify things like "this is the JDK 1.4-compatible version", so I wonder why Scala library authors haven't used them to publish the same artifact compiled against multiple versions of Scala.
'Jar' files are actually zip files with a '.jar' extension. On my Windows machine I have .jar associated with a zip application, very handy to quickly inspect its contents.
I think you've hit the nail on the head, that having a centralized package manager tool/host, a la the haskell hackage / cabal-install or the like, would substantially simplify the problem, but because of desired JVM library compatibility this path isn't explored as much as might be ideal
One major cause of the problem seems to be that Scala is stuck with the "precompiled bytecode in JARs" model of software distribution; while this may be appropriate for Java (where core developers go through great pains to maintain compatibility between versions) it seems like a poor fit for Scala. Not only do the core library APIs change between versions, but the conventions used by the compiler to "link" Scala code together are also altered, for good reason.
An alternative distribution mechanism that's more resistant to these kinds of issues might make sense. Even just distributing source code and build files (like, say, Ruby) could be a better solution. Building dependencies might be slow at first, but centralized caching could address this. Another solution might be an intermediate representation (after type-checking and other compiler phases, but before "linking") that gives the Scala compiler the knowledge that's missing from compiled class files.
Even namespace-mangling or class-loader magic could be helpful here. Loading multiple versions of a single dependency should in theory be possible, since OSGi manages it.
Sadly, there seems to be a pretty large investment by the Scala community in the whole Java/Maven/JAR ecosystem. It'd take a concerted effort by many people to create a more robust solution.