I will toot the author's horn for him. He has a fine series of Scala posts on his blog [1] and his book Hands-On Scala Programming [2] is a great introduction to building real applications with Scala so that any experienced developer can understand and extend them.
I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case even when that made it hard to understand how they were used for the 2 actual cases we needed to handle. Akka, Cats, and Shapeless were all over the place.
Those earlier employees have churned off to other places and I have successively simplified the code they wrote that is still useful, while encouraging the use of no more language power than necessary in new development. Hands-On Scala Programming is the book I give new hires as a language introduction that shows the sort of style to be preferred. It's much more like super-powered Python than like Haskell.
I have written C, Fortran, JavaScript, Python, and Scala for money. When I started on Scala I had never written Java nor used any JVM language. I have come to really appreciate the rich ecosystem of JVM libraries, the instrumentation and profiling tools I get, and many aspects of the Scala language and standard library. I love Scala's collections and miss their power and ease when I'm writing Python. (Which I still do for certain scripting tasks and for accessing Python-ecosystem libraries.)
> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case
Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.
> Scala the language is quite fine, but it attracts this kind of programmers.
It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected.
I am now retired, but Scala is still my goto choice for engineering software. It checks all the boxes on concurrency, bulk I/O, ecosystem. In the hands of a sane and pragmatic practitioner, it's hugely productive.
Well, Scala compiles to JavaScript and native through Scala.js and Scala Native. Plus you can also compile to native with GraalVM Native which works very well.
Interesting, in my experience long compile times in Scala equates with a bad code base. And invariably involves an uber object with a whole lot of implicit conversations and is imported on every other source file.
I think the pertinent part of the comment you're replying to was graalvm, not so much Scala. I was compiling a small Scala codebase that took 15 minutes to compile - most of that was not my code, it analyses the reachability of the entire classpath and compiles it to native machine code.
I don’t know about bad code but a lot of derivations that send compiler into proof loops usually is what is causing slowdowns. You can write the same code by hand, pick your poison. Honestly, scalac is probably still faster than typescript compiler and somehow no one complaining about ts
> I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language.
I think this is a bit contradictory as it is phrased. Learning the language is essential to use it effectively. Otherwise one is better off with a language already known. And it is very possible to build things to learn the language and still keep it sane.
If we went back to let's say 2014 "when the cake pattern was the solution of all problems in world" I think I'd agree with your statement, even back then when I was a bit of a Scala fanatic.
I speak monads. I am a 50 yrs old high school dropout. Cool thing is - just because you are not seeing FP structures in non FP focused languages doesn't mean they aren't there.
I worked at Databricks in the past and Scala is just a way to get things done there. Honestly, for me writing code day-to-day wasn't that different from Java once you understand how to structure your classes. Handling nullables with typed Options is pretty nice though, and I miss it in other languages now. Most people who join Databricks do not know Scala beforehand, it's just a useful tool.
Sure, sometimes progress is slower than you'd like, but that's the case in every large monorepo.
> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case even when that made it hard to understand how they were used for the 2 actual cases we needed to handle. Akka, Cats, and Shapeless were all over the place.
I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.
Yeah that’s got to be up there on the list for dumbest technical related rejection I’ve ever heard of, particularly if the code was done well otherwise.
If I'm being honest here, a quite significant part of day-to-day backend JVM software development positions is dealing with the build tool. While I wouldn't outright reject a candidate for not being familiar with the latest standard toolchain (Gradle), I would certainly consider it as one of many factors - Gradle is pretty complicated, yes, but also quite powerful in what it is capable of achieving. It's not as eyebrow-raising as, say, if you sent in a coding challenge compiled using Ant (which would, in fact, be quite a fun discussion to have with a candidate) but it would certainly be something considered.
> If I'm being honest here, a quite significant part of day-to-day backend JVM software development positions is dealing with the build tool.
perhaps, because your work has over complicated it [1].
[1] If you are doing multi-distributions or multi-repo stuff perhaps it's warranted, but those complex build features better be providing serious value to the business to be worth losing 10% of the development workforce's time on build tooling.
While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage.
Alternatively you can work in Go where braindead simplicity is the mandated norm.
Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for.
I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it.
The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.
And when you tried to tell them that abstractions don't come for free, but with a cost because they make code harder to understand, the answer for the last 25 years has always be the same, although rarely said straight: that it was only hard for -you- to understand, and not for the enlightened master himself.
I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.
And the problem is that this mindset continues to be cultivated by CS books and conferences. Few people want to read a book that tells them that the secret of being productive as a team depends on mostly the culture in the team and the simplicity of the code, and not on the latest hyped framework, language or paradigm.
> I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.
Agreed, and what did we get? dogmatic decrying of how OOP is completely useless and objectively bad, not too dissimilar from some of the comments on FP and Scala on here.
> dogmatic decrying of how OOP is completely useless and objectively bad
And just who were the ones pushing this? The people peddling FP.
The people in this thread are not pushing anything. They're just sharing their experiences with Scala and how unproductive it is to deal with FP zealots.
> before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.
Ah! Just write a quine and be done with it!
> I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.
"cake pattern" on one side and scalatest on the other.
As for
> I’m constantly surprised to see this type anti-intellectualism.
You hardly need any Category Theory to theory to understand type theory or mostly important Scala's type system. This the exact knowledge and allows you use it effectively. Yeah, the way he puts it makes those who like to understand the theory behind everything look bad.
For most tasks programming it's more an engineering discipline than an intellectual pursuit. In that context complexity must always be justified.
I know, and have been told, to have written hard to read code. In my case it's usually vectorized code in numpy or a C or cython extension for really hot code. But I always have a good reason, usually performance when it matters.
It's usually said that premature optimization is the root of all evil, but it's nothing compared against premature abstraction. Optimization at least is local in its nature while abstraction tends to expand all over a code base, and when the assumptions made for the abstraction no longer hold people is still forced to keep dancing for a music that no longer plays.
"Educate yourself" assumes that your interests are everyone's interests but this is a vast field and not everyone has interest nor time to learn about category theory. Most probably your coworkers are quite intelligent. If they are not interested in your idea it may be that it's not appropriate or maybe not correctly framed, or not mature.
I'm stealing that phrase:
"Premature abstraction is the root of all evil".
It's hard for me sometimes to justify in a code review why an abstraction is not required (yet) when someone has put some effort into it.
The thing is I DO understand category theory and FP after going off the deep end in Scala land. It did not make me a better programmer or make writing code any easier or faster. In fact it makes me spend 50% of my brain power on juggling those concepts and how they are encoded into a language that does not and never will support them as well as say Haskell.
I learnt the basics of FP a long time ago and understood the data-centric view of computation, the benefits of immutability etc - those things actually do help me write better code. The advanced FP, not so much. The nightmarish encoding of advanced FP shoe-horned into a language not built to support it that compiles into Java definitely not built to support it actively hurts. There are so many dirty macro shenanigans and hacks in the guts of the FP-purist libraries to force the language to implement things it just isn't particularly well suited to implement, which makes reading or debugging library code a nightmare. Like if you want Haskell, just use Haskell FFS and let us code reasonable Scala in peace.
There are so many parts of math that are much more useful in various fields - basic optimization in solving sudoku puzzles[1], linear programming in Z3, gradient descent in ML, calculus & vectors in graphics programming etc.
Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are way more useful day to day. Once you’ve mastered calculus, linear algebra, probability theory, statistics, numerical analysis, optimization, algorithms, complexity theory, combinatorics, asymptotic analysis—sure, go pick up category theory.
But Category Theory is different to all the other examples you gave, as it describes program composition. i.e. building larger programs from smaller ones. Computer scientists have been searching for the "mathematics of program construction" for a long time, desperately trying to make the process more formal. Category Theory is what many seem to have arrived at. There are even books about it, e.g. "The Algebra of Programming".
Most computer science graduates are well schooled in algorithms, complexity, linear algebra etc. However, the vast majority of their time commercially will not be spent implementing some core algorithm, but instead gluing systems together and consuming/writing APIs. Any mathematics that helps them better accomplish this would be good to add to their toolbox.
The author of "The Science of Functional Programming" book (https://github.com/winitzki/sofp) makes an argument that CT is at best tangential to practicing FP.
I found category theory to be helpful in giving a vocabulary for some very simple and essential things that otherwise get lost. My goto example are Monoids and Semi-Groups, which are almost mindbogglingly simple but pop up everywhere.
Category theory gives you the precise language to talk about such stuff and helps to communicate and talk about more complex concepts and that's always a plus.
I sympathize with your thinking. There are two "readabilities".
1. How easy it is for an average programmer to understand the code.
2. How understandable the code actually is if the reader is fluent with that prior knowledge.
If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conclusion drawn.
---
But my experience is that those techniques do not yield better code.
So, I’ll probably get shit on for this, but I really cannot wait to leave this industry.
I hate how “best tool for the job” really just means “what we can have a large hiring pool of people who’ve only ever learned one programming style and possibly only one language of” or “for legacy reasons this is the only choice”.
Yes, it's disappointing but hardly surprising to see the phrasing of management parroted by so many developers. It's their job to worry about hiring, deadlines, etc. It's yours to make sure things don't shit the bed at one in the morning, especially if you're going to be paged about it. There are some languages that make it very easy for grads to deliver features, and others that are far more likely to be correct at compile time. The "best tool" is born from these competing demands; those too meek to engage with this inherently adversarial process don't do themselves or anybody else any favors.
Apparently "best tool for the job" means find he bottom of the barrel that can still engage somehow with the codebase. This Idiocracy went to the point where returning function pointers gets questioned during code review because "this might be too complex compared to imperative call".
The code base had multiple problems, none of which I would blame on category theory or Cats.
- Engineers had written higher abstractions seemingly just because they could. When I audited how internal libraries were used across our services, calling applications weren't making use of the advanced abstractions. I'm talking about things like using Cats to abstract across AWS S3 error handling. Cool, except that it wasn't needed because we never actually encountered the exotic compositions of failures anticipated by the libraries.
- The abstractions written for our own business logic were worse than abstracting over S3. They were premature. Our business logic had to change frequently because the end user experience was still evolving rapidly. Changes that violated previous assumptions and their corresponding abstractions took longer than they should have and/or led to very awkward code.
- At least at the time, tooling had more problems with the "advanced" code. The IntelliJ IDEA Scala plugin could not yet show how implicits were used. It couldn't find senders sending to an Actor the way it can easily find plain callers of a function. You would need to manually force a "clean" in certain modules before code changes would compile as expected. IDEs would also fail to flag code that couldn't compile, and incorrectly flag code that would compile, at a higher rate compared to plainer Scala.
I'm still glad that I have access to Cats, Akka, and other advanced parts of the Scala ecosystem. They're still used in a few places where their value is greater than their cost. Even in the plain code, I'm still very glad I have pattern matching, immutability-by-default, rich collections, map, flatMap, filter, fold, scan, find, etc. I have no plans to transition our company off Scala internally. If I were starting a greenfield project with myself as the sole developer, I'd probably be using Scala for that too. But I prefer to write a bunch of simple repetitive code first, then develop abstractions after it's clear what the commonalities are.
There are others options besides Go and category theory. I'd say that for now, languages that are like ML are enough. They're coming in the mainstream with pattern matching, immutable records, etc. It'll take some time for people to learn this, learn to use it properly, teach it to others. And maybe some years later a monad will be a pattern as common and well-known as an iterator. But today isn't that day yet, and until now you have to collaborate with your peers. You also have to assume that you may be wrong about some of this stuff, and keep an open mind for FP alternatives. For example, I think algebraic effects aren't part of category theory, yet they're an exciting new feature that might have an impact in "industrial" languages one day.
There are plenty of companies where everyone has a very mature understanding of monads and where new joiners are coached to reach that level of understanding.
And there are plenty of places that aren't using source control, CI and unit tests.
If all you do is add numbers together, is it anti-intellectual to question why one needs to grok Principia Mathematica first rather than just doing some arithmetic?
In other words, do you solve problems that actually require category theory or are you just navel gazing?
This is development nowadays, hire people fresh out if am eigh week javascript boot camp and everyone has to write code they can understand.
Instead of bringing the juniors up, we drag the more experienced guys down to their level.
The result is really shit basic code everywhere. or you end up using a language like go where there's basically only for, if and arrays and writing code in it is miserable and tedious.
This ain't anti-intellectualism. Scala fanatics (more like FP fanatics who use scala) write overly completed code just for purity's sake. Its ok, but it creates a maintenance nightmare and a huge hiring problem.
I've seen a variety of Scala codebases from awesome to terrible (and a lot in between). Guess what, they all use the same Scala. Scala offers you many design choices that work well in skilled hands, but because it is so non-prescriptive, is is also very sensitive to individual talent and engineering culture. If you can't or don't want to maintain a team of strong developers, save yourself and use some other technology. But if you can, and you're working in a complex business domain, Scala can be hugely productive and result in very reliable software.
Just my counterpoint to all the negativity that people invariably feel compelled to pour out in any thread about Scala. Tons of devs use Scala productively without deep dives into monadic code golf, you know.
> Scala offers you many design choices that work well in skilled hands
I agree that Scala works well in some skilled hands (see Li's libs), but the people that write the most terrible Scala programs actually know a lot about Scala. Scala has a unique way of encouraging genius programmers to write crazy code. People that know little Scala can just use it as a "better Python" language and actually write amazing Scala code.
> Scala programmers generally frown on programmers that want to use it as a better Python and I think it should be the opposite.
It's hard to make generalizations like this that are meaningful. There are a number of different Scala ecosystems, as you probably know. Spark is really a world of its own. Lightbend's libraries form their own ecosystem, as do Typelevel's and ZIO's.
My own opinion is that using algebraic data types (sealed traits, final case classes) with a sprinkling of type classes (implicits) where needed is pretty easy to work with, and that's the style that Typelevel is using. It is mostly inline with Li's "Least Power" post, though using IO pervasively instead of Future.
Is Typelevel's approach a better Python? I don't know. I don't really know what that means, and I'm not convinced that Python is a great model to follow. I see you posting along similar lines in many Scala threads on HN, so you seem quite invested in the language. I think it would help make your point clearer if wrote up a bit more of your thoughts, perhaps as a blog post.
The only Typelevel library I've used is frameless, which I blogged about here: https://mungingdata.com/apache-spark/frameless-typed-dataset.... Don't think the lib is really usable. Typelevel stuff is not what I'm referring to when I say better Python. I'll try to make another blog to clarify.
I think Scala is a better language than Python. But for some reason it just seems like Scala is full of mind bending code that I can't understand. Full disclosure: I am way less smart than lots of the Scala ppl I've worked with.
Agreed on Frameless. I think Frameless was an experiment that didn't pan out. I don't think Scala 2 is capable of implementing a usable system that does what Frameless attempted to do (a typed data frame). Scala 3 might.
I find what most people mean when they say "simple" is really "familiar". When I first started teaching most programmers weren't familiar with first class functions and they were "complex". Now almost everyone has been exposed to them. Similarly I'm seeing more students are learning FP at university and thus find FP techniques "simple".
I always find it funny how ppl will whine about Scala's complexity but then when Scala's "complex" features get re-implemented in java(usually in some bastardized fashion) they are god-givens
> Scala has a unique way of encouraging genius programmers to write crazy code.
Scala is a popular platform to explore FP concepts without fully understanding their caveats or having enough respect for the rest of the team's ability and willingness to follow, I'll give you that. But that's only because it's a great platform to explore FP concepts in general!
You have to reign in the "geniuses", that's what engineering culture is for. I really don't like that label for people writing crazy code, btw. They shouldn't be on a pedestal just because they lack the empathy to write maintainable code.
The pythonic-Scala approach is certainly gaining steam in the ecosystem. The problem is, it's hard to see because it's boring and doesn't require many special tools – Scala has much of that built-in. So the popularity of this approach is not apparent if you just look at open source activity. But internally companies who work with Scala use it in very benign ways for the most part, in my experience at least.
Still. For some data scientist who knows nothing about programming, or for some CS student getting started with his first programming assignment - which is easier - Python or Scala? I'd say Python, by a lot.
> But if you can, [...] Scala can be hugely productive and result in very reliable software.
Yes, but, in my experience, the productivity gains are not that big. Strong developpers can and _will_ fare well in any technical environments, therefore choosing Scala over anything else won't give your team any advantage over a "simpler" technology. It's actually riskier to pick Scala, because you will increase your recruitment difficulty by a factor of 100-1000 (less developers available in the pool, even less when you only consider skilled developers, and huge barriers to teach and bring up to speed anyone that haven't worked in Scala before). What do you think will happen when the "good" developers are gone ? A rewrite in Python.
Absolutely not! I'm serious, and I stand by it. Rust ecosystem is like a better Python. You have to consider why someone is willing to move away from Python and into a foreign language: Developer supply, performance, safety, ecosystem, political reasons.
I can't cover all cases so I'm going to focus on the one that makes the most sense _from my personal point of view_.
I do AI Software Enginering and Devops when I'm not project managing. So Python is the only and primary choice for all things related to Data Science, dashboards, APIs, pipelines and transformations (on data).
Why and where would I choose Rust ?
In data pipelines, for performances and memory efficiency. This also ensure we have proper boundaries and documentation (APIs, file formats, schemas) between Data Science-related code and pipelines.
In Dashboards and APIs, for improving type safety, and handling more concurrency on background computations than python without resorting to other frameworks (Celery, Airflow, ...).
In data versioning and supporting infrastructure, for additional type safety and mostly making sure that _nobody_ who is not a software engineer won't cross component boundaries by accident. These components are often very delicate and essentials for a working platform. Changing them can have ripples into other parts of the project, and I only want trained engineers on these.
In the opposite, direction, I could pick Python over Rust when it's most appropriate: Restricted developper availability, improving coordination with Data Scientists, single container deployments, existing experience with framework or ecosystem...
The main blocker for not using Rust isn't because the language is new or more complicated, it's because Python offers more advantages to a situation. At the difference from Scala, I can pick _any_ skilled developper and put him from Python to Rust without any side effects (appart from the training part). This is because writing Rust code is much more consistent at all skill level than Scala. There no "flavor" of Rust. There no bickering about "coding style". The Rust tooling is best in class (Scala's SBT is actually worst-in-class, I believe). You can use the same IDE in Rust an Python (VSCode). This leads to a very similar coding experience between writing type-hinted-Python and Rust. This is not the case between Python and Scala.
You can use Scala with VS Code just in the same way you can do it with Rust. Tooling support for Scala is world-class in many ways. I think your knowledge is pretty outdated.
I haven't used Rust but my understanding is that memory management is explicit, which doesn't sound great for general business applications. I've used Java, Python, JS, Kotlin, Go and Scala, and found Scala to be the best at representing domain models in a type safe and readable way.
We develop the majority of our code in scala at my company (in fintech).
There seems to be an experience in this thread that training in Scala is difficult. For anyone who is considering and wants another perspective - I have found that junior engineers tend to get up to speed more quickly in Scala than in other languages and that I can give them more freedom (because I know their programs won't compile if they don't line their types up and our use of the type system very immediately forces them to deal with the possibility that an operation can fail, e.g.). I can simply look at types and give them a nudge in the right direction (heyoo -fp either joke-). Yes they suffer a little in the beginning, but are usually shipping code within a few weeks then their trajectory really begins to rocket. It's very cool to watch them grow to building complex systems in a complex domain so quickly. As they grow the ease of refactoring means that it is easy to change their prior work as they gain more understanding, so for our management the penalty of allowing them to learn in the wild about something that's suboptimal is pretty low - they will come back and fix it. Of course what I'm saying above is really about FP and type systems and an implicit (heyoo -scala joke-) presence of thought about how our systems work top to bottom and how we train people. I think that is also required no matter what tools you use or what you're doing in any organization?
>> I can give them more freedom (because I know their programs won't compile if they don't line their types up
In what meaningful way can you give them more freedom? When i think about how i spend my time with junior folks, it tends to be on things like:
1. Do they understand enough of our problem domain that i can begin to describe a problem that needs to be solved?
2. Have they understood the problem i described? How confident am i that they're not going to implement a solution to the wrong problem?
We don't spend time on "hey, you passed the parameters in the wrong order here".
To put it another way based on my experience, what's more useful to a junior developer?
Option A: Hey, here's a test case, here's how to run it. I need you to change the behaviour of this component to do X instead.
Or
Option B: Hey, here's a component with well designed types. I need you to change the behaviour of this component to do X instead.
My experience has been that Option A - play with the component hands on, try things, run it, tweak the data supplied to it and observe the results - these encourage engagement. The developer will be excited even.
And for Option B, they'll be quite dejected, not that invested in the project. I can't even compile the stupid thing because of some cryptic compile error.
When I say more freedom, I primarily mean freedom to focus on the problem domain and freedom to contribute in a major way more quickly. Upon reflection, I realize that my expression about optimal solution was about optimality in modeling and understanding the problem domain vs. wrestling organizational programming pattern you're trying to learn / hold in your mind while also trying to solve the domain (which you don't have to do as much with scala's compiler)
And in terms of morale, I view almost the opposite of what you're saying. Because types and the compiler enforce correctness in the program and
compatibility within our systems I can have a relatively junior person thinking more about the domain and knowing that they can get something "real" into prod much more quickly with almost no risk to the org. They can move on to bigger blocks of work more quickly. Solving real, big problems and making impact gets engineers jazzed up. Regarding compilation, I find the "this won't compile" phase is almost non-existent in practice. People quickly adapt from "this thing doesn't compile [insert sad longing for python]" to "this thing won't compile - somewhere there's a misalignment with what I said I'm going to do (via types) and what I'm actually doing, [insert silent acknowledgement that they would have shipped an error in python and not known it]."
I'm not seeing where the gap is between A and B except that in option A it appears that there is no compiler involved, which per the above, means that I have to ask them to focus more on exercising mental discipline about coding conventions (boring, low value add work) and also I can't allow them to contribute as quickly because the risk they threw an exception somewhere they shouldn't have (e.g.) is higher.
>> they can get something "real" into prod much more quickly with almost no risk to the org
What are the most common risks to an org (originating from its code) and how do types get you down to almost no risk?
When i think of common risks, most of them don't even originate inside the process's address space so the type checker is completely disconnected from the risks (e.g. hey - you misunderstood what we wanted here...). For the risks that do originate directly from the code, it's stuff like hey - your solution is quadratic time, sure your code compiles and your tests all pass but i'm telling you this won't work under prod data volumes.
I'm trying to steelman your argument and the best case i can come up with is an IO monad.
Scala is a great language when Li Haoyi ecosystem of libraries is used. His libs do crazy Scala stuff under the hood, but expose a clean end user interface. The problem with Scala is that most ppl don't use his libs.
It's so annoying working on a Scala project with dependencies that all use different JSON libs.
Even new, awesome projects like Delta Lake still use Scala dependencies like Scalatest and SBT that should be replaced with Li's libs in my opinion (utest & Mill).
Scala could be an amazing ecosystem if the community unified around Li's libs. It just doesn't seem like that's going to happen. Li wrote the "whats wrong with SBT" article back in 2017 and it seems like everyone is still using SBT.
I wrote an article on the maintenance challenges with Scala projects that trended on HN a while back: https://mungingdata.com/scala/maintenance-nightmare-upgrade/. Li's libs would solve a lot of these issues cause most of them are intentionally dependency free.
It was frustrating for me to see how Li basically singlehandedly fixed the Scala library ecosystem, but how none of the projects I actually used depended on his libs. I ended up leaving Scala after several years even though I like the language when it's used a certain way.
Scala is such a powerful programming language that has so much potential to be mainstream, but it's just so abused by "genius programmers".
I seriously love Li Haoyi Scala libraries especially the JSON one.
The hilarious thing is/was that Scala tutorials/books would show writing JSON parser as one examples. Yeah Scala makes it easy to write a JSON parser but you still do not want to roll your own.
Yea, there was a JSON parser in the Scala standard lib at one point and it was then deprecated & removed. Guess it's easy to make a "hello world" JSON lib, but maybe not the best addition to the standard lib, haha.
agree.. I really like scala and Li's blog and book have helped me immensely. I use it mostly for spark, but I've found it really useful now for other things. The concurrency really is pretty nice (compared to other languages). His API's are written in the same way as his blog and his book: zero condescension; he's not constantly trying to impress you. This seems to be the problem with the language.. the language allows you to really go nuts if you want to and the result can be an incomprehensible type system that takes a week of hacking just to figure out what it wants you to do. That said, json4s isn't terrible.
In my personal experience, Scala in 2021 is de facto best language for business logic. Effects are easy, parallel code is beautiful and easy to reason about. Ecosystem is big and no other languages comes close to productivity I have with Scala.
For example, its very fast and easy to prototype code with ADTs and scale that code to big codebase. I rarely have to run my code and if I do - it most of the time works as intended because of the type system. Its like magic. In other languages if you want to write code without bugs you are stuck with TDD or a lot of tests. With Scala usually few tests are enough, everything else encoded in its expressive type system.
I would say the only big negative I see is using Scala as Haskell. With tagless final, Kleisli and so on. If you do - it becomes very hard to onboard new people and I am not sure the positives outweigh the negatives.
Tagless final is a good way to enforce the principle of least power, which has many advantages when onboarding inexperienced developers. It can become unwieldy if abused, but that's true for any style.
And if it really bothers you, then you can use your concrete "super monad" everywhere. It's the only way with ZIO, but you can also do that with other effects.
Regarding Kleisli, it's such a broad abstraction that you'll find examples everywhere, whether explicit or not. In practice, even with heavy usage of libraries like fs2 or http4s, it's mostly an internal implementation detail, in most cases you'll only see type aliases.
There is development, then there is academic curiosity and code golfing. I have trouble understanding how people who say Scala is bad in the same breath praise the reflection like there is no tomorrow Java alternatives or Go’s ocean of text.
There are pros and cons in every tech. Maintaining a large Scala code base is going to be much easier, less to write, less to maintain, easy to refactor.
Don’t use a library if you have no need for it but don’t dismiss it as fanatical unless you have actually compared what the same functionality in your fav tech looks like.
Choose the right tool for the job, don’t aim it at your foot. Get the job done and have a side project for “fun” coding.
Scala is a damn good option for targeting JVM. Don’t blame the tool.
I completely agree with their pragmatic approach to Scala. Scala is a very powerful language, and you can do super crazy stuff. My approach to writing Scala code is similar:
1. Write the code like a better-Java/typesafe-Python (that is, imperative style).
2. Glance through the code. For collections, usually there's a more concise approach.
3. Read the code again. If you think what you did in #2 is making the code _too_ concise, you might need to think twice (or add comments).
I don't think so. Kotlin is an unmaintainable soup of features: it can match all the selling-point examples of Scala but only by adding ad-hoc special cases for each one. (For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approaches). In practice Kotlin codebases still use magical incomprehensible reflection (Spring Boot) and magical compile-time manipulation (Kapt), so you have the worst of both Java and Scala worlds.
Are you sure you're not confusing Kotlin with Scala?
> For example, Kotlin has null safety and it lets you write code using errors-as-values style "either" types - but it has two completely separate syntaxes for these things, and so it's impossible to interoperate or reuse code between those two approaches
And that is a problem how? Stick to one style.
> In practice Kotlin codebases still use magical incomprehensible reflection (Spring Boot)
Switching an API from "a result or nothing" to "a result or an error message" happens all the time, and switching in the other direction is only slightly less frequent. And of course most programs have some APIs where one is appropriate and some where the other is. So consistency is valuable.
I'm arguing my lived experience of working on Kotlin codebases. It's not about the specific framework, it's that people always use a framework that breaks the rules of the language, just like in Java, because the language itself isn't expressive enough. Whereas the thing that makes all those complex Scala libraries worthwhile is that they enable you to make a whole enterprise-scale codebase using just plain functions and values.
> If I had to describe the style of Scala written at Databricks, I’d put it at 50% Java-ish, 30% Python-ish, 20% functional
This is interesting as in taking a different approach to most other Scala shops I’ve experienced so far where die-hard monoid/functors fanatics take over and make the code base completely incomprehensible to outsiders / new hires. This seems to be a much more sensible approach.
die hard functional scala is the exception not the rule. this isn’t really constrained to scala, language zealots can exist in any language. the real problem is a lack of understanding what maintainable code is
Yes, over-engineering is common in many languages, especially Java, where Spring and OOP patterns would be deployed everywhere. It is not a unique problem to Scala.
Scala was a terrible technology that was adopted by my organization five years ago and were still paying the price.
How to deprecate this at scale?
The extent to which i like scala is as a dsl for describing etl jobs with spark and i prefer the RDD api.
Engineers are consistently crippled by it, we dont even use shapeless or scalaz. Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit.
In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem.
I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :)
> In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem.
It's funny that if you tell Cat-purist that Spark is functional-style, they would laugh at your face :)
That's an unhelpful thing to say. Most people agree that the Spark scala DSL is functional style. It's not as advanced in the style as other scala ecosystems but it has been a very good on ramp for people discovering functional style and is widely celebrated for that
Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.
To discard Cats as 'basically Haskell ported to Scala' is doing it a bit of a disservice. Yes, it explores pure functional programming, but that design space is huge.
> Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit.
Yes it is. Deprecate using it and migrate away from it. I hate that these shitty overengineered Cucumber wannabees are what "Scala" means to most people.
It's a great language IMO, but you need to make sure fancy frameworks pay their weight. IMO Shapeless and Scalaz-or-equivalent do - replacing reflection or AOP with plain old compiled code, even complex code, is a huge win. But SBT and the mess of Scala testing libraries are just huge complexity for marginal benefit.
As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced.
Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark.
The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after the first transition from 10K lines of java+sql to 1K lines of scala.
Never experienced something like this before or after.
At the next place I wrote 400 lines of C++ mapreduce over 3 days, which could have been 40 lines of sql or 10 lines of scalding/spark.
Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around?
Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.
No. There was no transpiling to SQL. This was several years ago, before Spark SQL.
The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs.
HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop.
The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA.
In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.
Most of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig.
Spark jobs were native RDDs before the SQL Spark era.
Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify scalas pluses. Ymmv.
I think scala can absolutely account for that complexity, like it is not a one-way street. It can scale both up and down on expressivity. But I agree that often times it is not used that way.
I wouldn't say spark is terrible, but it is a sort of a "reverse bicycle". If you put a spark codebase down for a week or two and pick it back up, it feels like you're learning everything all over again.
It's true that some features of Scala have made it to Java, but I am not sure the gap between the two languages has shrunk much. For example while Java has added first-class functions and closures, it is still clumsy to attempt any serious functional programming approach with Java; Scala's type system is still way ahead; Scala compile to the JVM, JavaScript, and now native via LLVM.
> having libs tied to what version of the compiler you're using
It's never been that. However libraries used to be tied to the major version of Scala (for example 2.11, 2.12, 2.13) as major versions were not binary-compatible with each other.
Scala 3 addresses this thanks to the TASTY format which creates a strong foundation for binary compatibility. [1] You can use Scala 3 libraries with Scala 2.13 and, more importantly, Scala 2.13 libraries with Scala 3.
It saddens me to see that the HN crowd had been diluted to such an extent in terms of engineering quality that straight up luddism has become the norm, often accompanied with silly ad hominems.
One thing I didn’t see in the article was that Scala is really god for refactoring code. It gives you a lot of great tools, that allow you to do things which would be impossible in other languages. Dependency injection is really natural in Scala, which can help a lot with refactoring code, both when it was being originally written and when it’s in the process of refactored later. Also, advanced features like implicits, can actually save your butt in certain situations, even though people hate them because they don’t understand them.
Also I agree with everyone that so many people write the worsttt Scala.
I love Scala by the way and have written hugs code based on it. However the reason why Scala can encourage bad code from smart people is that it allows people to max out their budget for writing smart code.
It’s like, let’s say it requires 2x more intelligence budget to read/debug than it is to write code. So if we all have more or less an intelligence budget of like 10, maybe some it’s 12 or others even 15-20, and you write crazy intelligent Scala, like at a 6, then you’ve already maxed out most peoples ability to read and debug the code, as we all more or less have a 10.
Scala let’s you write code that maxes out these budgets if you’re not careful. Sometimes, very rarely, it’s necessary, but most of the time there is a simpler, dumber way to do the same thing that is actually much better.
For most of my career in Scala, when I had conflict with other engineers, it was over this kind of stuff. They insisted on using cats or scalaZ or writing these crazy complicated, overly generic abstractions, and it’s like… I get what you’re doing and even I think it’s stupid and could just be done more directly and more attainably for other people.
> For most of my career in Scala, when I had conflict with other engineers, it was over this kind of stuff. They insisted on using cats or scalaZ or writing these crazy complicated, overly generic abstractions, and it’s like…
While I kinda agree that you have to be careful with your abstractions, I see no harm getting familiar with some of the concepts that these libraries use.
Being able to just implement a Monoid and then use it to do nested aggregation in maps is pretty neat, for example.
It's a fairly accurate predictor of a company downturn after getting sunk in total complexity and ensuing certain employee churn.
The only reason for Scala adoption was Spark 10 years ago. Scala/Spark engineers were pulling the top buck. With pyspark and other cloud big data options on the scene, no one sane is using scala for this anymore.
Spark wasnt even released 10 years ago :) I think Twitter, Linkedin, and some others gave Scala adaptation big push. Twitter in particular, when they killed their fail whale by moving to Scala/JVM
F# has some nice features that scala is lacking, such as a more terse syntax for ADTs or type providers.
But other than that, Scala is miles ahead. And if you want to do pure FP, then Scala is the only option.
Also about that video, almost all of the points mentioned are either not available in other languages like F# anyways, or they already fixed in the latest Scala version.
> Scala is the worst of the category of "functional looking languages built on the top of an imperative ecosystem".
Could you elaborate? Because that's not been my experience at all. I have found Scala to provide many tools to wrap around the idiosyncrasies of the underlying imperative that they are barely noticeable anymore.
To be honest with you my social circle of software engineers gave up Scala long time ago and we do not even consider it to try it ever again. Most of the projects we do we simple use Java for JVM (since version 8 there are many useful features that you can use and have relatively terse code that is also easy to read) for the .net ecosystem we use C# of F# depending on how relaxed the engineering team and the stakeholders about it.
For data engineering (ETLs, glue code, etc.) Python works. Since Spark has a Python interface it is also not needed to use Scala there.
For frontend obviously Scala was probably never a serious option.
There is no use for Scala for me or for the companies I work for. The last time I tried Scala it was horrendous. SBT? Seriously? Binary incompatibility between minor versions? I could go on an on of the collection of anti-patterns I have encountered. Since I have limited time and there are a ton of other software environments that we can use and have much better experience why on Earth would I want to waste time on Scala ever again?
Paul Phillips misses the point, and the "more than one way to do it" complaints are superficial. There's no language that prevents you from writing two functions that do the same thing, and that's all that most of those examples are (e.g. a -> b is just calling a function that returns (a, b)). Most of the time there aren't multiple semantic ways to do the same thing, which is what matters.
SBT is awful, don't use it.
"Minor versions" is overstating things - 2.11 was released in 2014, 2.12 was released in 2016, 2.13 in 2019, and 3.0 onwards are backwards compatible with 2.11. So that's one compatibility break every 2-3 years, which is better than most languages.
Scala is the only vaguely mainstream language I've found where you can do a whole enterprise-scale project in the language itself - no reflection, no AOP, no decorators, no macros, no monkeypatching, no nothing. So until I find another language where I can do custom context-like types (e.g. a type to represent "this must happen in a database transaction", and libraries already offer me functions like "take this list of database actions and combine them into a single database action" that work with my custom type) and "walk the object graph" style tasks (e.g. JSON serialization) in plain old type-safe code, I'm sticking with Scala.
It's not experience with Scala at all, but we all have different journeys. Should you ever find yourself using Scala again, I hope you make the same pleasant discoveries as I have.
I worked at a company full of Scala zealots and elites. It was insanely unproductive and people spent a lot of time trying to design around all the floors in Akka, trying to get Akka working in modern cloud computing environments and the like.
I think at some stage specialists from some Akka pro shop had to be air dropped in to try fix it all, but it didn't help much.
It always made me laugh how well all the services written in Python and Go worked and how easy they were to extend, operate and deploy comparatively.
I've never seen a case where the actor model actually helped with the business problem. Every time I saw Akka used, replacing it with straightforward Futures-and-for/yield code was an improvement.
Cats et al are frequently abused - 90% of the time rather than using a complex tool to solve a complex problem you want to think about it a bit and simplify it. But when you need them they do things that you can't do any other way, and they're vastly better than the reflection/AOP/bytecode-manipulation that people use to solve the same problem in Java or Kotlin.
> I've never seen a case where the actor model actually helped with the business problem.
I've inherited and built some actor based applications and libraries, both in Erlang and .NET (via Akka.NET). Usually these systems are:
- heavily Domain/event driven
- have 'funnel points' (example; bids on specific cars)
- have strong (but not extreme) requirements around consistent, fast response.
- be reasonably maintainable without a whole lot of gotchas
In all cases the actor model was the easiest way to solve the problems presented. That last point may be a bit contentious, but what every shop I've been at has found that while sometimes domain evolution results in extra boilerplate code to handle different versions of events, it is usually not the same sort of nightmare to do larger redesigns of the system as it ages and needs evolve.
> Every time I saw Akka used, replacing it with straightforward Futures-and-for/yield code was an improvement.
I'll admit I've over-used Akka.NET from time to time. I'd say the most overkill case is 'I need background workers running on timers with automatic recovery.'
Not using Akka, but I have a side project right now being done primarily in Erlang that felt natural to architect with Erlang “actor model” and the OTP (not the only reason I chose it of course, Erlang’s history and certain other features from that history made it a natural fit). There’s a small JVM-integrated component as well, but I’ll likely just be developing it in Java as it’s really just involved implementing an old JSR and I don’t foresee more advanced feature coming too much in handy.
Can’t speak to category theory. Not familiar with it, but I have done my fair share of “reflection metaprogramming” that might get me me shot (dynamically constricting and using generic types through reflection gets pretty nasty).
The standard library is where modules go to die, just like in Python. And unlike Python, there's a good build/dependency manager available for Scala (Maven). Keep that stuff out of the core language, let it be libraries released on their own schedules.
JSON has a standardized spec and the API you design has to do a few things: Marshal and unmarshal.
How frequently do these need to change?
The problem with the library landscape is that one library will be promoted by the purists and another by the company with the money. Suddenly you have to learn both to understand Scala codebases.
Well literally today I got a notification from GitHub that I need to upgrade the version of jackson-databind that one of my project depends on because of a security vulnerability. And apparently that's the 8th such fix to this patch version of jackson-databind.
Well, you can always use tried-and-true Java library like Netty, Jackson, and plain-old JDBC API. You get to enjoy all the goodie like `val`, `case class`, `pattern matching`, etc... without all the headache of those pure functional libraries.
Those are the least confusing libraries in the Scala ecosystem lol. Slick is just like any ORM-light framework. Akka-HTTP abstracts away all the Akka-specific things and you just create routes that return Futures essentially. How is spray-json complex? You're just using macros against your case classes to create (de)serializers.
If anything, you should be saying things like Shapeless provide the extra complexity.
Go is at the same level of complexity. JS I can agree with but we're starting to compare completely different things at that point. Type systems make things more complex at the benefit of safety.
Having used both Scala and Go for the past 6 years, I can tell that for the purposes of creating a database backed web API, they're definitely not at the same level of complexity.
This isn't about the type system. This is about complicating things by making everything FP while getting nothing in return.
That's the great thing (and the point of many of the comments in this thread) about Scala - you don't have to do any FP with Scala if you so choose. You can operate in a completely OO fashion and gain all of the benefits of the language (primarily the type system) for free.
People promoting this view fail to understand that people work in teams and it only takes one FP zealot to force everyone to do FP and your PRs don't get merged unless it's idiomatic Scala.
Jackson has its own parallel "type registry" to make up for not being able to use something like Shapeless, because the problem still has to be solved. Getting a failure at compile time if there are any cases that wouldn't serialize properly is a huge advantage; a nontrivial proportion of production outages I've seen were Jackson.
What Shapeless gives you is the ability to treat value-like types - case classes and sealed traits - generically. And so you can do "walk the object graph" style stuff at compile time - particularly, typeclass derivation.
https://circe.github.io/circe/codecs/auto-derivation.html is one popular library that does this - note that the `.asJson` will fail at compile time if you add a member to Greeting or Person whose type can't be serialized/deserialized (e.g. File or Lock or something) i.e. one that doesn't have an Encoder typeclass instance available.
It amazes me that such big projects use monorepos.
My company also though about doing so, because we have a few typescript projects that share a common module with business logic and everything lags behind when the common module was upgraded, but I don't know if I should sell them to switch over to a mono repo as well.
My biggest pro argument would be the speed up in development and review time, because everything is in the same PR/MR and not split into each project
Are there any huge downsides?
Navigating well designed, huge monorepo Scala codebases is surprisingly easy. Spark is a great example of a huge monolith that's relatively easy to work with considering the complexity of the app.
But I've also worked on lots of large Scala codebases that are completely incomprehensible and painful to change. Scala allows for devs to make beautiful monoliths, but that's not usually what happens.
Scala is an amazing language I love writing software in. The tools it offers to model business domains, the power of expressive types make it next-to-none when it comes to shape, remodel and refactor code.
Then there's the ecosystem. Sure, you have the whole of the Java ecosystem, but then there are also tons and tons of Scala libraries that are so well designed (Quick shout-out to Lihaoyi here; your libraries are amazing!) and work together so well is astounding.
I was very worried with the migration path to Scala 3 but that turned to be out pretty much painless. Scala-2.13-backward-compatiblity is amazing and together with scalafix the transition is as smooth as anyone could reasonably expect.
Anyone getting into Scala or worried about complexity with Scala should have a good look at https://www.handsonscala.com/
My experience with "Scala at scale" was extremely unsatisfactory and I found Databricks to give very unhelpful responses to basic performance problems arising from the intersection of Scala and Spark, bordering on gaslighting. Memory bloat is one of the most pernicious issues, you have to handle large data very carefully to not end up with massive memory footprints that outstrip any available hardware (and causes hard to debug cascading failures).
could you elaborate please? I'm especially interested in the Play/Akka part.
So what would the alternatives look like, using common Java Frameworks instead or none at all?
Depends on what you're trying to do. For example I think that Lightbend is no longer planning to maintain Play. I would probably go with Spring as a replacement.
For Akka, do you really need it? You have to be building a pretty specific kind of system
to need it over simpler constructs built into the language.
I work at a small company that has been using Scala for 7 years. Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case even when that made it hard to understand how they were used for the 2 actual cases we needed to handle. Akka, Cats, and Shapeless were all over the place.
Those earlier employees have churned off to other places and I have successively simplified the code they wrote that is still useful, while encouraging the use of no more language power than necessary in new development. Hands-On Scala Programming is the book I give new hires as a language introduction that shows the sort of style to be preferred. It's much more like super-powered Python than like Haskell.
I have written C, Fortran, JavaScript, Python, and Scala for money. When I started on Scala I had never written Java nor used any JVM language. I have come to really appreciate the rich ecosystem of JVM libraries, the instrumentation and profiling tools I get, and many aspects of the Scala language and standard library. I love Scala's collections and miss their power and ease when I'm writing Python. (Which I still do for certain scripting tasks and for accessing Python-ecosystem libraries.)
[1] https://www.lihaoyi.com/
[2] https://www.handsonscala.com/