This unfortunately breaks down with (bad acting) NIFs. Thankfully you can mark 'em as the dirty evil little things that they are (with negligible overhead): ERL_NIF_DIRTY_JOB_CPU_BOUND. [1]

I implore anyone interested in Erlang or its surrounding languages, to read its source code. [2] More specifically, the BEAM. I'll warn you that it's very 80's hackeresque, but in a good way. Incredibly pragmatic. The way they achieve their world-class scheduling? Tip: not via a perfect theoretical design cooked up in some comp-sci lab. [3] If you've been following game engine design within the last 9 years or so, you'll probably have a good idea of how they do it. [4][5]

[1]: https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...

[2]: https://github.com/erlang/otp/tree/maint/erts/emulator/beam

[3]: https://hamidreza-s.github.io/erlang/scheduling/real-time/pr...

[4]: http://blog.molecular-matters.com/2012/04/05/building-a-load...

[5]: There's a hell of a lot of depth to this problem domain. Charles Bloom's guiding principles for Oodle are spot, and well worth a look.

> I'll warn you that it's very 80's hackeresque, but in a good way. Incredibly pragmatic.

I consider the BEAM VM as one of the marvels of software engineering.

You know it is good, when you explain to other programmers that you can have something like an isolated memory process just like an OS process, with preemption and only a few Ks of memory, with a low latency GC, with distribution across machines built in -- and they don't believe me.

Even experienced senior developers are skeptical saying stuff like "well yeah but those are green threads then and they have to yield explicitly -- nope, they don't", "so you have callbacks then somehow, no, not callbacks", "what do you mean separate memory spaces, it is a single OS process right?" and so on. It sounds like magic -- this stuff shouldn't exist, right, but the awesome thing is, it does.

Moreover, it is not a hacked up version from a lab some place or a PhD disertation proof of concept -- this is what powers banking, databases, messaging, and probably more than 50% of smartphone access to the internet today.

I'll just put this here in case anyone's interested.

https://github.com/mbrock/HBEAM

It's a Haskell executor of .beam programs, in very early prototype stage, and I abandoned working on it 5 years ago (apparently).

Of course it's not meant to be competitive in any way, it's basically just for fun, and because I wanted to learn more about how Erlang works.

The function `interpret1` in the middle of this file has the main opcode switch.

https://github.com/mbrock/HBEAM/blob/master/src/Language/Erl...

I think it's about the smallest subset needed to run a factorial program, but also to implement the very basics of mailboxes with send/receive/timeout.

It uses GHC's shared transactional memory for the mailboxes:

https://github.com/mbrock/HBEAM/blob/master/src/Language/Erl...

Someone, fork it and finish it! :-)

I'm interested. How does the preemptive scheduling work with blocking system calls? So if an Erlang process tries to read from standard input using the read syscall (suppose we haven't set non blocking mode on the fd), why does it not block the scheduler that it runs on? Or does Erlang implements its own set of syscall wrappers that uses epoll under the hood?

There is an IO thread pool for some blocking operations like file IO, and there is also epoll for sockets, for example if I see this in the prompt on my laptop:

   $ erl
   Erlang/OTP 18  ... [smp:4:4] [async-threads:10] ...

The async-threads indicates it has started 10 IO threads. So if a process needs to read from a file on a slow disk, it will dispatch that request to one of those threads and then it will be descheduled (put to sleep).

The smp:4:4 says there are 4 schedulers configured and enabled. Usually there is a scheduler (as an OS thread) which runs on each core (also highly configurable, with custom topoligies, affinities etc). Those schedulers will pick and run Erlang processes.

Typically to ensure fairness, each process is allowed to run a set number of reductions (think of them like equivalent bytecode, but say an internal C driver like a regex parser should also periodically yield and report it consumed a given number of reductions, so it can be descheduled).

Funny enough all that sounds a bit like an operating system, and that's a good way to conceptualize it. Erlang/OTP is like an OS for your code. A modern OS is expected to be resilient against bad processes messing with other processes' memory, multiple applications should run and be descheduled preemptively etc.

You've got it: processes are not allowed to run blocking syscalls under the hood. Transparently that work is handled by backing IO threads.

I'm interested as well! How does it work then?

(1) How does it preempt threads (my guess: it doesn't actually preempt threads, the interpreter yields after a certain number of instructors, or (if compiled) the compiler inserts conditional yields in each loop/function call/return)?

(2) How are memory spaces isolated (my guess: they aren't really, it's just that the memory allocator doesn't mix memory allocated by different threads)?

(1) In the normal case it just counts a certain number of VM instructions each process runs before it gets to rescheduled. But it gets interesting as well with internal modules or C modules (for example regex matcher), in that case the C module periodicaly as it works through the data, reports that it consumed some number of reductions and is possibly told to yield now. (On that note: in 19.0 we'll have dirty schedulers by default so in that case blocking long running C code will be handled much better).

(2) Erlang VM instance (it is called a node) is an OS process (plus some helper processes, but they are not important in this case) is of course one heap from kernel's point of view. But internal allocator keep spaces separated for BEAM's data. It is not always as basic as in some cases for binary blocks and sub-blocks it can actually share and ref-count them.

In the new release I like that it could have mailbox outside the main heap of each Erlang process. That could be an interesting parameter to play with.

> (1) How does it preempt threads (my guess: it doesn't actually preempt threads, the interpreter yields after a certain number of instructors, or (if compiled) the compiler inserts conditional yields in each loop/function call/return)?

Yep. The interpreter lets each thread do 2000 reductions (roughly == function calls), or until it waits for new messages if that's sooner.

> This unfortunately breaks down with (bad acting) NIFs.

Where possible, I think the Erlangy thing to do would be to just have a separate executable and communicate with that. Also ensures that things keep running if there's a segfault or something.

Exactly! Well, kinda.

You can always throw the NIFs onto other nodes. You can also write ports that are glorified external processes.

Other times, you just need low-latency and high throughput. In that case, you expect failure and design accordingly.

     (bad acting) NIFs. Thankfully you can mark 'em as the dirty evil little things that they are (with negligible overhead): 

As a side issue, is there an easy method to determine if a NIF is problematic in this regard? I've used jiffy[0] in several codebases, but I keep reading these warnings and wondering whether I should be doing so.

[0] https://github.com/davisp/jiffy

The warnings are mostly about NIFs you write yourself, which you typically avoid if possible. And jiffy itself goes quite a long way to cooperate with Erlang VM's internals (at least so I've heard).

There is one other. Haskell has green threads, a preemptive scheduler, and it has a pretty decent implementation of Erlang-inspired multi-node concurrency primitives and higher-level framework including supervisors, gen_server equivalent etc in the Cloud Haskell project. It does NOT have Erlangs deployment base and track record but it is still a very promising framework and very appealing if you like Haskell's type system.

http://haskell-distributed.github.io/

Just a nitpick, but Haskell's scheduler (like Go's) is "less preemptive" than Erlang's. If you have e.g.

    i = 0
    for{
        i += 1
    }

or equivalent in Go/Haskell, then the HS thread/goroutine running it cannot be interrupted, as interruption can only take place at certain points (e.g. allocation or function calls).

Erlang on the other hand assigns a certain amount of time units to each process, and each pure-Erlang operation consumes these, so an Erlang process can always be interrupted as it will eventually use up this time allocation and yield (unless it's actually calling a C function etc).

Haskell's scheduler preempts on memory allocation. Memory allocation is ubiquitous in Haskell; the example you are giving cannot really be written in Haskell as values are immutable. You could try to do something with IORef and bang patterns to avoid thunks being allocated but you are now in a very tiny corner case, much smaller than the surface area of NIFs that can cause you problems in both Erlang and Haskell.

Here's a minimal example:

  package main

  import "fmt"

  func main() {
    go hog()
    for i := 0; ; i++ {
      fmt.Println(i)
    }
  }

  func hog() {
    for {
    }
  }

It stalls after about three seconds on go version 1.6.

Here's the issue on github:

https://github.com/golang/go/issues/543

Here's a modified version that doesn't stall:

  package main

  import "fmt"
  import "runtime"

  func main() {
    go hog()
    for i := 0; ; i++ {
      fmt.Println(i)
    }
  }

  func hog() {
    for {
      runtime.Gosched()
    }
  }

Admittedly, Haskell is less preemptive than Erlang, but definitely not at the same level as Go. A mutating variable is not what you commonly use in Haskell, so the idiomatic equivalent of your code would have memory accesses anyway, and it would be possible to interrupt it.

Haskell scheduler switches at memory allocation.

When your CPU intensive code does not allocate, it will not switch to another green thread.

So point in the example above is absence of memory allocation. And, having decent codegenerator, Haskell could produce the code above where registers are reused for new values (effectively, mutation).

Yes, an Erlang process can be interrupted at any point, unlike Go/Haskell, but this is at the cost of bytecode interpretation. I'm not saying it is bad, but there is a tradeoff.

I've been confused for some time as to why people get excited about green threads. From what I've read, the main advantage seems to be that you can have threads on hardware that doesn't support threads natively, which is cool if you're on that kind of hardware. There's also some spin-up advantages I guess? But they don't get load-balanced across cores, right?

I feel like I'm missing something important.

Green threads aren't just a substitute for when you don't have system level threads. Instead, they're a way of structuring code that allows you to express highly concurrent programs without requiring the heavy overhead of launching and switching between operating system threads.

Linux switches between threads at some frequency, I think it used to be 100 Hz. It involves swapping out the process registers, doing some kernel bookkeeping, etc—this is called a "context switch" and it's quite costly. Also, Linux threads allocate at least one memory page (4 KB) for the stack. [If I'm wrong about these details, please correct me!]

Basically, the cost associated with an operating system thread comes from the fact that it has to be isolated from other system threads on a low level... whereas language runtimes that offer green threads impose their own safety via language construction, e.g., Erlang processes can't reach in and mess with other processes memory (without C hacks).

So green threads can be much more efficient, but they require some care in the implementation, especially to support I/O, and to have fair and efficient load balancing, etc. Then you run N operating system threads to get balancing across cores, and distribute green thread work.

The advantage is you can have many more green threads than you could have OS threads (hundreds of thousands vs thousands), due to green threads being more lightweight. This allows a programming model based on message passing between green threads, which many people consider nicer to reason about. E.g. for a web server you could have a goroutine/Erlang process for each client with 50k clients and still have excellent performance, whereas if you had 50k OS threads you'd likely suffer performance issues and use a heap more ram.

And this is a huge benefit in the ease of coding.

If you only have two or four or sixteen threads you write overtly threaded code. But when you have millions you don't. Your program looks single-threaded and yet operates better and more safely.

One advantage is spinning up new green threads can be very quick. Starting a new Kernel thread requires at least 1 syscall.

For example: on a network service, you have 1 thread listening for new connections, when a new connection is made. It starts a new thread, which calls the handler. The listener thread then goes back to listening for new connections.

Now the advantages can depend on your green threading implementation. If a listener thread blocks on reading from Disk or a DB. Then the listener thread can still wait for new connections and other connection handlers can still operate. Making you network application responsive, without increasing latency on clone syscalls.

Of course you can achieve this in other ways.

First, you can run TONS of them, which is an enabler for program designs that native threads doesn't work well with.

Second, they are much lighter on memory (well, comes with the first point, but still).

Third, the supervising VM/environment has more fine-grained control over them than with native threads.

And in any decent implementation, they are absolutely load balanced across cores, why wouldn't they be?

Specific to Erlang processes -

As others have indicated, they're extremely lightweight, cheap to create and throw away, and are load balanced across cores.

But also, each has isolated memory, with share nothing semantics (with a couple caveats) which means that an exception in one won't affect others -unless you want it to-. That's huge.

But as has also been mentioned, you can create many of them. Someone else threw out 50k; nevermind that, try a million of them on a single box. That kind of concurrency opens up an entirely new paradigm of coding. One that is actually very useful, because it turns out, a lot of problems are naturally concurrent problems, that we've been trained to think about in sequential patterns because of how hard concurrency is.

An example I like to give is from the real world - task scheduling. We had to write some simple task scheduling for an application. Each task was multiple steps, many of which were time based (i.e., "execute this command, wait X amount of time, execute another command, wait Y amount of time, execute a third command, once that is successful execute a fourth command"). The traditional way of doing this would be some sort of priority queue, with tasks weighted by how long from now until they were to be done. You check how long until the next event, sleep until then, fire it, then repeat. Simple, right?

Except...each event leads to more events. And event timings can change. And events can happen simultaneously, so you actually need a pool of threads to actually execute the events on. Locks everywhere. Task logic is very hard to isolate from the execution logic (i.e., the bit that says "do X, and create an event to execute at time Y" is hard to keep entirely separate from the "pull event from priority queue, throw to a new thread to execute on, sleep until next event", since there are so many interactions between the two that can affect one another, changing when events happen, and when the queue puller needs to wake up).

In Erlang though? Trivial. Write your entire task as a single job. I.e., do x, wait, do y, wait, do z, wait for a message that z has completed, do a, etc. Then spin up one of those for each task that you need and let the VM handle the concurrency aspects of it. Even additional complexity, like "in the event of a message, change the amount of time until the next event to be half of what it was" is trivial; it's all contained in the same module, it all describes the same lifecycle of a single task. All the concurrency, the running of many of those tasks, and their interactions, and ensuring none is blocked, etc, is -free-.

This sounds like an obvious, ideal example once explained, and yet, every person where I worked who was unfamiliar with Erlang (and even some of those who had coded a little in it, but hadn't come to grasp the paradigm as well), who was explained what we were trying to do, described it as "easy, we just need to use a priority queue and pull from it!"

I think it's not as much green threads as very many threads. And then they're not as much threads as independently threadable parts of the code.

You don't have to intentionally write a work queue and balance the number of readers vs writers, etc. You just let the runtime make things go as fast as possible.

They do get balanced across cores in almost all cases. Erlang is a functional language which really lends itself to this.

+1.

I think a preemptive scheduling model is really the only way to implement that model effectively. Any other scheduling method eventually causes the consistency of the programming model to break down as it's now pushing the cognitive burden about what to run, where, and when onto me as the developer again.

> AFAIK, Erlang is still (as of 2016) the only distributed actor model implementation with preemptive scheduler.

It is one of the open issues with Golang.

https://github.com/golang/go/issues/11462

User aclements notes:

> @GoranP, you can think of the Go scheduler as being partially preemptive. It's by no means fully cooperative, since user code generally has no control over scheduling points, but it's also not able to preempt at arbitrary points. Currently, the general rule is that it can preempt at function calls, since at that point there's very little state to save or restore, which makes preemption very fast and dramatically simplifies some aspects of garbage collection. Your loop happens to contain none of these controlled preemption points, which is why the scheduler can't preempt it, but as @davecheney mentioned, this is fairly uncommon in real code (though by no means unheard of). We would like to detect loops like this and insert preemption points, but that work just hasn't happened yet.

And goroutines are not distributed, while actors can be transparently deployed across remote physical nodes, both in Akka and Erlang.

How does this work with state? Isn't it inefficient to transfer the state of the actor to another node compared to just letting the actor run on the same machine?

The API only defines a way to start a new actor remotely.

Now, each actor per actor model convention has to survive restarts. In distributed environment, the state can be persisted e.g. in distributed memory cache / storage (Hazelcast, Cassandra).

Also, coroutines (while also cool; I admire Go and its efforts to bring concurrency to the masses) are not actors. Actors can be persistent and have lifecycle.

It is specifically because of the preemptive scheduling that I find it worthwhile to write code that would otherwise be better suited to a different language in Elixir.

Numerical and statistical stuff, for example, is never going to be performant--but that's okay, because I'd much rather have a whole bunch of slow actors making incremental progress safely than a couple of blazing fireballs that could take down my system using NIFs or whatever.

So is preemptive or cooperative better in your opinion ? I couldn't figure it out from your comment.

preemptive. Cooperative scheduling runs the risk that a long-running actor might starve the other actors

Note that "better" means better for some kinds of things. Preemptive scheduling has a cost, just like running a program on top of, say, Linux costs more than if you coded it specifically to run on the chip itself with no OS.

Exactly. In terms of fault tolerance, a buggy actor is isolated in regards to scheduling others. Bad behavior has a more limited scope.

This isn't to say that one can't create a process that eventually has harmful consequences to the entire system, it just means it's much less likely. These little points of isolation add up. Scheduling, memory management, lifecycle, crashes, &c... It's why it's a bit funny when I hear Akka compared. Akka is quite impressive but I'd never consider it a true alternative to an Erlang or BEAM style runtime.

Akka is quite good in terms of reliability (I developed in both Akka and Erlang). All actors in Akka are supervised by default, and restarted if any exception happens.

This allows skipping most of error handling in Akka code. If something's wrong, your actor will be restarted. Works great!

This assumes failure is clean. Some failures are tougher, for example, imagine code going into an infinite loop. It either system, you have waste but the scale of the disruption in Erlang is kept isolated where Akka would fail to release the thread. Otherwise, I'd definitely agree that Akka handles vanilla failures just fine by using supervision trees.

Quasar has preemptive scheduling for fibers/actors on the JVM.

Could a function with a body like just

    for(;;){
    }

be preempted?

You could, but we removed time-slice-based preemption a few versions ago b/c it doesn't make sense for fibers if you also have access to heavyweight threads, so now we only preempt on IO/sleep etc.. The thing about time-slice preemption is that it just doesn't help with lightweight threads anyway, as your machine can only support a tiny number of execution threads that require time-slice preemption while you can have hundreds of thousands, or even millions, of lightweight threads. So runtimes like Erlang that don't give you direct access to kernel threads have no choice but to support time slice preemption, but on the JVM we realized that it serves no purpose, so we took it out.

>now we only preempt on IO/sleep etc

The IO you are talking about - that is I assume IO methods/libraries that are specifically written/adapted for use in Quasar, yes? If you just use an off-the-shelf JDBC provider, you would block the entire thread when one fiber calls out to the database right?

If so, that is not preemption, that is cooperative multi-tasking.

It's not cooperative because the fibers don't explicitly yield control; they are preempted as soon as they perform an operation that does not require use of the CPU.

As to the choice of libraries, that is an artificial distinction. Runtimes like Erlang and Go also require libraries specifically written/adapted for the runtime; calling an off-the-shelf IO library from Erlang/Go would also block the entire thread. As integrating a library with Quasar does not require changing its interface, the question of whether to trap calls to specific implementations (which is trivial on the JVM) or require the use of fiber-friendly implementation (wrappers) is a question of design. So far, we've opted for the latter simply for the sake of "least surprise", but we may choose to do the former, too. Also, unlike Erlang/Go (I believe), any accidental blocking of the kernel thread is automatically detected by Quasar at run time, and reported with a stack-trace.

Quasar operates in the exact same manner as Erlang/Go, only that we've disabled time-slice preemption once we realized it neither helps nor is it required on the JVM. The only real difference is one the nature of the ecosystem: while all pure-Erlang libraries are designed to work with fibers, most JVM libraries aren't, and so require a thin integration layer that is provided separately. OTOH, thanks to the size of the JVM ecosystem and the standard interface approach, I believe it is the case that today there are more IO libraries that support Quasar fibers (e.g. all servlet servers) than those supporting Erlang processes.

I'd like to mention Celluloid here as well. Generally I prefer Erlang/Elixir, but in Ruby centered apps, Celluloid (especially run under the JVM with JRuby) solves a lot of concurrency problems easily.

> All other major implementations (including Akka) have cooperative scheduling, i.e. forbidding blocking code in actors. Erlang allows it.

Curious as to what people's thoughts on Celluloid:IO are with respect to solving the problem of blocking code in actors. Celluloid:IO provides an event driven reactor that works in parallel with the actors. I haven't used it myself, but it seems an interesting approach.

Care to elaborate on 'streaming market data aggregation'?

Nothing too complicated — connecting to various FX liquidity providers (LMAX, Currenex, CFH etc.) via FIX protocol and merging their individual offers using VWAP (Volume weighted average price).

Cloud Haskell allows blocking code in actors.

Please don't do programming language flamewars on HN. By that I mean please don't make sweeping statements that are inflammatory and contain no actual information.

> but so far, Erlang has failed to deliver on most of the promises that its advocates repeatedly make.

Speaking of citations...

Facebook, Whatsapp, almost every phone call you make for decades, and much of the backend for League of Legends were all built on Erlang.

Not that the others are unsuccessful from an engineering or financial standpoint, but the last one, Riot Games' League of Legends recently hit 1 billion in revenue. It's a fact that it's the most popular video game on Earth today.

If it didn't work or "failed to deliver on its promises"- none of these world-class, top-tier services would be where they are. To repeat myself so it sinks in. Yes. It is a fact that every phone call you make hits Erlang code somewhere.

Because people actually stand in the face of Erlang and spout this nonsense to this day... I'll take the claim one step further and will back it up if challenged as if it's not already self-evident.

Erlang has proven itself better, for a longer period of time than Java. Many have tried to replace Java. No one, and I mean no one has tried nor has anyone succeeded in replacing Erlang.

Only C rivals it for carrying the world on its shoulders.

Erlang was developed for developing telecommunication systems and is still used heavily in that area today. When was the last time you heard of someone unable to make a phone call because a phone network went down?

That's not what Wikipedia says at all.

> In 1998 Ericsson announced the AXD301 switch, containing over a million lines of Erlang[...].[8] Shortly thereafter, Ericsson Radio Systems banned the in-house use of Erlang for new products, citing a preference for non-proprietary languages. The ban caused Armstrong and others to leave Ericsson.[9] The implementation was open-sourced at the end of the year.[5] Ericsson eventually lifted the ban; it re-hired Armstrong in 2004.[9][...]

> Erlang has now been adopted by companies worldwide, including Nortel and T-Mobile. Erlang is used in Ericsson’s support nodes, and in GPRS, 3G and LTE mobile networks worldwide.[10]

https://en.wikipedia.org/wiki/Erlang_%28programming_language...

Not to mention

"In 1998 Ericsson announced the AXD301 switch, containing over a million lines of Erlang and reported to achieve a high availability of nine "9"s"

That was the first commercial use of the language. The first commercial use of a 'niche' language, with over a million lines of code, achieved a downtime of just over half a second over 20 years. That's total downtime, too, not just 'unplanned'. And even if you take into account the numbers touted by critics of that quote, of 5 nines...that's still considered world class. For the first damn commercial product.

That's delivering.

You literally have no idea what you're talking about, said the guy who knows several people on the current OTP team at Ericsson.

> Erlang has failed to deliver on most of the promises that its advocates repeatedly make.

Need some citation there,

I am sorry for you being so downvoted, since your comment is legitimate. Indeed, Erlang is a niche language.

Nonetheless, it powers WhatsApp and Facebook Messenger, with their billions of users. Quite a proof, I'd say.