I've read your post three times, carefully, and no, I actually can not extract a clear question from it. (I do not mean that in a mean way, just as feedback.) It seems to resolve to an observation that functional programming is not the only paradigm that has produced successfully-running programs, which is certainly true, but not terribly information-rich.
There are actually a lot of interesting possible directions you might be trying to go, but I'm not sure I want to try to guess them. (I will take one stab with: http://en.wikipedia.org/wiki/Effect_system )
I will observe that baking directions aren't a useful comparison; the difference between programming paradigms do not manifest until you have something that is a real size. Imperative is just fine for a single page program. Anything is just fine for a single page program; I have unashamedly written outright spaghetti code for single page programs.
Yes, real size and real world examples are what counts. And then, as much as numerous examples of real world and big size problem (Google, Wikipedia, etc.) have been tackled with a mix of OOP glue code (Java, C++, Python) and declarative style data retrieval (SQL), I have much harder a time finding examples of FP uses in the real world.
I suppose some data analysis is done FP style somewhere, but is it comparable in complexity? (The data might be very complex, the algorithms too, but the code itself probably has much less branches, corner cases, ifs, and other PM-added mess).
Oh, and yes, I am aware of a web app written in lisp by pg, but I wonder if this experiment is really reproductible. Is this Viaweb code still maintained and running?
When the "world" from the point of a view of a program is an enormous drealocks-like hairball full of heterogeneity (which is the case for the code bases of most of the tools we use, Facebook, PowerPoint, etc.), I have hard time imagining that it can be modelled in a nice FP ether where well-behaved functions pass to each other all the data they need, and magically "reduce" it to a magically deterministic output.
The thing that bothers me about this attitude is that it discourages considering alternatives. I've had a blog post on this stewing for awhile, but the gist of my counter argument is that it is a common human condition to feel that the current level of technology is sufficient - people have made that argument countless times in the face of progress.
Or perhaps it encourages think about alternative-alternatives. Functional programming has been presented as the alternative to OOP for a looong time now and the indications are that it just isn't to replace it ever.
OOP is a paradigm that does work for gluing huge piles of flawed-but-useful code together. And OOP indeed has lurid flaws to it on multiple levels (the ambiguity concerning what "inheritance" is supposed to mean is one of many ridiculousnesses).
Something that replaces OOP would need to have SOME winning combination of gluing hunks-of-together so you didn't have to start from scratch and being-so-powerful-but-understandable that medium skill-level programmers could replace the junk reasonably painlessly.
If we acknowledge FP isn't going to be that paradigm, then we can start searching for a different one.
Interesting; I am amenable to that idea, but don't have any experience with alternatives. I think a hybrid OOP/FP approach (i.e. Scala) can actually work really well, though I don't know if there is a perfect reference implementation.
The data might be very complex, the algorithms too, but the code itself probably has much less branches, corner cases, ifs, and other PM-added mess).
Being able to do complex stuff with less mess sounds like a big win to me. Current users of FP in industry tend to be in the financial sector(though a quick check of haskell in industry shows several CAD users too).
Though I have to agree with you, I haven't seen a really convincing case study yet. For me at least it has wiggled its eyebrows and said look over there convincingly enough for me to take a look.
I guess nobody ever thinks of Excel in these cases? I've heard it's used sometimes for calculations. It doesn't need to do destructive updates to intermediate values.
I think the argument would be that, though Excel is awesome for small data sets and so validates that it is indeed a great way to think about things at a level, it does not scale well. Indeed, one of the first things people suggest is to run away from large scale solutions that were done in a spreadsheet.
Excel strengths come from the fact that the main mode of operation is generally declarative.
Excel's weaknesses for use in large scale systems largely come from the fact that it isn't easily maintainable because, in the same main mode of operation, the actual code is hidden in cells that need to be accessed individually (in terms of usability for someone trying to maintain it, its almost like having one line of code per source file, since you have to open each cell to bring the code in it into the "editor".)
The most common way to circumvent the problem posed by the second feature in Excel is to resort to coding in an imperative language, which negates the benefit of the first.
FP languages share the first feature, but not the second.
I would argue that Excel's sole strength is that it directly features the data to the user. It is not that things are declarative that is nice, it is that the data is all you see. Because that is all a lot of folks care about.
Indeed, the way in which many people populate spreadsheets is far from the functional way. It is typically much more imperative. They literally have a list of commands from someone for how to make the chart or tabulation they are interested in seeing appear.
The fact that spreadsheet regular users are programmed by spreadsheet power users (the ones building charts, formulas, etc.) in an imperative way does not mean that the creation of the spreadsheet by the power user is not declarative, and is really completely irrelevant to what is being discussed.
Especially from a Clojurian's perspective, the tool to watch in this realm is Datomic. It's a fundamentally different approach to databases, based on the functional paradigm, and should it succeed in massive scalable situations (which I suspect it will), it will be a triumphant validation of the particular functional/immutable paradigm Rich Hickey has been promoting.
Jane Street Capital uses OCaml for their trading system. Yaron Minsky explained that one of the main benefits of it was that people could read and understand the code much better than the other languages they tried (including C#).
Not taken in a mean way. Was an honest question in my post. :)
Not sure how much I care for effect systems, just yet. Though, I think there is a high chance that is exactly what I want.
My main observation from such things as baking directions seems to be that functional (being a highly declarative style) is great for high level, but that you need a way to carefully logic about the lower level details as well.
To go with my baking example. The declarative would be, "Dinner should be enough pizza for X people..." This is great and all, especially if you are strictly at the management "high level" view of the task at hand. (Especially since this is likely to be part of a whole other set of declared outcomes. "Party room with X tables, etc...") However, one is not going to be able to actually get pizza for the crowd this way.
Now, one might say that at the lowest level one can have relatively "pure" functions such as "dice onions." It certainly seems this is where most blog posts spend there time. At the overall high level declarations and the low level functions. Often pointing out that one could even implement this with the "order from local store" keeping the high level meaningful and completely divorced from the lower level. My problem is most of these posts/papers completely lose track of the area where the actual steps to make the pizza, or arrange the party room, take place.
That is, at some level imperative works because it actually is easier to model in your mind. Why does nobody focus on identifying where that level of abstraction may be? Or do I just have a broken mind? :)
"dice onions" is not a pure function. To be a pure function, you'd need to have the diced onions, but still be able to use the original onions in another context.
The real world doesn't have any pure functions. For one thing, pure functions are timeless; you can view their inputs and outputs as just existing. Of course in the real world it requires computation time to determine what the output is, but that is arguably a characteristic of our universe rather than the function. Trying to understand pure functional programming from a real-world perspective is to start at a grave cognitive disadvantage.
Which rolls nicely into...
"at some level imperative works because it actually is easier to model in your mind. Why does nobody focus on identifying where that level of abstraction may be?"
My opinion is that it certainly is easier to start with imperative, but the problem is that by bringing in all the complexities of the real world you are therefore inevitably building with complicated pieces, and the end result is complicated. Programming with pure functions and trying to isolate the real world is cognitively unnatural... but then again, so is trying to assemble a multi-million line program. That unnatural task is made easier by shearing away the real-world aspects of the function, and deliberately retreating to a much simpler components as much as possible.
My own attempt to elaborate on this theme: http://www.jerf.org/iri/post/2908 The braveclojure.com post grazed my point, but then pretty rapidly left it. My feeling is that people say things like "pure functions are easier to reason about", but still mentally leave that statement classified in "academic brain space" and fail to explore or explain why that's also a very important statement in "practical brain space". (If my mind is broken, it is in that I have refused to build the wall between those two that so many people seem to have.)
Right, I realize it isn't a "pure" function in the direct sense, but at the abstract level it can and often is presented as such. In programming this is trivial to simply return a new one that is diced. (Consider the typical "cons" example, where the trick is to actually return a new list, not modify the existing one. Something that is not exactly intuitive for most folks.)
And, I do strongly believe that declarative works best to start. This is mirrored in the physical world where people say to focus on easily specified goals for tasks. (And why my examples started high level.) Similarly, I suspect once you get down to the nitty gritty, it is again nice to be in the declarative realm. It seems that "somewhere between" these layers is where the imperative really belongs. I just don't know where that is.
I will probably take a while to digest your post. Thanks for sharing it!
"Dinner should be enough pizza for X people" will totally get pizza for the crowd if the system I am interacting with is sufficiently smart. Eventually, the system will say "Sorry, I don't see how to do that." If we're still smarter (in relevant respects) than the system, then we can instruct it further...
I think generally, a move to a declarative specification and operational constraints, plus some operational "here's how you do it" to fill in gaps, would be a good long-term move.
> My main observation from such things as baking directions seems to be that functional (being a highly declarative style) is great for high level, but that you need a way to carefully logic about the lower level details as well.
Seems to me that its the other way around; programs tend to have, at the highest-level, an inherently ordered set of interactions with external stateful interfaces, but at a lower level have logically pure computations on values derived from those interactions which control later interactions.
When I think of functional programming I think of the verbs or actions. So instead of worrying about the onion, I would think about dicing. There is an input, onion and output, diced onion. That is a function.
> My main observation from such things as baking directions seems to be that functional (being a highly declarative style) is great for high level, but that you need a way to carefully logic about the lower level details as well.
To go with my baking example. The declarative would be, "Dinner should be enough pizza for X people..." This is great and all, especially if you are strictly at the management "high level" view of the task at hand. (Especially since this is likely to be part of a whole other set of declared outcomes. "Party room with X tables, etc...") However, one is not going to be able to actually get pizza for the crowd this way.
Functional programming is very declarative, while still being totally specific. It is declarative in the sense that it describes what things are (as opposed to do this and that to get this thing), not in the sense that you give a list of constraints to a program and leave it to the program to make sense of those constraints - such as logic programming and linear programming. Descriptions like 'dinner should be enough pizza for X people' sounds like a constraint, like 'no more than 20 onions, at least 10 potatoes, potatoes are 0.5 currency per potato, you should have less cabbage than cod; maximize the nutritional value according to (linear function)'... Of course languages like Prolog has an understandable way of finding if a program satisfies all your constraints, but it is different from the execution model of an imperative language. Functional languages have the same execution model (well, from those that I've seen), only with a more functional style (and tools to do the functional style in a non-horrible looking way). The mathematical way of defining factorial is very declarative, yet it is also as specific as any imperative program you'd want to write to define factorial.
So I don't see what you mean about low/high level here.
I was referring to level of abstraction. At an abstract level, many functions simply describe what something is, not how to get it. Easiest example for me is X to the Y. The naive implementation of calculating that is terrible compared to a good way. Yet both are mathematically the same.
Essentially, the difference between a descriptive equation and an algorithm. :)
> I was referring to level of abstraction. At an abstract level, many functions simply describe what something is, not how to get it.
All functional programs describe what something is and how to get it (the what is the semantics of how you get it).
> Easiest example for me is X to the Y. The naive implementation of calculating that is terrible compared to a good way. Yet both are mathematically the same.
"X to the Y"? Are you talking about time complexity? That is just as much a problem in imperative programming as in functional programming. Or are you talking about some constant inefficiencies, such as using recursion instead of a loop? Even though you might write the algorithm in a functional manner, that doesn't necessarily mean that it is all cons lists and recursion all the way down. Those things might have been translated to a more efficient, imperative form by the compiler (GHC for Haskell will do that). The compiler can do more aggressive optimizations for a pure function than for a method that is restrained by a type signature like "takes a String, returns a Boolean, and can do everything from turning on the kitchen light to blowing up the Universe in between".
> Essentially, the difference between a descriptive equation and an algorithm. :)
All pure functions are algorithms. It's all programming, after all. I don't get the distinction in this context.
Yes, I was referring to the time and space complexities of actually calculating the values. If the pure function were the algorithm, then there would be no separating the two. However, since a pure function can be calculated by several algorithms, then we are really just putting off some of the major points, no? (Hell, even in the simple 8 * 2, we are glossing over what really happens. Few programmers really appreciate the way in which that math happens on the computer. And fewer still are aware of when this changes in their architecture. And for anyone that thinks this is just some elementary exercise, I challenge that you have not seen the effort it takes children to really appreciate how to calculate 8 * 2.)
This is all really missing my main assertion, though. What is your function for getting in your car and driving somewhere? Could this be written in such a way that everything was a "pure" function? Probably. My hunch is that it wouldn't really be that readable, though. Now, if we shift the paradigm such that one of the "primitives" is your current location, suddenly the directions are a lot more readable.
For an easy example on what I mean here, just look elsewhere in the thread where folks were writing the program for cooking things. By far the most readable is the imperative one.
> Yes, I was referring to the time and space complexities of actually calculating the values. If the pure function were the algorithm, then there would be no separating the two. However, since a pure function can be calculated by several algorithms, then we are really just putting off some of the major points, no? (Hell, even in the simple 8 * 2, we are glossing over what really happens.
Again: both imperative and functional algorithms can be implemented in several ways. Both imperative and functional algorithms can solve the same problem, but have different time complexities.
"However, since a pure function can be calculated by several algorithms, then we are really just putting off some of the major points, no?"
What? The semantics of a pure function shouldn't be changed when you actually compute it - there is no "functional algorithm execution algorithm". The actual code that gets evaluated can be pretty different from the actual code you wrote, though - maybe your lazy lists got implemented as for-loops. Maybe you used a list in some way that made it look like you had O(n) space complexity, but it got turned into a loop that didn't store intermediate values so the space complexity is actually O(1). So yes, time complexity can be a little decieving, in some cases, if the compiler is sufficiently smart. However, there is no magic about the actual semantics of a functional algorithm.
> This is all really missing my main assertion, though. What is your function for getting in your car and driving somewhere? Could this be written in such a way that everything was a "pure" function? Probably. My hunch is that it wouldn't really be that readable, though. Now, if we shift the paradigm such that one of the "primitives" is your current location, suddenly the directions are a lot more readable.
Maybe you could. I don't think many care, though, not even functional programmers. The only decently widespread language that I can think of that actually enforces purity is Haskell, and some people even prefer to write imperative code in it over other languages. I don't get this obsession of yours with trying to argue against pure functional programming in the very, very strict sense of the whole program being pure, if that is actually possible. I think there are very few that are upset about having to write imperative code for code that has to do with sequential commands. They might use pure functional programming for the computations that they need, but they'll effectively write imperative programs at the top level.
What they probably do want, though, is to be able to clearly distinguish what parts of their programs are pure and what parts of their programs are potentially effectful.
Ah, I think I see the misunderstanding I am causing. I am not arguing against pure functions as a whole. Just acknowledging the weakness that they can not often be the whole. Many of the arguments for pure functions seem to be based on the "if your application is nothing but a collection of pure functions, then it will be easier to logic about." My assertion is "use that which is easier to reason about where you can." In some cases, this will actually be mutable state based entities. In others, it will be pure functions.
I also fully grant that when one is trying to work in a massively shared state application, pure functions have massive benefits. Seems the real win is typically identifying that which can be easily shared and that which can not. And partitioning the program accordingly. (Which, is what you ended with. So, I believe we are actually in fairly violent agreement. :) )
I've read your post three times, carefully, and no, I actually can not extract a clear question from it. (I do not mean that in a mean way, just as feedback.) It seems to resolve to an observation that functional programming is not the only paradigm that has produced successfully-running programs, which is certainly true, but not terribly information-rich.
There are actually a lot of interesting possible directions you might be trying to go, but I'm not sure I want to try to guess them. (I will take one stab with: http://en.wikipedia.org/wiki/Effect_system )
I will observe that baking directions aren't a useful comparison; the difference between programming paradigms do not manifest until you have something that is a real size. Imperative is just fine for a single page program. Anything is just fine for a single page program; I have unashamedly written outright spaghetti code for single page programs.