There is no single thing, it's more like death by a thousand papercuts.

But if I did have to pick "the one" (I see you trying to pull a Scott Adams, good stuff) it would be that it has the semantics of Literate Programming completely backwards.

Literate programming is about 'code' first. It should be that you have a codebase which honours code semantics first, and allows you to create a meaningful report from that code second. This guarantees that you have a codebase structure that is easy to traverse and read, while still following good software engineering practices and usable as code in itself, as well as being able to trivially create a report from it.

Jupyter notebooks is an app (not even just markup, but an app, and a clunky one at that) for writing reports first, in the form of snippets from which you may or may not be able to generate useful code or outputs later on. And ultimately, even if you do, that code is crappy and unusable in any context other than the notebook it was written for, because of the way jupyter notebooks are designed. They force the programmer to create a monolithic spaghetti structure, preventing modularity or meaningful code hierarchies to take place in the codebase, promoting imports and definitions appearing just before the point of their use rather than in reasonable scopes, promoting modifiability instead of extensibility, while preventing programmers from using a vast majority of appropriate software tools for versioning, testing, etc.

And for some bizzare reason they are becoming the de facto standard for data scientists sharing code. It's like we're encouraging people to return to "Academic Matlab" code-quality standards all over again after years and years of trying to teach academics proper software engineering practices.

No, Jupyter is not an app for writing reports and it's definitely not its primary use across DS.

When it may take 5 minutes to simply load the data, you can no longer rerun some Python scripts to mess around and experiment with stuff. You need that kernel with data and everything else to have 100% uptime.

Similarly, you may want your temporary experiment results that may have taken a while to compute to stay within the kernel even if you're already working on something else.

That, plus an ease of inline visualisation, displaying tables and all that.

Eh, I think this misses the point of why Jupyter Notebooks are useful, and who is using them.

I agree that in terms of literate programming as Knuth defined it, Notebooks are not great. There are tools to improve that story; I wrote https://github.com/agoose77/literary which at least lets you do a bit more "tangling and weaving" than you can out of the box. It doesn't let you define functions in arbitrary order, or implement fragments of a code block, but it does let you "boil down" a literate representation into something that is zero-cost at runtime and imports. There's also nbdev, although it's not my cup of tea.

The real point, though, is that most data-scientists aren't using (imo) notebooks to write and share libraries of code. Instead, they're using notebooks as semi-reproducible reports. I'm a physicist, and that's what I've been using Jupyter for. For me, Jupyter Notebooks are fantastic - the cell mechanism lends itself to rich-outputs that augment the narrative, and present the information in-line with the code that wrote it.

For me, the biggest gap here is writing _libraries_ that are leveraged in these notebooks. That's why I wrote Literary - to try and resolve some of the pain points that currently require you to use two tools (Jupyter Lab & e.g. PyCharm). I'm not saying it will work for everyone, or solve all of the problems, but for me it's enough to write my analysis as a package, so that's a limited success in my book.

aldanor also mentions another use case which I only really allude to despite it being an important part of the process: exploratory work. Having a live kernel that maintains kernel state with the benefits of rich outputs is a mainstay of research.

I don't know about the grandparent but I hate the browser based Jupyter notebooks, luckily I found that you can run Jupyter notebooks inside of Visual Studio. It's a more fluent experience overall.