But, going the other way, I worked for over a decade on Goldman Sach's SecDB system. It's a quirky steampunk alternative future that branched from our light cone around 1995. There's a globally distributed eventually consistent NoSQL database tightly integrated with a data-flow gradually-typed scripting language (and a very 1990s feel 16 color IDE). I'm sure in the late 1990s/early 2000s (before globally distributed NoSQL was popular and before gradual/dynamic typing had a resurgence) it was more like discovered alien technology than steampunk alternative future. (Also, with source code being executed from globally distributed immutable database snapshots, deployment is much nicer than anything else I've used to date. After release testing, set a database key to point to the latest snapshot, and you're deployed.)
There's a service that watches the transaction log of your regional replica so that you can make long-poll HTTP requests that return when any change matching your filter is committed. (Edit: usually the HTTP result handler is used to invalidate specific memoized results in the data flow graph, letting lazy re-evaluation re-fetch the database records as needed.)
It makes a lot of sense for a financial risk system, where you end up calculating millions of slight variations on a scenario. The data flow model with aggressive memoization makes this sort of thing much cheaper.
However, I saw plenty of systems written where you'd attempt to write your request to the next key matching some regex (and retry with the next key if it already existed), where your request would contain some parameters and the database key and/or filesystem path where results should be written.
Under-experience with databases easily results in rewriting a database using message queue/bus. Under-experience with message queues/busses easily results in rewriting a message queue/bus using a database.
Message queueing triggers a host of psychological needs. Synchronous jobs rarely need monitoring and management features, but move the same work to a queue, and everybody loses their minds.
I've seen so many people spend weeks if not months "working" to avoid doing a trivial database migration. Database fear is overwhelmingly powerful in a lot of people it seems.
> Database fear is overwhelmingly powerful in a lot of people it seems.
Database are still fairly poorly documented when it comes to administrative work.
There is an incredible amount of tutorial, books and courses on how to write sql queries and stuff... But there is almost zero content on how to properly administer a database.
I mean, from novice admin to DBA-level capabilities.
I said all this before and i'm ready to write this again: i think there's a good market space for dba-style courses.
This does not honestly seem true for the main sql databases. For everything else, yes, but if people were actually learning databases instead of hiding from them most of the uptake there wouldn’t exist anyway.
I think I've seen enough complexity created by engineering teams given total autonomy, with hands-off leadership, that I'd prefer a much more constrained approach. There should still be autonomy, of course, but proposals for new tech, languages and paradigms should only be considered with due diligence.
The most unpleasant codebases I've dealt with are ones that have suffered from a lack of strong leadership, and they are almost uniquely microservice setups that pull in everything but the kitchen sink, usually because it's just trendy to use it. Monoliths can get pretty damn ugly too but at least it's contained in one single codebase.
The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
Literally instead of a table.
This stuff is everywhere. Microservices made it worse and half legitimized it.