For example, right now I’m building a system that will take in a URL, crawls the webpage, and does some processing on the data. The whole process takes a good 10 seconds.
I designed this as a micro service, simply because I know URLs will need to be queued up. Should this have been done as a monolith? Or am I right that micro services was actually the right approach?
In rails this would be an activejob (part of the mono), and spinning up workers would be trivial.
The "workers" would pull off a redis queue, and you could add different jobs to the queue easy-peasy.
That's fair, maybe I'm already planning to far ahead. But I ended up using RabbitMQ and Node (since i wanted to use Puppeteer for this, I actually needed to render the webpage).
I did something like this too, and figured I'd rather do it for free as a lambda running under Vercel rather than create a droplet or something else and pay for it...
For example, right now I’m building a system that will take in a URL, crawls the webpage, and does some processing on the data. The whole process takes a good 10 seconds. I designed this as a micro service, simply because I know URLs will need to be queued up. Should this have been done as a monolith? Or am I right that micro services was actually the right approach?