Just to be sure I've understood correctly, what you're saying is "Bah, humbug!" Is that right?
Edit: OK, sarcasm aside, I can't see how this comment amounts to more than "I don't need anything fancier than shell scripts." Fine, you don't. But there are lots of people out there that do.
For several years, I ran a cluster that consisted of 10 or so EC2 instances that had to rebuilt and replaced every day. Shell scripts didn't cut it for me. Chef and Puppet were painful. I ended up using Fabric, boto and a pile of custom Python code.
And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
> And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
Four years ago, I was going this with shell scripts, a little bit of Python, and a cluster management tool called Rocks (built in Python and shell scripts). This was for a 5500+ Linux server cluster processing data from the CMS detector at the LHC. I had the ability to do a rolling reinstall of Scientific Linux/CentOS while the cluster was actively processing data with no interruption to the end user jobs.
Everyone keeps reinventing the wheel, but in most cases, shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
...shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
I've written loads of both, and I realize the thread had already started down this path, but how is a shell script really different from a python script? If you're targeting Bourne, your shell isn't a separate installation, but it isn't onerous to install your favorite python version on machines you control. Sure, python has an ecosystem of libraries that shell doesn't have, but you don't have to use those libraries if you don't want to. What terrible consequences does someone who prefers python to shell scripting face?
> What terrible consequences does someone who prefers python to shell scripting face?
Terrible consequences? Probably not. Shell script dependencies are easier to handle, because the commands you're relying on are built right into the OS. They're going to be there because they have to be. How often does sed, grep, awk, and cut change?
It's a matter of complexity. If you're munging files and using regexps, shell is fine, and quicker than Python. If you're starting to think "I need a set" or "I need real error handling", it's about the time I move to Python or Perl.
> called Rocks (built in Python and shell scripts)
Regardless of if it "worked", I think mixing languages in an application is a bad idea, and should be avoided whenever possible. There are a few reasons,
1) maintainability - you're requiring anyone to know more things to maintain the application
2) Attack surface - lots of security bugs come in between boundaries between tools
3) Correctness - it's harder to automate testing
Did you have any sort of automated testing?
Just because shell scripts can do a lot of heavy lifting, doesn't mean they should.
10 or so EC2 instances that had to rebuilt and replaced every day
Just to be understood correctly, you're saying the exception proves the rule? Sidestepping the necessity of the task above, which I'm sure was rendered easier by Fabric or whatever, but it's not like that's a typical use case. You would have to ignore nuance to extrapolate that to generic VM instances from known-state images, etc. that might respond just as well to a bunch (not even a bunch) of scripts.
Edit: OK, sarcasm aside, I can't see how this comment amounts to more than "I don't need anything fancier than shell scripts." Fine, you don't. But there are lots of people out there that do.
For several years, I ran a cluster that consisted of 10 or so EC2 instances that had to rebuilt and replaced every day. Shell scripts didn't cut it for me. Chef and Puppet were painful. I ended up using Fabric, boto and a pile of custom Python code.
And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.