Servo based systems are both incredibly power and incredibly fast. So fast that if you've never seen a servo based arm move at top speed and come to that top speed at top acceleration you'd be forgiven to think that I'm exaggerating.
Depending on the application there are different layers of safety surrounding these systems, including perimeter guards, optical barriers, limit switches, resistance based detection and so on. But when a system is broken someone has to go in and fix it, and you tend to do that with the robot powered up, some of the safety systems disabled so you can actually work on it and if you're really unlucky a motor will end up shorted against a + or - rail while you're within reach. This is obviously dangerous, and it is more dangerous because broken equipment can't be trusted to behave in a predictable way.
They won't stop. Not until whatever is obstructing has moved or the motor has burned out (or someone has the presence of mind to hit the e-stop). I've seen a 3" thick mount that must have weighed well over a ton sheared clear of its bolts (which themselves were an inch thick) by a malfunctioning servo on a very large lathe under construction (think 8 foot chuck for crane cable idler wheels). Do not fuck with servo systems unless you are 100% sure they are safe to approach or you may well end up dead or gravely injured.
I just want to echo the 'so fast you'd be forgiven to think that I'm exaggerating'. I work with industrial arms every day, but pretty much never with them running at full acceleration up to the speed limit between the extents of reach. Even when they bump you at a speed that would be like a mild shove from a person, the unyielding force is remarkable, and if you're ever near one and surprised by a sudden 'full send' motion, it's terrifying.
Terrifying is the proper term. The only time I've ever been really mad with a co-worker is when they decided to do a demo to potential customers with the newest (pre-release) firmware of a massive lathe without my very explicit permission. That could have well ended up with people seriously injured, as it was - of course - it happened to have a pretty bad bug in it but besides people getting scared it didn't cause any damage or injury.
My own process to ensure things were safe was to first run it in air without the spindle powered up, then on foam, chalk, and finally on aluminum in that order before progressing to steel.
I had the pleasure of touring BMW’s Welt manufacturing plant while I was there and it was the first time I’d seen manufacturing robots in action. Even standing well outside of their reach zones and behind at least two layers of fencing, their movement was terrifying. Precise, powerful and completely without any awareness of their surroundings. They remind me of horses to be honest.
All of that is to say that it seems like working on a manufacturing floor with robots has the potential to be so so dangerous. Getting into a rhythm and doing repetitive work all day, it’s easy to imagine slipping and ending up as human soup.
There's a catch 22 when repairing machinery, you have to see it in action to determine what is going wrong, and sometimes this means getting up close with it during operation. I've seen many techs/mechanics bypass guards and not use lockouts when diagnosing machinery problems. And of course everyone is "ok" with it because they are supposed to be experts and know what they are doing.
> There's a catch 22 when repairing machinery, you have to see it in action to determine what is going wrong, and sometimes this means getting up close with it during operation.
This is 2023 and we are well into the age of ubiquitous, cheap, small cameras.
Place cameras such that paths can be observed at a safe distance and diagnose problems in powered machinery without placing a person in harm’s way.
Between depth of focus, various obstructions, details that are too small to notice until they bite you, multiple angles, very large spans of travel etc I don't think that's feasible. It's one of those 'why don't they' statements where an outsider thinks they have an obvious solution to something when the experts in the field are anything but stupid.
What does work is to disconnect the motors while keeping the rest of the electronics powered up but not all industrial robots have such a facility. The e-stop will disconnect everything and that stops you from doing things like firmware upgrades. And when things are broken it gets even more complicated and unpredictable.
Servicing machinery like this is difficult, sometimes dangerous (but not even the most dangerous, for that you have to go visit a steel mill) but not impossible. The biggest danger really is familiarity with the machines to the point that you stop to respect them, that's when you are really in danger. Personally you can't pay me to go near one when it isn't locked out, I'd much rather field strip it and test the components one by one than taking a risk but a service tech might be promised a bonus if they can get it working again quickly and that might cause them to work in less safe ways.
I've seen people do incredibly stupid stuff with machinery and I've also seen the results in terms of fingers and sometimes eyes or whole limbs lost as evidence of prior fuck-ups. And some of those people still took risks afterwards.
Walk into any metal machining shop and just watch, it won't be an hour before you see someone do something that they shouldn't be doing. And in almost all cases it will end up without anything being damaged or someone being injured. One more step on the 'normalization of deviation' track. It always ends the same.
I’ve never seen a company reward a person for taking extreme risks. Usually they get coerced threatened or do it out of their own volition. The latter being very common just to make the big easier dispite being an hourly employee
I've seen the weirdest things (but then again, I'm old), including a guy that got a premium per knife fuse replaced (because they're known to occasionally spew fire upon re-insertion). There isn't much context here but between complacency and tiredness there are plenty of ways already in which this could have gone wrong, but outside factors such as pressure (this is Asia, after all) may well have been part of it.
And before anybody sees that 'Asia' comment as somehow politically incorrect: the statistics are pretty damning, Asian operators abroad have less than stellar safety records and very high pressure on industrial workers to meet their (sometimes unrealistically high) quotas and I would not expect them to do any better in their home countries.
I know parent said “see it”, but a more accurate phrase would be “experience it”. What’s going wrong can be very nuanced, and it’s a bit naive to think you could diagnose complex industrial machinery by watching a few cameras.
I would say seeing is nothing here. It will be a lot about vibrations and sounds. Now you could also use pickup mics for this, but I guess it is sometimes just more straightforward to listen with your ears and avoid another layer of indirection.
Now experts will also do stupid shit, but they are the ones who should be able to judge what risks are involved.
At least as a software developer, when I test on production because that's the only place the bug manifests, I'm not risking my life to reproduce the issue.
> I thought OSHA and friends didn't allow this. Lockout/tagout is standard.
Yes it is, for plant workers it is a firing offense in many places.
But your typical maintenance tech that is supposed to work on the machinery itself rather than just to be using it may well use lockout/tagout to ensure that the machine is powered off when they are working on it and don't need to be able to do any kind of diagnostics. But short of a complete disassembly and testing each component in isolation - for which there often isn't really time and which given the pressure on maintenance technicians to get a piece of gear working again - people tend to take shortcuts such as to hook up analytics gear to a machine that is live. Stupid? Yes, absolutely. But this kind of stupidity is the result of usually many years of things working just fine and bad practices creep in. The guy that manages to get stuff done rapidly is the one that gets called out. And eventually an accident will happen. Weirdly enough even near accidents tend to reinforce the belief that it worked. When actually the lesson should be that it didn't but the person just got lucky.
One very memorable occasion at an industrial plant I visited is that someone got beaten up for removing a lock and I felt absolutely no sympathy for that guy. He could have well gotten someone killed (large 5 axis mill).
Yeah, that tracks. Workers in plants like that don't have a lot of patience for someone trying to get them killed. People always accuse manufacturing and construction unions of "using safety as a cudgel", but the reality is that a lot of these people have lost friends to industrial accidents and they're elected by workers who care a LOT about safety. Sure, they're political and they are going to bargain and pressure in labour disputes, but they are also genuinely fanatical about safety. Some of the most ugly union/management interactions I've ever seen was union safety reps who had personally seen their comrades die in industrial accidents arguing with management who had a bad safety track record and very much gave off the vibe of neither caring nor understanding as to why you couldn't lift four-point hoisted concrete slabs within horizontal metres of workers at ground level, no matter how tight the construction space is. The reason, if anyone is curious, is that suspended heavy slabs where one or straps break won't fall straight down, they swing away from the side of the straps that broke and then swing back, and the remaining straps past their safety limit can break or lose hold at any time. You don't have to be underneath a multi-point load to be able to be crushed by an accident, you just have to be in the general vicinity, so to do it safely you have to evacuate said general vicinity and have an on-site safety officers who understands the physics of the thing to make sure people are standing far enough away.
I've almost f'd up myself using a portal crane indoors. I lifted a relatively light rotor of a windmill onto the stator only to find out that the magnetic field of the rotor was strong enough to lift the entire assembly, including a 500 pound welding table clear off the ground when it came close enough. Seconds prior my fingers were in the airgap between them (still quite large). I had to take a long break after that before going back in, the surprise factor really got me, I absolutely never saw that one coming and a very large part of working safely is to be able to predict the failure modes. I hadn't counted on 'anti-gravity'.
Roughly halfway into your sentence about the concrete slab my mental kinetics prediction model already declared a zone described by the pendulum at maximum extension (hanging from one remaining strap) as no-go area. And depending on the state of the crane and how far debris could have been shot out from a falling slab that area may well have had to be much larger still. People that don't understand such dangers should not be in management positions, which is one of the reason why I'm always happy to see industrial companies that promote people from the ranks to management rather than to bring in outsiders with only theoretical knowledge.
Yeah, "only theoretical knowledge" is exactly how I would describe this guy. He was complaining later that the unions had intimidated him, that they were a gang, and they were economic terrorists besides. I was there for the whole interaction, the only reason he felt intimidated was because he was a spineless coward trying to argue with people who knew what was going on while he knew nothing. He felt bad because he was dumb and in charge and he blamed that feeling on those under him.
People actually die because of characters like that. It's interesting that right here in this thread there are people arguing that industrial workers can't be pressured because of OHSA but in fact the opposite is true: short of a shutdown OHSA doesn't really do much that scares employers, they might issue some minor fine (a few thousand bucks at most in almost all cases for injuries and sometimes even death) so they feel just fine about putting the screws on employees and in states where you can be fired for anything people will definitely cut corners if they believe their employment is in danger if they don't take risk.
Similar humbling experience for me a couple of decades ago. I made a mistake when programming the toolpath to cut a large aluminum part on our Haas VF-3SS machine. I didn't know about a bug in the CAM software and trusted the simulator.
The machine plunged into a large block of aluminum at full rapid speed with a 1 inch rougher end-mill. It then proceeded to run the cutting program, again, at full rapid rate. The sound it made is something that I will never forget. This thing was churning aluminum like it was butter. It was literally turning it into goo, not cutting it at all. It did not care one bit. The cutter was large enough and stubby enough that it did not break.
These things are fast enough that, by the time your shocked brain is able to respond it could have killed and dismembered you five times before you can even think of slamming the e-stop button.
The end-mill came out with a ball of aluminum, 2 inches in diameter, permanently welded to it. It was one of the scariest industrial accidents I've been around. Thankfully the enclosure and rigidity of the machine made it so that the only consequence was aluminum that looked like churned butter.
Years later I worked with an aerospace client who uses stir welding to assemble rockets. Same process, except you are doing it on purpose and with controls. The idea is that stir welding preserves the metallurgical properties of the material, whereas TIG/MIG welding might not preserve properties important to the design.
To echo your other comment, since that happened I learned to never trust CAM software or simulations, always run it on air first and, if necessary, machinable wax blocks. Decades later, I still don't consider myself a machinist at all. I am pretty decent at it. It's a complex art that requires time and dedication. I do too many other things to be that guy. I do have a VF-2 and a Bridgeport knee mill (full manual still) in my garage though...
It's close calls like that that are the best vaccine against getting over-confident.
I have a 30 Watt diode laser that came without any safety measures at all. It kind of surprised me that you could just fire it with a G-code and absolutely nothing stood between you and the beam. It cuts wood like butter though, up to 20 mm in a single pass (solid wood, not birch ply) and I'm super happy with it but that tool is going to go through some serious modifications to make it safe. It can set just about anything on fire fair distance away and will fry your eyeballs from half a kilometer if not more. Any idiot with a few bucks to spare can go and order one and they have absolutely no clue about the kind of damage a machine like that can do. It looks so innocent, it's completely deceptive.
>> by a malfunctioning servo on a very large lathe under construction
I got hit right in the dick by a milling machine axis lever spinning up full throttle.
Only once, never again. Still surprised something terrible didn’t happen - hurt like a bitch. Shop dude says it happens pretty often though because people don’t look down at the caution tape telling you not to stand there.
People have waaaay too much faith that motorized devices won't injure or kill them. Even motorized lie flat seats in an aircraft can probably break something. Many of these devices rely on simple current limiting. While this can help, its definately not a guarantee that it won't kill you.
Reminds me of the car salesman who put his head in an autoclosing minivan door to show off the safety sensors. Sure, it didn't kill him. But having your head pressed between two thin metal plates until current limiting detects the skull in the head is gonna suuuuck.
>One day in 1979, a robot at a Ford Motor Company casting plant malfunctioned—human workers determined that it was not going fast enough. And so 25-year-old Robert Williams was asked to climb into a storage rack to help move things along. The one-ton robot continued to work silently, smashing into Williams’s head and instantly killing him. This was reportedly the first incident in which a robot killed a human; many more would follow.
That's in Amsterdam, and it is one of the most stupid designs ever. Still amazes me it got approved. I can think of so many ways in which that could cause problems it is a study in irresponsible engineering.
it's not even the first this year. But yeah, from https://www.bbc.com/news/world-asia-67354709 (non-paywalled coverage of the same story) it sounds very much like not doing lockout/tagout or at least hitting the e-Stop before inspecting something.
> it sounds very much like not doing lockout/tagout or at least hitting the e-Stop before inspecting something
That was exactly my thought when reading it. If the e-stop had been depressed this likely would have never happened. But with the system live all bets are off, especially if it is already malfunctioning. You're essentially one queued G-code away from getting mauled.
Article is missing some key details. Every industrial robot that I've seen has at least a line painted in the floor, or more often a physical barrier, and workers are required to stay behind it when the robot is powered on. Did this factory not have such basic precautions, or did the dead worker violate protocol, or did another worker turn on the robot at the wrong time?
Korean versions of the article have more details. He was from a robot maintenance company visiting the factory to perform a software update. He was working on the robot alone and checking the sensors when the incident happened.
Hmm, that would suggest that he both had better training about the robots (compared to a random food-packing-plant worker) but was also more comfortable around them (which is a huge risk with any powerful equipment, including automobiles.)
It could well have been a maintenance situation. Those tend to be extremely tricky to do safely, in spite of all the precautions, especially if you want to do some diagnostics without taking the whole thing apart.
I have religious respect for those arms, if they're powered up you couldn't pay me to get within reach.
All of those are possible, but food handling usually uses less powerful robots (simply because they're a lot cheaper) so there may have been some complacency about them not being that dangerous.
(Physical barriers can be tricky to get right too, though - there have been incidents of "worker crushed between robot and inside of safety cage" though design standards have been updated in response to that.)
Any idea what model? The bbc picture looked like implausible artwork (way too much expensive hardware in-frame for a "replace cheap humans with cheaper robots" scenario) but I could see them picking a low-end industrial/automotive arm instead of a "collaborative" arm...
I've been trying to figure out the brand but enough of the text on the arm is obscured that I can't make it out beyond NZ...MI or something to that effect. I'd love it if someone could figure out what make and model it is. It's a palletizing robot, it looks superficially quite a bit like the Kawasaki models but the brand name fragment clearly doesn't match that (and that would be Japanese, not South Korean).
Ooh, thanks for the reference. Kind of looks like https://www.zonesunpack.com/products/zonesun-industrial-arti... - that four-bar linkage for the arm segments isn't unique (as you mentioned, the big kawasaki arms use it too) but it's really not something you expect in human accessible spaces, both for pinch hazards and just catching on things...
Ah, found nachirobotics with a matching logo (in a video about a refrigeration factory modernizing their automation, but it was on a welding robot there.)
Yes, you've found the brand, thank you! I thought I new most of these by now but this one threw me.
That thing has 3 meters of reach, so a dome with a 3 meter radius would be within reach of the arm at maximum extension. You might feel quite safe with the arm retracted or pointing away from you but all it would take is a limit switch that is busted for it to be able to reach you.
You couldn't pay me to get near one of these with the power on.
This would be an absolutely horrifying way to go. I hope that whatever his beliefs he finds peace, of if he were an atheist, I hope that memories of him live on in those who knew him.
It used to be something that you only really and truly understood when it happened to your buddy or somebody your buddy knew. Now we don't need somebody we know to get blown up in an arc flash incident or grabbed by a lathe to actually understand the forces at play. It's awful and you shouldn't subject yourself to very much of it, but if you're around these machines, it's critically important that you have visceral fear of what comes after a mistake.
> The man, who was in his 40s, had been inspecting a problem with the robot’s sensor at a distribution centre for agricultural produce in South Gyeongsang province
So not that it diminishes the tragedy but it sounds like this was a worker that was working on a malfunctioning piece of machinery and that's what caused the accident.
Not a case of ML vision misclassified something, but rather an industrial piece of machinery malfunctioned.
A manufacturer of office gear in the Netherlands had one that would 'every now and then' come down randomly. It turned out to be a relay in an ancient AC unit that generated so much interference that it triggered the control of the press. Super dangerous, and very hard to trouble shoot what the cause was.
Why is this in the news? The worker didn't LOTO or wasn't given provisions to, fucked up, and the inevitable happened. I guess it's kind of noteworthy that this same facility has had it happen twice?
technically LOTO is to lock the switch with a padlock in the off position and tag the lock with an identifier so the plant knows who is doing the work but overall generally right.
I do not have faith in worker protection laws and safety culture in Korea.
Many do not spend the money to prevent similar deaths from happening again and I boycott several brands and brands owned by those groups.
https://www.bbc.com/news/world-asia-67354709 also quotes "South Korean news agency Yonhap", and quotes statements from "the Donggoseong Export Agricultural Complex, which owns the plant" - and gives "In March, a South Korean man in his 50s suffered serious injuries after getting trapped by a robot while working at an automobile parts manufacturing plant" as additional detail of the previous incident. (That may not be enough to find earlier articles, or maybe they're just eclipsed by the current one.)
Depending on the application there are different layers of safety surrounding these systems, including perimeter guards, optical barriers, limit switches, resistance based detection and so on. But when a system is broken someone has to go in and fix it, and you tend to do that with the robot powered up, some of the safety systems disabled so you can actually work on it and if you're really unlucky a motor will end up shorted against a + or - rail while you're within reach. This is obviously dangerous, and it is more dangerous because broken equipment can't be trusted to behave in a predictable way.
They won't stop. Not until whatever is obstructing has moved or the motor has burned out (or someone has the presence of mind to hit the e-stop). I've seen a 3" thick mount that must have weighed well over a ton sheared clear of its bolts (which themselves were an inch thick) by a malfunctioning servo on a very large lathe under construction (think 8 foot chuck for crane cable idler wheels). Do not fuck with servo systems unless you are 100% sure they are safe to approach or you may well end up dead or gravely injured.