This article uses the term "evolutionary algorithm" far too narrowly and it is causing a lot of confusion in the comments. I would STRONGLY recommend checking out the book "Evolutionary Optimization Algorithms" by Dan Simon[1]. It is incredibly readable. The type being referred to in the article is the classic "Genetic Algorithm" variant of evoluationary algorithms. But genetic programming, evolutionary programming, simulated annealing, ant colony optimization, particle swarm optimization, differential evolution, estimation of distribution algorithms, biogeography-based optimizations, cultural algorithms, and opposition-based learning algorithms are just a few examples of other types of evolutionary algorithms.
In general, they are a great approach to solving any non-convex optimization problem where gradient descent is not a practical choice.
From the evolutionary algorithms perspective, the work presented in the article is tremendously ordinary, even formulaic. There’s always somebody publishing a new evolutionary search operator with a flavor of the month biological analogy. The test functions are bog standard, and although they’re designed to present certain difficulties, they do a pretty bad job of resembling actual black box engineering models; their main advantages are fast evaluation and analytically known optima. Your best bet for single objective optimization is probably still CMAES.
Can confirm. I've played around with differential evolution and a few others. Nothing was able to beat CMA-ES. At best, after some manual tuning it was just barely matching the untuned CMA-ES. (The task was to optimize a small neural network, gradients not easily available.)
CMA-ES is not suited for very large problems (more than a few hundred parameters), but there are several variants that can handle this, like sep-CMA-ES.
Differential evolution definitely comes in second as far as I’ve seen. In fact, what I’ve seen is that evolutionary search operators are effective inversely proportional to how faithfully they model actual biological processes.
No, the comments are not confused. I wouldn't consider myself as a expert, but I've worked a lot with evolutionary algorithms. This is not a good work and should be rejected from ICLR, although again it is my personal opinion. Terminologies are there to help distinguish between different concepts, not to be obscured in some handwavy way to seem deep and knowledgable to sucker some ICLR paper reviewer with no scientific rigor. It is indeed possible to optimize NNs using GAs but claiming diffusion is an evolutionary algorithm is a huge stretch, and they even didn't draw the connection properly in the paper. There might be something worthwhile in their work but I would say portraying it this way is just flat-out not correct.
I think we are agreeing. I'm not claiming the comments are incorrect in the regard you are mentioning, I mean to say that the paper is conflating genetic algorithms (GAS) as tho they are the only sort of EA.
Words mean things and my goal is to enhance the discourse by clarifying the imprecise language used in the paper regarding the term "Evolutionary Algorithm".
I make no comment on any other issues with the paper.
In general, they are a great approach to solving any non-convex optimization problem where gradient descent is not a practical choice.
[1]: https://books.google.com/books/about/Evolutionary_Optimizati...