I guess at its root the problem may just be how much compute is available to throw at the optimization. Alternatively there could be more efficient algos... I looked into but never fully tested this, it seemed promising:
https://news.ycombinator.com/item?id=16241659
Absolutely. Adding to the list, SIMPLE [1] was recently trending on HN [2]. It looks very applicable to hyperparameter optimization.
[1] https://github.com/chrisstroemel/Simple
[2] https://news.ycombinator.com/item?id=16241659