I've done this exercise and generally what you want is a combination of your suggestion (which is elaborated by 'polyphora[0]) and 'p1necone's suggestion [1]

Polyphora mentioned CIELAB as just one example, and it's a good example. I believe state of the art these days is Oklab[2], talked about here[3]. I'd like to pull out a comment from 'jiggawatts in that discussion:

> This is a tour de force of colour theory, and should be mandatory reading for anyone serious about computer colour!

I completely agree.

With regards to 'p1necone's suggestion, k-nearest neighbors is one simple and relatively easy way to separate the colors into bins. I've only done this on a single image, but with multiple images maybe you could also k-nn bin the resulting colors from each image and only return bins which have multiple members.

0: https://news.ycombinator.com/item?id=25828733

1: https://news.ycombinator.com/item?id=25828773

2: https://bottosson.github.io/posts/oklab/

3: https://news.ycombinator.com/item?id=25525726

See the recent Oklab post[0] for guidance on choosing a perceptually uniform color space for gradients. It's better than CIELab and CIELuv, both of which I would consider inferior to newer alternatives. In particular CIELab has particularly bad hue shifts in the blue range.

I'm also working on a blog post on this topic (there's an issue open in the repo for my blog, for the curious).

[0]: https://news.ycombinator.com/item?id=25525726