After seeing some posts online on the clustering of dominant colors in movies presented in a very appealing way, I decided to improve upon my original movies fingerprint algorithm.

The first thing I realized whilst looking at other persons’ implementations was that they tended to focus on one color on each of the sampled movie frames, which simplified the presentation of the results. The second thing, that they presented the results in a more appealing way than I originally did.

To remedy the second point, I settled on doing square representations of the colors with a title overlay. In these representation, each vertical bar stands for a sampled frame in the movie:

Now, on the trickier part, I looked at my code and realized that it worked fine with the case of one cluster for the whole frame. This, however, has one important drawback: as it assigns every pixel to the same cluster and then returns the centroid, so it pretty much calculates the average color of the image (as in the example shown above). This is not a problem for cases in which movies are very skewed in color-space, and is, in fact, the way many of the algorithms found online work; but, I wanted this algorithm to be more general and to be able to detect several clusters even if it only returned a subset. For this extension, a couple of changes were needed: we needed to calculate and detect the required clusters, and then return a subset.

As we can clearly see in the image below, the resulting colors were far more bright, as compared to the original that looked “washed away” due to the centroid being “pulled” from the detected color towards the other colors present in the frame.

For bonus points on the scripts, I decided to parallelize the processing of the frames (as I was a bit tired of waiting for scripts to run). Doing this was a bit trickier than expected, as it required the use of a memmap to be able to modify a common numpy array from different threads.

Finally, I coded a bash script to automate the whole process of re-scaling, extracting frames, and exporting the fingerprint.

Documentation and Code

• Repository: Github repo
• Dependencies: opencv-python, ffmpeg-python, Pillow, numpy, scikit-learn, matplotlib, ffmpeg