The magic that isn’t
The recently updated Paper app by FiftyThree is a pleasure to use. For the (alas rare) occasions when I want to sketch, draw or scribble on an iPad, this is the app I have installed. So to start off, huge congratulations to the team who made it happen, it’s a success and an great example of usability, design and finesse! I also greatly enjoyed reading about their colour endeavours in the article “Magical Tech Behind Paper for iPad’s Color Mixing Perfection” in FastCompany Magazine, and I like the solution they came up with in the end: a hand-(eye)-chosen set of 100 color mixtures that are pleasing and good looking and serve to extrapolate/interpolate any other color mixtures they let us users play with. Ultimately the objective for an app like this is to give pleasing results and they certainly have achieved that with flying colors (excuse the pun)!
However, I also have a bone to pick with the article. I am sure I am not the only person who to point this out (even some of the comments on the article point in this direction): just do color right! I will try to highlight a couple of mistakes or misconceptions in the article’s description of what went on in the process of solving the challenge of mixing color (at least those I noticed at first glance, without going into a lot of analysis) below which stem from a fundamental lack of knowledge of color science.
At the expense of sounding apologetic, I am not taking anything away from the great solution the Paper app team arrived at – if anything kudos to them for coming up with the solution they did and for making the effort of dabbling in Color Science (as Julian Walker, co-founder and lead-engineer pointed out in a friendly twitter exchange) – I do question some of the things written about the various terms used and choices made… It’s also important to highlight that all my comments below are based on the FastCompany article and it is hard to know how much marketing speak or spin went into it (from FastCo or from Paper’s marketing team), so I certainly feel sympathetic with those who actually worked on the app in case their words have been twisted or misused - apologies to them in advance if this is the case. With all that out of the way, here goes:
Perhaps the most obvious comment to make is that to model colorant mixing you need to use subtractive color models. Adding a blue paint and a yellow paint together results in a paint that subtracts light in both the areas that the individual paints subtract light in and relate to their concentrations and relative proportions – an over-simplification but conceptually true. None of the color representations referred to in the article (RGB, HVS, LAB, LUV, …) are subtractive, which is why the “yellow plus blue don’t give me green” issue arises – linearly mixing RGBs and expecting a paint-like behaviour was never going to work as it’s not meant to :) Even taking a simple CMYK (e.g. a standard one like ISO Coated v2) space would have already solved this one issue alone…
Next, Kubelka Munk (K/M): an incredibly valuable and important model that lets you predict the reflective and/or transmissive properties of homogeneous colorant mixing (used in offset printing or custom paint mixing). This is certainly an interesting direction, but K/M makes several assumptions that make it more complex than what the article seems to suggest. What is the ink/colorant-set that is being mixed? In other words, are all colors derived from 3, 4, … 6 base colorants (inks/paints)? What are their characteristics? What instead is described seems to be a mixing of some form of RGBs. Now, if the combination of any two RGBs from the full RGB cube are supposed to be mixed using the K/M model, corresponding colorant characteristics are needed for each, such as the reflectances and opacities for those colorants to be able to predict the various proportions of mixture. This is certainly not impossible but is not trivial either (there’s a good conference paper to be written on this - the process is more or less clear to me but the beauty would be in the details). The alternative to this is to say (as mentioned above): all RGBs are the result of mixing a small set of base colorants - the way it happens in offset or paint color mixing.
So, mixing paint is a subtractive and non-linear (K/M is non-linear) process to begin with, but another issue is that K/M is designed to be a spectral model. While the article refers to some 6-dimensional parameters, even in the crudest representations used for reflectance and opacity it would be more like 16 dimensions per ink/colorant - a reflectance and an opacity, both defined on a 400nm to 700nm range at 20nm steps. So, mixing colorants is a subtractive, non-linear process that can be predicted using the spectral K/M model - there are additional assumptions about homogeneity but I think those can be assumed safe here.
That’s not the end though. Let’s assume colorant spectral data is available and K/M is applied for mixing, there is still another step which isn’t mentioned, which is to consider tone-curve processing like gamma correction that takes place on the display part and a simulation of dot-gain that would take place in a printing/painting part (the phenomenon whereby a colorant spreads into the substrate and results in both optical and physical effects altering it’s total color - more of an issue in print than in paint) - these are likely (some of) the causes the K/M transitions were allegedly too dark. Ultimately though, it’s crucial what color space the mixing is performed in and again this highlights that a simple linear combination (even in the right space) will likely not give good results as is.
The other thing to say though is this: what an app like Paper wants to achieve is to give pleasing, nice results to the users, not necessarily be color-science, optics or physiology-strict, especially given that color science is not a hard science. Color is ultimately the overall effect of physical properties of the environment (which in this case is artificial: the illusion of mixing paint on an iPad) as well as physiological and cognitive characteristics of and processing by the observers. Especially the last components make it clear that having exactitude is a real challenge in color science and may not even be a valid aspiration in many cases.
Which brings me to my last point: the final approach they used is a great idea and follows a long history of psycho-physical experiments driving a lot of color science! I’m really impressed by the team who have come up with such a great solution in spite of (although perhaps also partly thanks to) such diversion :)
There’s another question to be answered (a future post on this half-dead blog?): how would I have approached their challenge, other than hand-picking the mixtures? I have thoughts, but I’ll wait for another slow day at work… If anybody wants to chime in - leave a comment!
[UPDATE 1 - 14/11/2012: This was a post I wrote a couple of days ago on a slow-work day and (undoubtedly influenced by the joys of ‘planning’), made bold statements like having a “a suggestion that may have saved [the Paper app team] perhaps 6 to 9 months of their 12 spent developing the "magical tech” that led to the new color picker…“. Frankly I have no idea about their schedules or plans, how much time was devoted to the various avenues I argue above to be misguided… it makes for a good story to be sure, so it may well be the story-tellers view alone :) My technical points above stand valid, but I’ve edited the article slightly - also in response to a surprising level of interest.]
[UPDATE 2 - 14/11/2012: I came across a blog post by Matt Mattson looking at interpolation in various color spaces. This is a nice overview, although unfortunately doesn’t include any subtractive color spaces or ones related to K/M. However it does show that linear interpolation in various spaces means different things - i.e. 'looks different’. The fact that Matt concludes CIE LAB to be pleasing makes good sense: LAB is designed to be perceptually near-uniform (it has it’s flaws) meaning a euclidean distance of 1 in any part of the space approximately corresponds to a just noticeable difference - a good starting point for a smooth, linear transition.]
