This is a new style of experimental blog post that I’m using as a way to retain the information I learn in my computer graphics course. In this post, I attempt to work out some questions that come up and the conclusions I’m able to draw from existing research. If there are questions, comments, or I’m completely wrong – please let me know!
Although my primary language when writing code has traditionally been C#, I’ve been working in C++ for a class and I came across a problem that I had seen before: when I’d coded a color for an object in my scene, it showed up as pure white. I was thinking in the RGB color spectrum as being 0-255, but I realized in both cases that the computer wanted the color value to be between 0-1. I was visualizing my colors as integers, when the code was actually using floats.
This got me thinking – why would we even want to store the values as floats in the first place? Well, it turns out the human eye can see approximately 10 million different colors, and integer representation of the RGB spectrum , using the numbers from 0-255, repetitively, allows us over 16 million ( 256 * 256 * 256 color value combinations). In theory, this should be able to be represented with integer values, because we don’t have a great enough ability to distinguish between more precise color differences than those allotted by what can be stored in an integer. Instead, many standards for color representation in code require that colors are stored as floating point values (32-bits) to provide a range of precision that far outpaces what our own eyes are able to distinguish.
vdb_color(31.f/255.f, 119.f/255.f, 180.f/255.f);
-C++ visual debugging VDB tool is one example where colors are represented as floating-point values
The Unity game engine also stores colors this way. Given that color values are constantly being evaluated and updated within a game scene in order to render properly, function calls are being made with floating values instead of integers all the time.
Today’s photorealistic rendering techniques require a significant number of color manipulation operations, especially within dynamic scenes with changing lighting and perspective. Because we do not preemptively know where the user or player will look, we must constantly calculate the correct color of an object at any given time, in as close to real-time as possible. These calculations are not cheap – any gamer can tell you the tradeoff with performance and graphics, it’s the reason your laptop can’t play League of Legends with Very High settings.
So why are we using float values to store colors we can’t even see?
| 0 – 255 | 0 – 255 | 0-255 | :
(16-bit, 16-bit, 16-bit) Thanks Michael Mara (@mikemx7f) for correcting this: (8-bit, 8-bit, 8-bit)
Floating Point Representation:
| 0 – 1 | 0 – 1 | 0 – 1 | : (32-bit, 32-bit, 32-bit) – 2x as long!
It seems as though there really isn’t a reason to use floats over integers, from our visual perspective. It doesn’t give us a change in what colors we are able to see regardless, and most monitors won’t even display that many different colors, even if we could see them.
There is one factor, though, that comes into play here: GPUs. These are generally optimized for floating point operations, so for hardware optimization, there may be performance gains on using floating points. This makes sense but would still be widely dependent on the manufacturer – that said, it’s one key to why the graphics architecture has been built this way.
The question still remains: regardless of why we’re using floats instead of ints, is it really even important, past what the computer does? Yes, it is.
It turns out that precision of color is more than just “how many colors” can we see – and in the case where floating point numbers represent the individual values of a color, it isn’t that we need to see all of the potential range of colors represented that way, but that the precision offered by floating point values allows us to get a much more detailed and finely-tuned image than that which is offered by integer values.
Consider an HDR image as seen above. If you’re familiar with image manipulation and photography, you likely already know that HDR stands for high dynamic range, which generally makes images “pop” and provides a different look and feel to a picture. Using floating point values, rather than integers, allows us to process an image in this way because floating point operations can store much more precise values – and although we might not be able to perceive every possible color difference on the spectrum when incremented by the delta permitted with floating point numbers, when you take this in the context of an entire image, the result is a much greater distinction between samples, even to the naked eye.
HDR is just one example of the way that floating point operations lead to a more flexible image. If we were only using integer representations of color values, images with a high range of different colors would not be as easily represented, and the rounding / truncation done with integer operations would result in artifacts and potentially inconsistent colors throughout the image.
At the end of the day, though – why does all of this matter?
Consider the effects that go into creating a realistic environment – or even a low-poly one, brightly colored, similar to the one found in social VR application Convrge:
The image above shows a scene with several ambient lighting effects, a variety of colors, and a good number of shadows. Whenever a player looks around the environment, the computer must do a lot of calculations – quickly – to determine what color each pixel on the screen is, and draw updates accordingly. Even without photorealistic renderings, this is done repeatedly, and takes into consideration a huge variety of factors: multiple light sources, the orientation of the HMD the wearer has on, whether or not the environment has dynamic colors (such as is the case with Convrge’s light show dance floor). Graphics cards are optimized for floating point operations because if they weren’t, these calls would be “dropping” pixels in a sense – renderings may not be as smooth, because integer operations have to make compromises and round a lot more. For virtual reality, this is especially important: random dots of incorrectly colored pixels throughout your scene would, perhaps, not be noticeable on a frame by frame basis, but the inconsistencies might be enough to prevent immersion.
The good thing is that we don’t really even need to worry about this – as I mentioned, this is the standard, and hardware has been optimized to support floating point operations anyway, which can be parallelized. So, although we will continue to think about colors in the RGB(A) space as whole, real numbers between 0-255, our code will continue to get more precise than we do – which is not out of the norm nor is it unexpected. Thank your friendly neighborhood GPU, folks – high dynamic availability with digital processing thanks to floating point operations is the reason we get stuff like this to look as awesome as it does:
A special thanks to Matt Green (@IAmMattGreen) and Stanford University’s Manolis Savvas for their help with understanding the concepts in this blog post.