Crunch Mode Blog - A State of Mind by Developers at D2Soft Technologies

Like us on Facebook

Displaying Real-Time Data Graphics

When it comes to understanding gathered data, graphics are the best way to represent them.  Graphics explain in a simple manner complex and heavy data.  I use graphics in my different sites as much as possible.

One big drawback of graphics is that they are costly to generate.  And as more data you gather, the more complex rendering your graphics gets.

Take my site for example.  One important part of being a community is to understand with whom you share it.  My users participate in this world-wide community and like to see where others live.  To represent this, I have created multiple graphics that display this information.  For example, here is a World map of all ratings done during the month of March 2007:

This kind of graphic is really cool:  in one second, I can grasp who else used the site during the whole month.

What you can't see from this picture, however, is that generating this kind of graphic takes a lot of processing.  Each dot on the map has to be calculated based on the geographical position of a user that rated his day.

Until today, I generated this kind of daily map once a day, at night, during low traffic hours.  The map was saved to a file and would not be re-generated afterward.  This was okay for a daily map, but for a monthly map with cumulative data, it's was not possible.

So the solution I used is to use pre-processing of data, occurring at relatively short interval to allow lowering the cost of calculating a huge amount of data.  Of course, pre-processing means that you must store this information somewhere, which is another drawback.  However, this is a small price to pay to have near real-time graphics on the site.  This process is done through background work so that the site is never slowed down, no matter how much data is processed.

My final word would be to always do as much pre-processing as you can if you want to be able to support increasing load on your site.  This way, you can save yourself long night of last minute optimisation :)