This is the question asked in a paper by Essex, McKitrick and Andresen in a fascinating paper which can be found here. (Mathematics alert!). This is my understanding of it - I haven't done any maths since university days, so if I'm wrong I'm sure someone will put me right.
Some quantities, like weight, can be added and therefore averaged. If you take an 2oz mass and a 1 oz mass you can say with certainty that their total mass is 3oz. Because the sum of the two masses means something, you can calculate an avarage of 1.5oz and this figure has a useful meaning also. These kinds of measures are called extensive variables. Pressure, on the other hand, can't be treated in this way. If you add a system at 2 atmospheres to one at 1 atmosphere you don't get a system at 3 atmospheres. Because the sum of the two pressures has no meaning, the average likewise is meaningless. These are called intensive variables.
Temperature, as you might suspect, is an intensive measure. This means that when you add two temperatures together, the answer cannot be a temperature. It's meaningless. As the authors point out, dividing this meaningless sum by the number of components cannot give you an answer which has a meaning.
If the average of temperatures is not a temperature, then perhaps it's an index - a number which tracks whatever it is that drives the climate? If this is the case, then it is presumably necessary to describe how the average of temperatures - a statistic - is driven by the underlying climate driver, or at least to show some correlation between the two. They also need to demonstrate that the statistical measure they have chosen is better than any other measure they could have chosen. These alternative measures might well demonstrate a completely different trend to the average.
A third alternative is that the average is neither a temperature or an index, but a proxy for energy. But unfortunately there appear to be problems with this argument too. For a start, to do so is to use an intensive measure as a proxy for an extensive one. Secondly, the relationship between energy and climate is not understood. How then is it possible to know that the average of temperatures is a valid proxy?
It's not instantly obvious to the lay reader, but there are lots of different kinds of means. We're used to dealing with arithmetical means ("averages") but you can also have geometric means, harmonic means and any number of other means. For some systems, physics suggests which is the correct one to use. But, alas, this is not the case for global temperature.
As if to rub this point in, the paper demonstrates that there are in fact an infinite number of different means for global temperature. Which, they ask, is the correct one? Why has the scientific community alighted on the mean it has? They go on to show that, for the same set of data, different means can show a rising trend or a falling one. In other words, if a different averaging method to the one used in climate science had been chosen, we might now be having a crisis about global cooling... again.
It's a fascinating piece of work, some of which is beyond my understanding. If you are mathematically inclined, do take a look and tell me what you think.