It is sometimes said that headphone amplifier output impedance doesn't matter much. The reasoning comes from basic electrical engineering and builds on the observation that headphone impedance is much flatter and higher than we see with speakers (a common point of reference). The problem is that while the reasoning is solid, the premises are wrong.
If a headphone has flat impedance, then we should see no difference in frequency response with different amplifier output impedances. The source of this is basically what is called a voltage divider, where the voltage drop from the amp output to ground is split between the amplifer impedance and the headphone or speaker impedance. Assuming the amp exhibits flat impedance with frequency, if the headphone has different impedances at different frequencies the voltage will be split differenty at different frequencies. That will lead to a frequency response difference from what was encoded in the input signal. But, if there is no variation in headphone (or speaker) impedance at different frequencies, then the voltage split is constant and the amp output impedance has no effect on frequency response.
A secondary factor to take into consideration is that headphone impedances (30-600 ohms) tend to be much higher than speaker impedances (4-8 ohms). If headphone amps had low (less than 1 ohm) output impedances like stereo power amps, then even with some variation in headphone impedance, we wouldn't see much impact. That's because the ratio of, say, 0.5 ohms to 600 ohms (.0008) isn't much different than the ratio of 0.5 ohms to 500 ohms (.001). This might be the case with some headphones. In contrast, the ratio of 0.5 ohms to 50 ohms (.01) is quite different from the ratio of 0.5 ohms to 5 ohms (.1). This is the kind of variation one might see in a speaker. In one case there is a 20% difference, in the other a 100% difference. Taking these as givens, one can see why amplifier output impedance would be a bigger deal in speaker-based audio than with headphone audio.
There are several problems here, though. First off, headphone impedance is not flat. A typical low impedance (30 ohms nominal) headphone might vary between 25 and 50 ohms impedance. Similarly, a high impedance headphone might vary between 350 ohms and 650 ohms. That indeed is less percentage variation than we often see with speakers, but it isn't zero variation. Some headphones vary more (e.g. 45 to 175 ohms), some a little less, but near zero variation is essentially unheard of.
The impact of this non-flat impedance case is exacerbated by the fact that headphone amps have much higher output impedances than speaker oriented power amps. Headphone amps are considered "low impedance" when they have 1 or 2 ohm output impedances. Medium output impedance would be 30-50 ohms, and high output impedance would be 120 ohms. The reason this matters is that the voltage divider effect is greater when output impedance is higher, all other things being equal.
In case this makes your head hurt, let's cut to the chase and look at some real examples. During my testing of the Audio-Technica ATH-W5000 I began to wonder if some of what I was hearing was due to amplifier interaction. The two amplifiers I was using (the PS Audio GCHA and the Grace m902) have output impedances of 44 ohms and 1 ohm respectively. Hmmm.... could this make a real difference?
Here is one view of this:
This graph shows the calculated difference in frequency response due to the different amplifier impedances of these two amps. Basically, the PS Audio amp will have somewhat more mid and upper bass than the Grace (and maybe a bit more upper midrange/low treble) with these headphones. 2 db is not a huge difference, but on the other hand it should be clearly audible.
To take another example, here is the impact of a 120 ohm output impedance on the Sennheiser HD800:
Note that in this case the impact in the bass is about half what we saw with the A-T 5000s. That's because, even though we've increased amplifier output impedance by about 3X, we've raised headphone impedance by about 10X.
So, it is pretty clear that different amplifiers can produce different results with different headphones, and this doesn't have to be due to magical qualities of the amps. There are a few other implications or issues as well.
First, we don't know in the above examples which case is "right", we simply know there are differences. Headphone manufacturers (or their distributors) seem loathe to specify what amplifier output impedance a given headphone was designed for. This is, to say the least, not helpful.
Second, a corollary of the above is that we can't say what amplifier output impedance is right. What is right is really what works with the headphones you have.
Third, it is hard to characterize how an amp sounds independent of specific headphones. We at least have to look at headphone impedances ranges to give an idea of what may happen. But even within an impedance range, amps will have different interactions with different headphones.
Fourth, we can say broadly that headphones with lower input impedances and more impedance variation will be more amplifier dependent. I should point out that this may be a bad or a good thing. It is bad in the sense that a review has to be taken in the context of how the amplifier impedance used for testing compares with your amplifier. As they say, "your mileage may vary". Impedance variation may be good for those users who like the ability to fine tune the sound of a headphone via amplifier selection (which is more difficult the flatter and higher the headphone impedance).