In hindsight, it’s obvious. Switching the ADC reference voltage takes a bit of time, the voltage on the AREF pin cannot change instantaneously. The Atmega48/88/168/328 datasheet recommends to discard the first sample after changing the reference voltage (in paragraph 24.5.2). So I did. But still I got false readings (or rather: my Atmega48A did).
I’m using both the internal 1.1 V reference, and the AVCC. AREF has a capacitor of 100 nF to ground connected to it. AVCC is connected to VCC, and is about 3.4 V.
I measured the voltage on AREF, to see how quickly this changed from AVCC to 1.1 Volt when the reference is switched from AVCC (REFS=01) to internal 1.1V (REFS=11).
Turns out AREF did not reach 1.1 V at all, no wonder my ADC result was inaccurate! (Also: changing from 1.1 V to AVCC is a lot quicker.)
Based on the oscillogram above, AREF drops about a volt in about a millisecond (and raises a lot quicker). Waiting a few (10-ish) milliseconds between the first (‘discard this one’) and next (‘use this one instead’) ADC samples indeed gave a usable reading.
Guess I should have asked my rubber ducky, because at first I was looking at issues with clock switching, but a simple test project with only the ADC revealed that all the other distracting bits like clock prescaler changes or interrupts had nothing to do with it, and it is simply a matter of the internal reference not sinking (or sourcing) infinite current into a certain capacitance. It just needed a bit more time.