According charts in google, a jet engine at 100 yards distance is 105dB louder than a whisper. 16 bits can do 96dB without dither.
More importantly, the problem isn't trying to hear 96dB of range all at once. The baseline is fixed in place, so when you have a quiet section all those high bits are 0 and you need the low bits to have enough detail by themselves. And a whisper is pretty far from the quietest thing you might have in a track.
If you can hear the dithering noise, then I'm pretty sure there are sounds you're hearing that would be wrong or missing without the dither.
More importantly, the problem isn't trying to hear 96dB of range all at once. The baseline is fixed in place, so when you have a quiet section all those high bits are 0 and you need the low bits to have enough detail by themselves. And a whisper is pretty far from the quietest thing you might have in a track.
If you can hear the dithering noise, then I'm pretty sure there are sounds you're hearing that would be wrong or missing without the dither.