Stress is cumulative with most aquarium residents. The more you handle it, the more stressed it will become. To oversimplify, a hormone is released by organisms in response to stress. Think our adrenal system, but instead of thirty seconds of adrenaline followed by a day long crash, stretch that whole operation over a month.
So you handle an organism, it releases this hormone in response to the stress. It takes several weeks for the hormone level to return to baseline, i.e. the organism to recover from the stress event. If during that time, the organism is stressed again, it will release the hormone again on top of that which remains in the system. So if it doesn't reach baseline before being stressed again, you will end up with higher overall hormone levels than the first event. If this keeps happening, your organism will reach a concentration of this hormone that it cannot recover from. Again, way oversimplified, but essentially the reason that change is bad...
The recovery time varies greatly depending on the organism and the severity of the stress, but this is the basic response of any organism to stress.
Cal Fish and Game did a big impact study with hook and line fishing and cortisol (stress hormone in vertebrates) levels in sport fish and found that it takes most fish almost two months to return to baseline after the stress of being caught via hook and line. I can only imagine how long it takes after being netted, thrown into crappy water at the collection facility, starved, bagged, sedated, shipped across half the globe, thrown into different crappy water at the wholesaler, bagged up, brought to the pet store, dumped in even more nasty water, bagged yet again, and thrown into our tanks...
Poor little fellers...