It doesn't seem like the hard work of setting up or taking down is the cause. I enjoy that mostly. There's something about the relative physical inactivity that is draining, especially if much travel is involved. Others of you might be able to relate to that.
It could also be that switching gears takes some mental time. Cleaning the studio and the house is a good way for me to transition back into the work routine, or pulling weeds. But there's still that day of sleep and rest.
After the French Revolution, out of rebellion against how the church had so been part of the system of oppression, a new week was established of 10 days instead of 7. It didn't last long. Animals used for plowing began to get sick or die from overwork. Our chickens, each one, will lay an egg 6 days out of the week, not every day.
After first starting to do pottery, any time away from it at all felt like a forced vacation, like when Mom used to make us take a nap. Boring, couldn't wait to get back into the studio. Now, however, it's become a refreshing break to hike, nap, read, etc. I even look forward to it.