It seems that our brains are not our only body parts with a clock. In fact they are ubiquitous at the cellular level. Evidently circadian rhythms developed pretty early in evolution and were strongly conserved. We know something about this partly from studies of caffeine.
From Rachel E. Gross's story in Slate:
Caffeine was already known to alter the circadian clock in red bread mold, green algae, fruit flies, and sea snails. But humans were liable to be a little different. For the first half of the study, published Wednesday in Science Translational Medicine, researchers at the University of Colorado–Boulder measured how caffeine influenced the circadian rhythms of five human caffeine consumers over 49 days.
Perhaps you already expected that there would be an effect on humans, and if so, you weren't wrong.
It also seems possible that the various clocks in the body might get out of sync after multi-time zone travel.
The way caffeine works on cells in the body might be different than how it works on the brain. “Your liver has clocks in it. Your muscles have clocks in them,” says Wright. “We know if you jet-lag a mouse, the brain adapts really quickly. Whereas it could take time for the other tissues to catch up. In other words, jet lag isn’t just the fact that your brain is in another time zone—it’s that your liver might be in a different time zone than your brain.”
Think of it as the technological equivalent of splinching - your brain is already home, but your liver is still in London or Tokyo.