I sometimes do leave Seymour in the fridge for a while, but I have noticed my bread rises much higher when I feed him twice before using him for bread. My husband is a bread-a-holic (the Dutch tend to eat a lot of bread) so he is always happy to help out with eating the stuff I need to bake to use up the discard. A couple of times, I've pushed it too far and have had to baby him back, so now I'm super cautious and protective of him. :)
I should add that in the summer he spends much more time in the fridge. Right now, it's chilly in my kitchen, so he doesn't go overboard left on the counter.