I’d bring it to room temp for its ultimate fluffiness, although it is also fabulous cold.
If it’s 2 layers, I like to give 6 hours, and lean toward 7 more than 5, if you have to go one way or the other. It’s amazing how long a cake stays cold in the middle. If you take it out too early and it’s still cold, as soon as everyone’s done with their cold piece, the cake miraculously comes to it’s room temp perfection of moistness, which will be known only to the late comers (and seconds-ers).
If it’s 1 layer, you can get away with 3.
It can still sit out all day after that, even with pastry filling. At least I’ve never had problems or any sick people!! And, don’t forget, during it’s coming to room temp, it’s basically cold, so that time doesn’t really count.
I sometimes get paranoid and lean toward less, rather than more, time, because I worry about it keeping, and I always regret it. The cakes always keep great, but it’s always a drag when they are not total room temperature when cutting. Also, they take longer to come to room temp if they’re in a covered carrier, so take the lid off. I usually let the lid be just sort of askew for an hour or so so it’s not shocked by the change, then I take the lid off.
Now I get up in the middle of the night if need be. It’s really worth it!