Maybe you don't pin a number on every weekend and head to the races, but when you do, you want to be ready for any course you might come across. A ripping fast city center crit race, a rolling road race, or a 1-day classic that features pavement so bad it's funny to even consider the road's paved - you want a bike that can handle it all.
You're looking for a bike that can race on the weekends and still log all your training miles without a worry. You want a bike that is robust and can take the day-to-day use without turning into a rattle-trap that drives you and anybody riding near you crazy.
Ride quality matters too. Maybe you live where you curse the road commissioner for letting your roads turn to dust, or perhaps you like the idea of hopping on a dirt road or trail to help make completely new rides. You want a bike with a comfortable ride and the ability to run the tires you want (just don't call it a gravel bike).