If you're trying to get faster, all your rides should fall in either the really fast or really slow categories.
These aren't necessarily fast overall, but they have intervals that push your body to the limit. In order to get the best results, you should be well rested and well fueled in order to get the highest possible end of your ability. If you're a bit tired all the time you can't hit the height of your potential, so your body won't adapt to that peak. So you'll just be a little bit faster and just kind of tired.
These intervals will depend on what you're training for and what your weaknesses are. If you're training for a time trial you need to do long intervals. If you're training for crits you need to do fast intown group rides to adapt to the surges of speed through corners, as well as train your sprint. If you're training for long road races you need to do hill work and fast, long group rides.
An excellent place to do specific training is on the trainer, where you can avoid stopping (lights, cars) and measure consistent output. If you don't have a power meter (like me) you need to keep track of the gearing you're using for each workout in order to increase resistance as you go.
If you're not going fast, you should be going slow. Like, really slow. For a lot of racers, this can be tough. You have to fight your natural inclination for speed (though I've met racers who like going slow, unless there's a finish line). Slow rides are an excellent way to have active recovery, develop general health, and lose weight (you burn more fat riding slow than sugars; you burn more sugars when going fast than fats).
Riding tempo can be good, but if you only ride tempo then you'll never build up your top end speed and you'll end up just being tired (if you put in plenty of saddle-time). You'll be good pack fodder, or you could be excellent help for your teammates when a break needs to be chased down.
Yeah I know ... "Riding for fun, what is that? A new kind of interval or something?" But sometimes it is good to go out there and just do a little joy riding. Once or twice a year.
Basically, I think it's just important to ride purposefully, and try to make sure you're rested enough to do super hard efforts when you're building up for the race season. The worst thing you could do is do a bunch of unstructured group rides, get exhausted and never really hit your potential.