In general, no. There is evidence that after extremely vigorous workouts there may be a small increase of lactic acid present in breast milk. This does not harm baby in any way, though some athletes have reported that their babies seemed to dislike the taste of their milk immediately after a grueling workout. Even in these extreme situations, the lactic acid is quickly metabolized out of the milk, so any change in flavor will be short-lived. For most moderate workouts there is no change in your milk at all, and babies will nurse happily even immediately after a workout.
Remember: It is important that you have good support for your breasts during exercise. Lactating breasts are naturally heavier, and you can experience discomfort if you do not have a very supportive bra. Nursing mothers shouldn't wear a snug sports-bra most of the time, but it is important to wear one during your workout.