Nope. With the help of a quality electric breast pump, you can leave bottles of your breast milk for baby. (You can purchase a pump online or at maternity and baby stores, or rent one from the hospital.) If possible, try to start pumping a few weeks before heading back to work to start building up your "freezer stash" of breast milk. A few bottles/bags of milk in the freezer will help take the pressure off as you master the art of pumping on coffee breaks. Talk with your employer (before you go back to work) about setting up a private place for you to pump for baby, and try to express milk as often as you would expect baby to eat.
Your breast milk is baby's best defense against illness and offers him the best nutrition available — sticking with it is totally worth it.