A small percentage of babies are sensitive to a protein found in cow's milk. If your baby has this sensitivity, then yes — if you drink or eat cow's milk products, it could affect your baby. If baby has suspicious allergy-like symptoms (like excessive gas, spitting up, fussing, rash, irritability, intestinal upsets, or green stools with mucus or blood) and you suspect cow's milk is the cause, you can try eliminating dairy products from your diet for a couple of weeks to see if you see improvement. If you don't see a change in about two weeks (it can take that long for the proteins to leave your system), baby probably isn't sensitive. If you do notice improvement, congrats — you've probably found the culprit.
Many babies outgrow food sensitivities as they grow. If you think baby has any sensitivity to cow's milk or another ingredient in your diet, work with your pediatrician on a game plan for reintroducing the foods as baby gets older.