No, you shouldn't worry. Drinking alcohol during pregnancy can lead to fetal alcohol syndrome, which is associated with abnormal facial features, growth problems and developmental and learning disabilities, among other things. Unfortunately, there is no "safe" amount of alcohol that is okay to consume during pregnancy. So, you are right to avoid it since conceiving. But, if the only alcohol you had was on the night of conception, you shouldn't worry about damage to the baby.