Is it true that pacifiers won't affect my sons teeth until the permanent ones come in?

My son will be 3 in december he was off the paci but now we have a 4 month old and he steals hers every chance he gets. Someone told me that it would not affect his teeth yet. He can talk really well and its not causing him any speech problems.