Batch won't stop for trouble. Addresses in the batch may fail but Swift just notes these failures down and carries on.
Check the contents of $swift->getFailedRecipients() *after* sending. If it's empty the SMTP server accepted all addresses for delivery. If it's got addresses in it those addresses failed.
Swift can only see what the SMTP server tells it however. Just because the SMTP server accepts the message for delivery does not guarantee it will reach the intended recipient.