When setting up a public folder don’t forget to allow contributor access for anonymous users so that users external to your company will be able to send emails to the public folder.
If you track the undelivered messages you will see this error if anonymous users do not have contributor access: 550 5.2.0 STOREDRV.Deliver. To fix this you can use the following Exchange Shell command:
Add-PublicFolderClientPermission -Identity \"public folder name" -User anonymous -AccessRights contributor