Add checkbox "Ignore SSL certificate errors" in SMTP Settings which reflects in option for nodemailer in server/core/mail.js:16:

tls: {rejectUnauthorized: WIKI.config.mail.ignoreSSLerrors},