A few "trivial" things to try when you do not succeed in adding TLS support to your qmail.
EHLO. This should show STARTTLS, otherwise the patch was not compiled in.
telnet localhost 25. Issue
EHLO. This should again show STARTTLS. Issue STARTTLS. The server should return
220 ready for tls.
openssl s_client -starttls smtp -crlf -connect localhost:25. There you should have a "normal" SMTP session.
ldd /var/qmail/bin/qmail-remotewill show whether your qmail-remote was linked to libssl and libcrypto.
/var/qmail/bin/qmail-remote host sender recipient. As host, use a host you know implements STARTTLS. (when in doubt, use s_client to check the remote host, as described above).