@Guest: As written above, the correct solution is to set SessionOptions.TlsHostCertificateFingerprint:
https://winscp.net/eng/docs/library_sessionoptions#tlshostcertificatefingerprint

I was able to make my code run by settings

Installing the latest version (5.1) the problem does not recur.
Thank you.

Hi,
I've the same problem.
I set the SessionOptions.SslHostCertificateFingerprint with the fingerprint of the untrusted certificate, but when I try to open the session, the application freeze and I've to kill the app.

Thank you.

@MrRanger: Fingerprint of the untrusted certificate.

What value would you set it to in order to accept the untrusted certificate

Make sure you set SessionOptions.SslCertificate (renamed to TlsHostCertificateFingerprint in the later versions) to accept the untrusted certificate:
https://winscp.net/eng/docs/library_sessionoptions

Even I am facing the same problem. Please let me know whether you resolved this issue.

HI,
I installed 5.0 beta WinSCP and reference Winscp.dll use C# code, but I can't connect to server successfully.
Throws exception -->

WinSCP.sessionRemoteException:peer certificate rejected .

What's problem, who can help me?