SSH configured without password, but Open session in PuTTY requires authentication

When connecting from Windows to Linux, I use the command ssh pi@192.168.0.2 and it connects to the Linux env without asking the password, because I have configured both the systems to work without password authentication.
Refer:
https://www.howtogeek.com/66776/how-to-remotely-copy-files-over-ssh-without-entering-your-password/

But when clicking the "Open session in PuTTY" in WinSCP it ask for authentication, even though both devices are configured to not use password auth.

What do you mean by "without password authentication"? Do you you mean that you are using public key authentication?

So how do you authenticate in WinSCP in the first place?

Note that PuTTY and OpenSSH ssh configuration are not connected anyhow. That you have a public key authentication setup for ssh does not make it magically work in WinSCP or PuTTY.
See https://winscp.net/eng/docs/guide_public_key#winscp