Thanks for your report.
I have sent you an email with a debug version of WinSCP to address you have used to register on this forum.

I transferred 2 small files to Oracle VM VirtualBox.
Then I double clicked on a tgz file of size 453 MB. This caused the out of memory error.
When I tried to reproduce the error, a dialog came up and complained about opening a too big file.

WinSCP 5.6.4

Error message:
Out of memory

Stack trace:
[0072CFC2] System::Sysutils::WideStrAlloc
[0072CFC2] System::Sysutils::WideStrAlloc
[005BC0D5] Vcl::Controls::TWinControl::WndProc
[005BB71C] Vcl::Controls::TWinControl::MainWndProc
[006DFFF8] System::Classes::_18201
[76B162F7] USER32.dll
[76B16D35] USER32.dll
[76B16DE3] USER32.dll
[76B16E41] USER32.dll
[77C70107] ntdll.dll.KiUserCallbackDispatcher
[005790FB] Vcl::Forms::TCustomForm::DestroyWindowHandle
[003A5D79] Vcl::Forms::TForm::~TForm
[00AE31C9] TEditorForm::~TEditorForm
[007502D0] System::TObject::Free
[005BC0D5] Vcl::Controls::TWinControl::WndProc
[00576E78] Vcl::Forms::TCustomForm::WndProc
[00B27051] Vclcommon::C2442_0
[005BB71C] Vcl::Controls::TWinControl::MainWndProc
[006DFFF8] System::Classes::_18201
[76B162F7] USER32.dll
[76B16D35] USER32.dll
[76B177BF] USER32.dll
[76B17885] USER32.dll.DispatchMessageW