I recently encountered this error on a Windows 7 system, after a vendor-provided update to some third-party software. RDP stopped working after the update, and the problem turned out to be layered. The TL;DR version is this: FIPS was enabled but the Cipher suite settings did not have any FIPS-compliant algorithms defined.
First, the simple part: the vendor disabled RDP via GPO. That's an easy thing to undo and I won't detail it here.
But even after enabling RDP via GPO, it still wouldn't work. The RDP client wouldn't connect. When I connected with openssl s_client I got this:
# openssl s_client -connect system:3389
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 247 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
Looking in the Windows Event Log, I found an error from Schannel with Event ID 36871, and the error text "a fatal error occurred while creating an SSL server credential. The internal error state is 10011."
I eventually narrowed this down to the fact that the vendor had turned on FIPS-compliant algorithms. However, on this system, I had set the allowed cipher suites to "modern" algorithms like ECDHE-RSA-AES256-SHA384, which is not FIPS-compliant but is more secure; i.e. FIPS-compliant algorithms are old and less secure.
So the fix was to disable FIPS again.
