I recently installed a local copy of SAS Viya 4 (2021.1.2) in our Kubernetes lab environment and was trying to configure it to work with a test AD server, Windows 2012 R2. The AD server had been configured to only allow TLS (LDAPS) connections and so after the initial installation of SAS Viya I configured the SAS Identities service to use LDAPS, as explained in the SAS documentation.
Unfortunately it didn’t work at first. I was getting errors that turned out to be a TLS cipher mismatch between the client (SAS Identities service) and the server (Windows 2012 R2 AD). I was able to fix it using two methods:
- Re-enabling LDAP connections to AD and switching back from LDAPS to LDAP connections for the SAS Identities service (not preferred)
- Tweaking the configuration of the SAS Identities service to support a cipher that worked with Windows 2012 R2 AD (preferred)
Of course I could have also upgraded AD from Windows 2012 R2 to a more recent version, but that is a project for another day as I still want to continue working with the older version for the time being.
For future reference, and in case it helps anybody else, here are my notes from troubleshooting and what I did to solve it.
I tailed the sas-identities pod log using the following command (I have SAS Viya 4 deployed in the sasviyadev namespace):
kubectl logs -f -n sasviyadev $(kubectl -n sasviyadev get pods | grep sas-identities | cut -f1 -d' ')… and saw the following:
{"version":1,"timeStamp":"2021-07-18T04:40:46.308Z","level":"error","source":"sas-identities","message":"[CACHE_REFRESH_ERROR] An error occurred while attempting to refresh the identity cache.\norg.springframework.dao.DataAccessResourceFailureException: Failed to borrow DirContext from pool.; nested exception is org.springframework.ldap.CommunicationException: simple bind failed: ad01.corp.example.com:636; nested exception is javax.naming.CommunicationException: simple bind failed: ad01.corp.example.com:636 [Root exception is java.net.SocketException: Connection or outbound has closed]\nCaused by: org.springframework.ldap.CommunicationException: simple bind failed: ad01.corp.example.com:636; nested exception is javax.naming.CommunicationException: simple bind failed: ad01.corp.example.com:636 [Root exception is java.net.SocketException: Connection or outbound has closed]\nCaused by: javax.naming.CommunicationException: simple bind failed: ad01.corp.example.com:636 [Root exception is java.net.SocketException: Connection or outbound has closed]\nCaused by: java.net.SocketException: Connection or outbound has closed","properties":{"logger":"com.sas.identities.provider.cache.IdentityCacheLoader","thread":"scheduler-1"},"messageKey":"com.sas.identities.LogMessages.CACHE_REFRESH_ERROR"}
{"version":1,"timeStamp":"2021-07-18T04:44:51.396Z","level":"warn","source":"sas-identities","message":"[IDENTITY_FETCH_LDAP_ERROR] Error occurred while fetching identity: Failed to borrow DirContext from pool.; nested exception is org.springframework.ldap.CommunicationException: simple bind failed: ad01.corp.example.com:636; nested exception is javax.naming.CommunicationException: simple bind failed: ad01.corp.example.com:636 [Root exception is javax.net.ssl.SSLException: Connection reset]","properties":{"logger":"com.sas.identities.provider.ldap.LdapIdentityQueryRepository","thread":"https-jsse-nio-0.0.0.0-8080-exec-5","traceId":"763c3e7ca7e75ea2","spanId":"763c3e7ca7e75ea2","spanExportable":"true","username":"sas.searchIndex"},"messageKey":"com.sas.identities.LogMessages.IDENTITY_FETCH_LDAP_ERROR","messageParameters":{"0":"Failed to borrow DirContext from pool.; nested exception is org.springframework.ldap.CommunicationException: simple bind failed: ad01.corp.example.com:636; nested exception is javax.naming.CommunicationException: simple bind failed: ad01.corp.example.com:636 [Root exception is javax.net.ssl.SSLException: Connection reset]"}}The root exception was “javax.net.ssl.SSLException: Connection reset” which I had seen could occur when a TLS handshake fails.
The Windows 2012 R2 Event log was showing schannel errors (id 36874 and 36888) relating to a TLS 1.2 cipher mismatch:
The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 1205.
The TLS 1.2 RFC5246 at https://datatracker.ietf.org/doc/html/rfc5246#appendix-A.3 lists handshake_failure(40)
It was clear the problem was a cipher mismatch but I didn’t know which incompatible cipher was being used.
A Microsoft resource at https://docs.microsoft.com/en-us/troubleshoot/iis/enable-schannel-event-logging showed how to turn up logging for schannel to get more info on ciphers in use:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL]
"EventLogging"=dword:00000007I set the level to 7, rebooted the Windows server, and tried SAS Identities with LDAPS again. The event log now showed:
Event 36880
Protocol: TLS 1.2
CiperSuite: 0xC028
Exchange strength: 256A quick search found that this cipher is TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028).
Next I wanted to see what ciphers were used by the SAS identities service and whether I could tweak them so did a little more digging.
I got a shell into the SAS identities container:
kubectl -n sasviyadev get pods | grep sas-identities
sas-identities-f97689bb9-x2ptl
kubectl exec -i -t -n sasviyadev sas-identities-f97689bb9-x2ptl -c sas-identities "--" bash… then used “ps -e” to find the sas-identities java process id (57 in this case).
PID TTY TIME CMD
1 ? 00:00:00 tini
7 ? 00:00:00 sas-identities
57 ? 00:02:21 java
173 pts/0 00:00:00 sh
180 pts/0 00:00:00 sh
181 pts/0 00:00:00 bash
689 pts/1 00:00:00 bash
701 pts/1 00:00:00 psBy running “cat /proc/57/cmdline” I could see the command line contained TLS version and cipher settings:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el8_4.x86_64/jre/bin/java
...
-Djdk.tls.client.cipherSuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_CCM_SHA256,TLS_AES_128_CCM_8_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
-Djdk.tls.client.protocols=TLSv1.3,TLSv1.2
-Djdk.tls.server.cipherSuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_CCM_SHA256,TLS_AES_128_CCM_8_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
-Djdk.tls.server.protocols=TLSv1.3,TLSv1.2
...I wondered if I could tweak or override these options. I had seen that the SAS Identities service jvm configuration in SAS Environment Manager already had a JDK memory setting (java_option_xmx) -Xmx384m so I though I might try to add one to override that jdk.tls.client.cipherSuites value and append the TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 cipher (but being aware that this is considered a weak cipher these days). The available JDK cipher names can be see at https://www.java.com/en/configure_crypto.html
Whilst logged into SAS Environment Manager, I navigated to Configuration > Identities service > jvm and used Edit > Add property to add the following:
Name: java_option_cipher_suites
Value: -Djdk.tls.client.cipherSuites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_CCM_SHA256,TLS_AES_128_CCM_8_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384… which was the original value from above but with TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 added to the end.
I restarted (deleted) the sas-identities and sas-logon-app pods and repeating my test using LDAPS and found it now worked. Looking at the sas-identities container command again I could see my setting had actually been appended to override the earlier one. I may not have actually needed to restart the pods but I wanted to be sure my changes had been applied.
I’m not yet sure if this is the best way to get this combination working but it did work for me!
If you know of a better method then I’d be very keen to hear about it. If you have any comments, tips, or suggestions then please leave a comment below or contact me directly.