In a previous post I’ve described a method for configuring Active Directory Authentication for SAS® on Linux (with realmd). One of the packages that’s installed is oddjob-mkhomedir. This package normally handles any requirement for auto-creating home directories for those AD users on Linux. Unfortunately it doesn’t seem to get used by the SAS Object Spawner. I ran into this issue again today when logging into SAS Studio 4.2 as an AD user on the SAS Viya™ 3.2 platform. I wasn’t able to login because the AD user’s Linux home directory didn’t exist and hadn’t been auto created. After manually creating the home directory the login succeeded. I would rather get auto-creation working so I wouldn’t need to manually create home directories for each SAS user that was likely to use SAS Studio. Thankfully I was able to find a solution that I’ll describe in this post. Continue reading “Auto Creation of Linux Home Directories for SAS Users”
I’ve been seeing some ‘Too many open files‘ exceptions in the SAS® mid-tier JBoss logs on my Ubuntu Linux server. I was surprised about this because I remember during installation I had followed the guidance in the SAS documentation and increased the nofile limit. It turns out that verifying the ulimit from a normal console/ssh login was not sufficient, I should have also verified it from an su based login too.
These were the sorts of messages I was seeing in the JBoss logs:
2012-03-11 09:22:38,463 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed
java.net.SocketException: Too many open files
2012-03-11 08:57:19,454 ERROR [org.apache.catalina.core.StandardContext] Error reading tld listeners java.io.FileNotFoundException: /opt/jboss-4.2.3.GA/server/SASServer3/work/jboss.web/localhost/SASBIDashboard/tldCache.ser (Too many open files)
There are some Pre-Installation Steps for JBoss for both SAS 9.3 and SAS 9.2 that should be followed during installation to avoid these errors. These were the instructions I had followed, but as you’ll see in a moment, it wasn’t quite enough for this (unsupported) Ubuntu installation.
The instructions specify to edit the /etc/security/limits.conf file directly. Rather than editing this main config file, where the settings might get forgotten or lost during an upgrade, I placed the settings I required for my SAS installation in their own dedicated config file: /etc/security/limits.d/sas.conf
# Increase the open file descriptors limit from the default of 1024 to 30720 for JBoss running web apps for SAS 9.2/9.3
* - nofile 30720
I knew that this had taken effect because I logged in, via ssh, to verify it as the sas user (I run JBoss as the sas user). Checking the ulimit I saw the following:
sas@server:~$ ulimit -Hn;ulimit -Sn
With nofile at 30270, how was it I was still getting ‘Too many open files‘ errors? After a quick session on Google I found a blog post suggesting the increased limits will only apply if the pam_limits PAM module is enabled.
Checking the /etc/pam.d/login file I could see the pam_limits line was already present and uncommented:
session required pam_limits.so
This made sense since the console/ssh login showed the expected numbers.
Google also led me to a stackoverflow question (how do i set hard and soft file limits for a non-root user at boot?). The answer provided there indicated that, for su commands, you also need to verify the pam_limits module is enabled in an additional su specific PAM config file, which on my machine was /etc/pam.d/su. My JBoss init script runs as root during system startup but uses su to run JBoss as the sas user. Looking in /etc/pam.d/su I could see that the pam_limits line was commented so perhaps that was the issue.
Before making the necessary changes, I verified the nofile ulimit for the sas user by running su as root:
root@server:~# su sas --login --command 'ulimit -Hn;ulimit -Sn'
Aha! It had the 1024 default rather then the increased value. It looked like this was indeed the problem. I uncommented the pam_limits line in /etc/pam.d/su and repeated the test:
root@server:~# su sas --login --command 'ulimit -Hn;ulimit -Sn'
It now shows the increased value as expected, so it looks like the problem’s fixed. I restarted JBoss and haven’t seen any ‘Too many open files‘ errors since.
These are some of my favourite papers from SAS Global Forum 2011. As a platform administrator and metadata fan I am obviously biased to a specific subset of papers. I’m sure there were many other great papers at the conference, but these are the ones that I liked the most based on my own interests. 🙂
- Best Practice Implementation of SAS® Metadata Security at Customer Sites in Denmark
Cecily Hoffritz & Johannes Jørgensen
SAS Global Forum 2011 Paper 376-2011
- PDF available from http://support.sas.com/resources/papers/proceedings11/376-2011.pdf
- Single Sign-On Configuration and Troubleshooting for SAS® 9.2 Enterprise BI Web Applications
Stuart J Rogers & Heesun Park
SAS Global Forum 2011 Paper 365-2011
- PDF available from http://support.sas.com/resources/papers/proceedings11/365-2011.pdf
- Using SAS® on UNIX with Multiple Active Directories as Authentication Providers
SAS Global Forum 2011 Paper 369-2011
- PDF available from http://support.sas.com/resources/papers/proceedings11/369-2011.pdf
- Understanding the Anatomy of a SAS® Deployment: What’s in My Server Soup?
Mark Schneider, Donna Bennett, & Connie Robison
SAS Global Forum 2011 Paper 363-2011
- PDF available from http://support.sas.com/resources/papers/proceedings11/363-2011.pdf
- Configuration and Tuning Guidelines for SAS®9 in Microsoft Windows Server 2008
SAS Global Forum 2011 Paper 370-2011
- PDF available from http://support.sas.com/resources/papers/proceedings11/370-2011.pdf
- Considerations for Implementing a Highly Available or Disaster Recovery Environment
Diane Hatcher & Jochen Kirsten
SAS Global Forum 2011 Paper 358-2011
- PDF available from http://support.sas.com/resources/papers/proceedings11/358-2011.pdf
I’ll definitely be recommending these papers in my SAS platform admin consulting and training work, and have done so a number of times already. They offer a great supplement to the standard SAS documentation and provide lots of additional background info for some of the common types of platform admin related questions I hear like:
- “How can I find out more about SAS architecture, what all the components are, and how they fit together?”
- “What’s involved in setting up single sign-on for SAS installations?”
- “What do we need to know about optimizing the performance of SAS software?”
- “How do we secure our SAS content, what are the recommendations, and what should we watch out for?”
- “What happens if we get a hardware failure on our SAS Metadata Server? What things do we need to consider in disaster recovery planning?”
I’ve also added these papers to my reading list so I can find them easily when I need to point someone at them.
Thanks to all the authors for taking the time to prepare, present and publish them to share with the SAS community.
Whilst troubleshooting why LDAP based users couldn’t authenticate against a SAS Metadata Server configured for host authentication, I found this handy resource: SAS Usage Note 39891: Using PROC PERMTEST to diagnose UNIX host authentication issues in SAS® 9.2. The server in question already had correctly functioning LDAP host authentication via PAM, it was just that the SAS Metadata Server wasn’t able to authenticate the LDAP users. The usage note helped me fix the problem by leading me to the SASFoundation/SAS9.2/utilities/bin/sasauth.conf file where I found it had methods=pw instead of methods=pam (must have chosen the wrong options at install time). I actually didn’t need to run PROC PERMTEST this time but the usage note contains clear instructions on how to when required. Interestingly PROC PERMTEST doesn’t seem to appear in the Base SAS® 9.2 Procedures Guide.