Month: March 2012

New SAS BI Book: a good read for Platform Administrators too

I’ve just added a link to the new Building Business Intelligence Using SAS: Content Development Examples book by Tricia Aanderud and Angela Hall to my reading list of information resources I find useful for SAS® platform administration. Their book is not just for BI content developers. I think it’s a good read for SAS platform administrators too. It helps us to understand what the BI people do and is also packed with information on what we can do to support them.

A large proportion of the SAS platform administrators I speak to in Australia come from a systems administration background rather than a SAS background (my wildly inaccurate guess would be about 50%). Those new SAS admins have great knowledge of sys admin but might have only just been introduced to the SAS platform. They’ve heard lots of terms like metadata, libraries, stored processes, cubes, information maps, prompts, web reports, dashboards, portlets and portal pages. Their customers, the SAS users, talk to them about applications like SAS Enterprise Guide, SAS Information Map Studio, SAS Information Delivery Portal, SAS Web Report Studio, SAS OLAP Cube Studio, SAS Add-In for Microsoft Office and SAS BI Dashboard. You can imagine it’s quite daunting when you’re starting out as a SAS admin! When I talk to them, the first things they want to understand are what do all these terms mean, what are all of these applications used for, and what do I need to know about them to be able to support my users. I think this new book is another great resource to help answer those questions and get a better understand of the needs of the SAS user community.

The book consists of many short practical examples of typical activities BI content developers need to do. I like that there are lots of screenshots showing you exactly where you need to be. When the BI developer needs administrator assistance, the instructions are provided right there in context. If the BI developer is also an administrator they can keep going without stopping. If they need an administrator to help them, that’s where they can phone, or send them an email, outlining what’s required based on what they see in the book. If you’re reading the book as an administrator you can see the administration task in context to know why someone might be asking you to perform that task for them.

Some of the admin related things you’ll see in the book include: a high level overview of the purpose of the various SAS client applications; registering library and table metadata and updating table metadata; granting application capabilities through roles; pre-assigning libraries & controlling library assignment mode; turning on ARM logging to tune OLAP cubes; OLAP member-level security; identity based filters for BI row level security; scheduling reports; and configuring portal group content administrators.

You can order the book from SAS Press and it’s also available from Amazon. I understand that there’s an eBook edition coming soon too.

I’m looking forward to meeting the authors in person at the SAS Global Forum next month. We’ll have our copy of the book with us to see if we can get it signed :)

I’d also recommend following Angela and Tricia’s blogs: Real BI for Real Users and Business Intelligence Notes. They have a new book on the way too: The 50 Keys to Learning SAS Stored Processes.

If you know of any other books that you think would be a good read for a SAS platform administrator please let me know.

SAS & JBoss: Too Many Open Files

I’ve been seeing some ‘Too many open files‘ exceptions in the SAS® mid-tier JBoss logs on my Ubuntu Linux server. I was surprised about this because I remember during installation I had followed the guidance in the SAS documentation and increased the nofile limit. It turns out that verifying the ulimit from a normal console/ssh login was not sufficient, I should have also verified it from an su based login too.

These were the sorts of messages I was seeing in the JBoss logs:

2012-03-11 09:22:38,463 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed
java.net.SocketException: Too many open files
...
2012-03-11 08:57:19,454 ERROR [org.apache.catalina.core.StandardContext] Error reading tld listeners java.io.FileNotFoundException: /opt/jboss-4.2.3.GA/server/SASServer3/work/jboss.web/localhost/SASBIDashboard/tldCache.ser (Too many open files)
...

There are some Pre-Installation Steps for JBoss for both SAS 9.3 and SAS 9.2 that should be followed during installation to avoid these errors. These were the instructions I had followed, but as you’ll see in a moment, it wasn’t quite enough for this (unsupported) Ubuntu installation.

The instructions specify to edit the /etc/security/limits.conf file directly. Rather than editing this main config file, where the settings might get forgotten or lost during an upgrade, I placed the settings I required for my SAS installation in their own dedicated config file: /etc/security/limits.d/sas.conf

# Increase the open file descriptors limit from the default of 1024 to 30720 for JBoss running web apps for SAS 9.2/9.3
* - nofile 30720

I knew that this had taken effect because I logged in, via ssh, to verify it as the sas user (I run JBoss as the sas user). Checking the ulimit I saw the following:

sas@server:~$ ulimit -Hn;ulimit -Sn
30720
30720

With nofile at 30270, how was it I was still getting ‘Too many open files‘ errors? After a quick session on Google I found a blog post suggesting the increased limits will only apply if the pam_limits PAM module is enabled.

Checking the /etc/pam.d/login file I could see the pam_limits line was already present and uncommented:
...
session required pam_limits.so
...

This made sense since the console/ssh login showed the expected numbers.

Google also led me to a stackoverflow question (how do i set hard and soft file limits for a non-root user at boot?). The answer provided there indicated that, for su commands, you also need to verify the pam_limits module is enabled in an additional su specific PAM config file, which on my machine was /etc/pam.d/su. My JBoss init script runs as root during system startup but uses su to run JBoss as the sas user. Looking in /etc/pam.d/su I could see that the pam_limits line was commented so perhaps that was the issue.

Before making the necessary changes, I verified the nofile ulimit for the sas user by running su as root:

root@server:~# su sas --login --command 'ulimit -Hn;ulimit -Sn'
1024
1024

Aha! It had the 1024 default rather then the increased value. It looked like this was indeed the problem. I uncommented the pam_limits line in /etc/pam.d/su and repeated the test:

root@server:~# su sas --login --command 'ulimit -Hn;ulimit -Sn'
30720
30720

It now shows the increased value as expected, so it looks like the problem’s fixed. I restarted JBoss and haven’t seen any ‘Too many open files‘ errors since.

SAS & IWA: Verifying Trusted for Delegation Status

Update 26Sep2018: This post is now several years old and naturally technology and security have progressed in that time. For more up to date information regarding delegation and, in particular, the requirement for constrained delegation when working with Windows Defender Credential Guard in Windows 10 and Windows Server 2016, please see Stuart Rogers’ very useful SAS Global Forum 2018 Paper: SAS 9.4 on Microsoft Windows: Unleashing Kerberos on Apache Hadoop.

I mentioned in a previous post that host machines need to be Trusted for Delegation when a SAS® software component, such as a SAS Workspace Server, needs to make outgoing connections to secondary servers when the initial incoming connection was made using Integrated Windows Authentication (IWA).

When a server needs to be Trusted for Delegation, it takes a domain administrator to change the machine account in Active Directory. I rarely have domain admin privileges when working at customer sites so I usually can’t do this for myself. :( This post describes the method I use, as a lowly domain user, to verify that a Windows server has been configured in Active Directory as Trusted for Delegation.

The screenshot below shows an example of what the domain admin might see in the Properties dialog Delegation tab for the machine account in Active Directory (via the Active Directory Users and Computers tool (under Start > All Programs > Administrative Tools).

This machine account (P1001) is not yet trusted for delegation. The domain admin would click the radio button for “Trust this computer for delegation to any service (Kerberos only)“.

Once the domain admin has advised that the change has been applied, we can test it out from the SAS platform. What happens if the test still fails? I like to double check that the server is definitely trusted for delegation before I move on to checking other things. Everyone makes mistakes from time to time, even domain admins; maybe the wrong machine account was modified (it does happen). So to avoid wasting time later on, I like to verify this pre-requisite before moving on. I could ask the domain admin to email me a screenshot of the dialog to confirm, but they’re likely very busy people, so why not do it myself? I often don’t have access to the Active Directory Users and Computers tool so I have to find another way to verify trusted for delegation. This is where the very useful AdExplorer utility helps. It’s one of the SysInternals tools available for download from Microsoft. As the name suggests it provides an explorer interface to Active Directory so you can browse the objects and attributes.

Here’s an AdExplorer screenshot showing the same machine account (P1001) from the dialog shown earlier.

I have selected the userAccessControl attribute and can see it has the value 4128. This is not good; I’ll explain why in a moment :) Essentially it means that the machine is not trusted for delegation. What I would rather see, is the next screenshot where it has the value 528416 meaning it is trusted for delegation.

So where do these magic numbers come from, and how do we know that 528416 is trusted and 4128 is not? The userAccessControl value is a bitmap value (or bit array). There are a number of possible flags that can be set in this value which are documented in the Microsoft resource How to use the UserAccountControl flags to manipulate user account properties. The main flag of interest here is TRUSTED_FOR_DELEGATION with value 0x80000 (hex) or 524288 (decimal) which is listed in the document as:

TRUSTED_FOR_DELEGATION – When this flag is set, the service account (the user or computer account) under which a service runs is trusted for Kerberos delegation. Any such service can impersonate a client requesting the service. To enable a service for Kerberos delegation, you must set this flag on the userAccountControl property of the service account.

The (trusted) decimal value 528416 (hex 0x81020) we saw above consists of TRUSTED_FOR_DELEGATION (decimal 524288 hex 0x80000) + WORKSTATION_TRUST_ACCOUNT (decimal 4096 hex 0x1000) + PASSWD_NOTREQD (decimal 32 hex 0x0020). The (untrusted) decimal value 4128 (hex 0x1020) we saw earlier only consists of WORKSTATION_TRUST_ACCOUNT (decimal hex 0x1000) + PASSWD_NOTREQD (decimal 32 hex 0x0020). It’s missing the TRUSTED_FOR_DELEGATION value. You might see other values in your environment, including other flag values, but the important thing to check is that it includes the TRUSTED_FOR_DELEGATION value.

If you know of any other ways to verify a server’s trusted for delegation status (as a normal domain user) please let me know by leaving a comment.

For more posts in this series have a look at the IWA tag.