Paul Homes – Page 22 – platformadmin.com

SAS & JBoss: Too Many Open Files

I’ve been seeing some ‘Too many open files‘ exceptions in the SAS® mid-tier JBoss logs on my Ubuntu Linux server. I was surprised about this because I remember during installation I had followed the guidance in the SAS documentation and increased the nofile limit. It turns out that verifying the ulimit from a normal console/ssh login was not sufficient, I should have also verified it from an su based login too.

These were the sorts of messages I was seeing in the JBoss logs:

2012-03-11 09:22:38,463 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed java.net.SocketException: Too many open files ... 2012-03-11 08:57:19,454 ERROR [org.apache.catalina.core.StandardContext] Error reading tld listeners java.io.FileNotFoundException: /opt/jboss-4.2.3.GA/server/SASServer3/work/jboss.web/localhost/SASBIDashboard/tldCache.ser (Too many open files) ...

There are some Pre-Installation Steps for JBoss for both SAS 9.3 and SAS 9.2 that should be followed during installation to avoid these errors. These were the instructions I had followed, but as you’ll see in a moment, it wasn’t quite enough for this (unsupported) Ubuntu installation.

The instructions specify to edit the /etc/security/limits.conf file directly. Rather than editing this main config file, where the settings might get forgotten or lost during an upgrade, I placed the settings I required for my SAS installation in their own dedicated config file: /etc/security/limits.d/sas.conf

# Increase the open file descriptors limit from the default of 1024 to 30720 for JBoss running web apps for SAS 9.2/9.3 * - nofile 30720

I knew that this had taken effect because I logged in, via ssh, to verify it as the sas user (I run JBoss as the sas user). Checking the ulimit I saw the following:

sas@server:~$ ulimit -Hn;ulimit -Sn 30720 30720

With nofile at 30270, how was it I was still getting ‘Too many open files‘ errors? After a quick session on Google I found a blog post suggesting the increased limits will only apply if the pam_limits PAM module is enabled.

Checking the /etc/pam.d/login file I could see the pam_limits line was already present and uncommented:
... session required pam_limits.so ...

This made sense since the console/ssh login showed the expected numbers.

Google also led me to a stackoverflow question (how do i set hard and soft file limits for a non-root user at boot?). The answer provided there indicated that, for su commands, you also need to verify the pam_limits module is enabled in an additional su specific PAM config file, which on my machine was /etc/pam.d/su. My JBoss init script runs as root during system startup but uses su to run JBoss as the sas user. Looking in /etc/pam.d/su I could see that the pam_limits line was commented so perhaps that was the issue.

Before making the necessary changes, I verified the nofile ulimit for the sas user by running su as root:

root@server:~# su sas --login --command 'ulimit -Hn;ulimit -Sn' 1024 1024

Aha! It had the 1024 default rather then the increased value. It looked like this was indeed the problem. I uncommented the pam_limits line in /etc/pam.d/su and repeated the test:

root@server:~# su sas --login --command 'ulimit -Hn;ulimit -Sn' 30720 30720

It now shows the increased value as expected, so it looks like the problem’s fixed. I restarted JBoss and haven’t seen any ‘Too many open files‘ errors since.

SAS & IWA: Verifying Trusted for Delegation Status

Update 26Sep2018: This post is now several years old and naturally technology and security have progressed in that time. For more up to date information regarding delegation and, in particular, the requirement for constrained delegation when working with Windows Defender Credential Guard in Windows 10 and Windows Server 2016, please see Stuart Rogers’ very useful SAS Global Forum 2018 Paper: SAS 9.4 on Microsoft Windows: Unleashing Kerberos on Apache Hadoop.

I mentioned in a previous post that host machines need to be Trusted for Delegation when a SAS® software component, such as a SAS Workspace Server, needs to make outgoing connections to secondary servers when the initial incoming connection was made using Integrated Windows Authentication (IWA).

When a server needs to be Trusted for Delegation, it takes a domain administrator to change the machine account in Active Directory. I rarely have domain admin privileges when working at customer sites so I usually can’t do this for myself. :( This post describes the method I use, as a lowly domain user, to verify that a Windows server has been configured in Active Directory as Trusted for Delegation.

The screenshot below shows an example of what the domain admin might see in the Properties dialog Delegation tab for the machine account in Active Directory (via the Active Directory Users and Computers tool (under Start > All Programs > Administrative Tools).

This machine account (P1001) is not yet trusted for delegation. The domain admin would click the radio button for “Trust this computer for delegation to any service (Kerberos only)“.

Once the domain admin has advised that the change has been applied, we can test it out from the SAS platform. What happens if the test still fails? I like to double check that the server is definitely trusted for delegation before I move on to checking other things. Everyone makes mistakes from time to time, even domain admins; maybe the wrong machine account was modified (it does happen). So to avoid wasting time later on, I like to verify this pre-requisite before moving on. I could ask the domain admin to email me a screenshot of the dialog to confirm, but they’re likely very busy people, so why not do it myself? I often don’t have access to the Active Directory Users and Computers tool so I have to find another way to verify trusted for delegation. This is where the very useful AdExplorer utility helps. It’s one of the SysInternals tools available for download from Microsoft. As the name suggests it provides an explorer interface to Active Directory so you can browse the objects and attributes.

Here’s an AdExplorer screenshot showing the same machine account (P1001) from the dialog shown earlier.

I have selected the userAccessControl attribute and can see it has the value 4128. This is not good; I’ll explain why in a moment :) Essentially it means that the machine is not trusted for delegation. What I would rather see, is the next screenshot where it has the value 528416 meaning it is trusted for delegation.

So where do these magic numbers come from, and how do we know that 528416 is trusted and 4128 is not? The userAccessControl value is a bitmap value (or bit array). There are a number of possible flags that can be set in this value which are documented in the Microsoft resource How to use the UserAccountControl flags to manipulate user account properties. The main flag of interest here is TRUSTED_FOR_DELEGATION with value 0x80000 (hex) or 524288 (decimal) which is listed in the document as:

TRUSTED_FOR_DELEGATION – When this flag is set, the service account (the user or computer account) under which a service runs is trusted for Kerberos delegation. Any such service can impersonate a client requesting the service. To enable a service for Kerberos delegation, you must set this flag on the userAccountControl property of the service account.

The (trusted) decimal value 528416 (hex 0x81020) we saw above consists of TRUSTED_FOR_DELEGATION (decimal 524288 hex 0x80000) + WORKSTATION_TRUST_ACCOUNT (decimal 4096 hex 0x1000) + PASSWD_NOTREQD (decimal 32 hex 0x0020). The (untrusted) decimal value 4128 (hex 0x1020) we saw earlier only consists of WORKSTATION_TRUST_ACCOUNT (decimal hex 0x1000) + PASSWD_NOTREQD (decimal 32 hex 0x0020). It’s missing the TRUSTED_FOR_DELEGATION value. You might see other values in your environment, including other flag values, but the important thing to check is that it includes the TRUSTED_FOR_DELEGATION value.

If you know of any other ways to verify a server’s trusted for delegation status (as a normal domain user) please let me know by leaving a comment.

For more posts in this series have a look at the IWA tag.

Role-Based Access to Metacoda Security Plug-ins in SAS Management Console

This post explains how to provide role-based access to Metacoda Security Plug-ins in the SAS® Management Console (versions 9.2 and 9.3).

Update 01Sep2023: The method described in this post is relevant for Metacoda Plug-ins version 5 and earlier. Metacoda Plug-ins version 6 and above, available since 2016, allow a much more flexible approach where you can control access to individual plug-ins and various other features. Metacoda customers can find more information in the documentation for the Metacoda Plug-ins Metadata Installer at https://support.metacoda.com/docs/plugins/v6.0/user-guide/tools/metadata-installer.html

Metacoda Security Plug-ins are initially only available to administrators, specifically members of the Management Console: Advanced role, of which the SAS Administrators group is a member by default. As unregistered plug-ins, Metacoda Security Plug-ins are controlled by the Access Unregistered Plug-ins capability which is only granted (by default) to the Management Console: Advanced role.

At Metacoda, we sometimes get asked about the possibility of providing wider access to Metacoda Security Plug-ins, for users that need/want to review and/or troubleshoot security metadata, but who are not members of the SAS Administrators group and/or the Management Console: Advanced role.

One possible option would be to modify the Management Console: Content Management role, of which SASUSERS is a member (by default) and grant the Access Unregistered Plug-ins capability. Whilst this works, it is not a recommended approach. It involves modifying the capability set for a SAS predefined role. The SASUSERS membership also means anyone with access to the SAS platform and SAS Management Console will get access to the plug-ins which is probably much wider access than is required.

These are the recommended steps to follow to provide limited, non-administrator, access to Metacoda Security Plug-ins in the SAS Management Console.

1. Choose/create Group(s) for Role-Based Access to Metacoda Security Plug-ins

The first step is to identify which groups of users need access to Metacoda Security Plug-ins. You might already have suitable groups you can use. If not then create a new group (or groups) and assign users (or nested groups) as appropriate. In this example we have a Metacoda Plug-ins Users group that contains a few individual users, including the SAS Demo User for testing purposes.

2. Enabled Role-Based Access to Metacoda Security Plug-ins

Whilst logged into SAS Management Console as an administrator, select Plug-in Manager from the Tools menu to access the Plug-in Manager dialog.

Locate Metacoda in the list of plug-ins, tick the check box to enable role-based access, and click the OK button to save the changes.

3. Create a New Role for Metacoda Security Plug-ins

Now that we’ve enabled role-based access to Metacoda Security Plug-ins, we can create a new custom role to provide access to the capability for the target users.

From the SAS Management Console User Manager plug-in, create a new role using the menu sequence Actions > New > Role. In the General tab, provide an appropriate name and description for your installation, as shown in the screenshot below.

4. Assign Capabilities to the New Metacoda Security Plug-ins Role

In the Capabilities tab for the new custom role, expand the Management Console application group and then the Plug-ins folder to locate the new Metacoda capability. Tick the check box to grant the Metacoda capability to this new custom role as shown here.

If necessary you could also grant additional capabilities for this role.

5. Assign Members to the New Metacoda Security Plug-ins Role

With the capability granted, all that remains is to use the Members tab to assign users and/or groups to the role to provide them with access to the capability. In the screenshot below I have added the Metacoda Plug-ins Users group I identified in step 1. You could also add individual users here but I prefer to use groups in role memberships and manage user access through group memberships instead.

Now the new custom role is ready, click the OK button to save the changes.

6. Test the New Metacoda Security Plug-ins Role

You can test this new role by asking one of the target users to log-in to SAS Management Console and verify Metacoda Security Plug-ins are now available to them. Alternatively, as an administrator, you could use impersonation techniques to log-in as one of the target users and verify this for yourself.

In the screenshot below you can see the SAS Demo User has limited access to just Metacoda Security Plug-ins and the standard SAS Management Console Authorization Manager, Data Library Manager and User Manager plug-ins, but none of the other plug-ins (such as Server Manager). Access to Metacoda Security Plug-ins has been provided via the new custom role. Access to the other plug-ins is provided via the predefined Management Console: Content Management role in its default state.

Final Thoughts

Whilst it’s easy to provide limited non-administrator access to Metacoda Security Plug-ins where required, bear in mind that those users may only be seeing part of the picture. Metacoda Security Plug-ins do not attempt to bypass metadata security, so users can only review security metadata on objects they would ordinarily be able to see (where they have an effective grant of the ReadMetadata permission). Since they are not unrestricted users and not user administrators, they will also only be able to see their own logins and logins for any groups they are a member of. If any folders or objects in metadata have been hidden from those users (with an effective denial of ReadMetada) then they won’t be able to review the security metadata for those folders and objects. If there is a requirement for those users to review security metadata for content they would not ordinarily be able to see, it is best handled by getting an administrator to export HTML reports from Metacoda Security Plug-ins and publish them in an area accessible to those users.

These instructions show how to provide role-based access to all of the features in Metacoda Security Plug-ins through a single capability. Depending on your role memberships you can either access none of the features or all of the features. If there is a need for it, in a future version, we can register the individual reviewers, and even the tabs within those reviewers, as specific individual capabilities to allow for much finer role-based access. If this is something that is important to you then please let me know.

Login Reviewer: Finding Accounts with Stored Passwords

In a previous post I mentioned how the Accounts tab in SAS® Management Console 9.3 now displays a blank in the password field when logins don’t have a stored password, and only displays ******** when there is a stored password. Compare this to SAS 9.2 where it always displays ******** regardless of whether there’s a stored password or not.

This is a great enhancement in SAS 9.3 because it allows us to know whether a password is stored in metadata or not. Sometimes it’s necessary to store passwords in metadata, but we generally try to minimize this. Passwords stored in metadata might be wrong and can get stale when password changes are enforced.

So now thanks to SAS 9.3 we can spot stored passwords when looking at individual users and groups. At Metacoda, we also wanted to be able to see, in one view, all logins that have stored passwords, across all users and groups, and in all repositories. This would show us how prevalent stored passwords are and which users and groups have them.

We’ve just enhanced the Login Reviewer for the next version of our Metacoda Security Plug-ins software to add a Password indicator column. Here’s a screenshot of this Password indicator column shown in SAS Management Console 9.3. I’ve sorted the Password indicator column to group together all the logins with and without stored passwords.

You might notice that one of the logins above is for a group found in a custom repository. I don’t recommend this approach, it’s just there for testing purposes. When I’m reviewing security metadata, I definitely want to know if there are things like this tucked away in custom repositories :)

Finally, for completeness, here’s another similar screenshot of the Login Reviewer’s Password indicator column, but this time in SAS Management Console 9.2. With SAS 9.2, when logged in as an unrestricted user, we can’t tell if there are stored passwords or not. This is why the screenshot below shows the column full of ‘Unknown‘ values. With SAS 9.2 we can only show Yes/No values when logged in as a normal user (in which case they will only get to see their own logins and any logins for groups they are a member of).

If you’d like to try this out, along with the other enhancements we’ve got planned for our next Metacoda Security Plug-ins version, then please let me know. We’re keen to talk to anyone who’d like to try out the beta when it’s available.

SAS and IWA: Two Hops

Update 26Sep2018: This post is now several years old and naturally technology and security have progressed in that time. For more up to date information regarding delegation and, in particular, the requirement for constrained delegation when working with Windows Defender Credential Guard in Windows 10 and Windows Server 2016, please see Stuart Rogers’ very useful SAS Global Forum 2018 Paper: SAS 9.4 on Microsoft Windows: Unleashing Kerberos on Apache Hadoop.

In an earlier post I mentioned that I would jot down a few notes about my experiences with SAS® software and Integrated Windows Authentication (IWA). This is the first of these posts and concerns the initial configuration. Chances are, if you knew you wanted to use IWA before you installed SAS, then it would have been discussed and implemented during the initial installation and configuration. If you decided to implement IWA after the fact then you would most likely have followed the instructions from either:

A basic SAS and IWA configuration might then look something like this. In the diagram below we have a client PC (saspc001), a dedicated metadata server machine (sasmeta) and an application server machine (sasapp). This is a homogenous environment consisting of all Windows machines in the same Windows domain. Other configurations might have multiple domains that trust each other, and now with SAS 9.3, some of the SAS servers may also be UNIX based (assuming the prerequisites are met).

In the diagram above a SAS Enterprise Guide user working on the saspc001 workstation initially connects (1) to the SAS Metadata Server on sasmeta using a connection profile with IWA enabled. When they run a project, an IWA connection (2) is then made to the SAS Object Spawner on sasapp to launch a standard SAS Workspace Server to execute the SAS code. The logical SAS Workspace Server has been configured in metadata to accept IWA connections. Both of these IWA connections involve only 1-hop from the workstation: saspc001 to sasmeta, and saspc001 to sasapp.

Problems might then arise when secondary connections need to be made from the workspace server to additional servers and access denied errors are seen in the SAS log. One example of a secondary connection includes executing code on the workspace server that reads a CSV or XML file from another file server (filesrv) using a UNC path (e.g. \\filesrv\share\file.xml). Another example might be assigning a library in the workspace server session that uses SAS/ACCESS Interface to ODBC or SAS/ACCESS Interface to OLEDB to connect to a Microsoft SQL Server database on another server (sqlsrv). These examples are shown in the diagram below as (3) and (4) respectively.

Both of these example involve IWA being used in 2-hops from the client. In the first hop (2) IWA is used to connect from saspc001 to sasapp as before. In the second hops the SAS workspace server process has to then connect and authenticate to the secondary servers: sasapp to filesrv (3), and sasapp to sqlsrv (4). It is these second hops which may fail if additional measures have not been taken:

Trusted for Delegation: the intermediate server (sasapp in this example – where the workspace server is running) needs to configured in Active Directory as Trusted for Delegation. This must be done by a domain admin. This configuration is mentioned in the SAS Intelligence Platform: Security Administration Guide on the Windows Privileges page for both SAS 9.2 and SAS 9.3. In a future post I’ll show the method I use as a non-domain-admin to double check this as part of the troubleshooting process.
Force Kerberos: you also need to ensure the Kerberos protocol is used and not NTLM. Whilst you could get all of your users to configure their SAS client connection profiles to use Kerberos, it is usually preferable to leave the clients alone and instead configure the SAS servers to only offer Kerberos and not NTLM. This is documented in the in the SAS Intelligence Platform: Security Administration Guide on the How to Force Use of Kerberos page for SAS 9.2, and the How to Configure Integrated Windows Authentication page for SAS 9.3. Things can get a bit trickier when DNS host aliases (or CNAMEs) are used in environments configured for disaster recovery. In a future post I’ll show some examples of additional Service Principal Names (SPNs) that might be required in these situations.

So if you find yourself getting access denied messages when using SAS and IWA in situations where multiple hops are involved, I hope this post gives you some ideas of things to investigate further.

For more posts in this series have a look at the IWA tag.