Promoting Job Flows

I am occasionally asked whether it is possible to promote job flows between metadata repositories (e.g. DEV to TEST to PROD) using the SAS package file (SPK) import/export promotion feature. In past consulting work I have needed to promote job flows containing hundreds of jobs all interlinked with dependencies, so this is a topic close to my heart :)

In the early days of SAS 9.1.3 import/export of job flows was not supported and so the only choices availably back then were to either manually reconstruct the jobs flows in the target environment or do a full replication/promotion (what used to be called full promotion in SAS 9.1.3 is now called replication in SAS 9.2). Manually reconstructing jobs flows was a tedious and error-prone process especially with large and complicated job flows. Full replication/promotion was often not appropriate because it would overwrite the target environment completely. But that’s all history (and has been for a while) …

With SAS 9.2 there is no question about it – import and export of job flows is standard with SAS 9.2 so I will ignore it for the rest of this post and concentrate on SAS 9.1.3 instead.

Since the import/export of job flows was not initially supported in SAS 9.1.3, I occasionally talk to people who assume it’s still the case now (which is understandable). It’s great to see the smile spread across their faces when they find out it is actually possible with SAS 9.1.3 today – I remember back to how I felt when I first found out. I was faced with the prospect of manually reconstructing some awe-inspiring job flows and contacted SAS on the off-chance that someone might know a better way – which is how I was lucky enough to get to try out an early version. You can imagine how happy I was.

The ability to perform partial promotion of job flows crept relatively quietly into SAS 9.1.3 with a couple of hotfixes some time back. There is a usage note about it now too: Usage Note 31008: Hot Fixes 913SMC04 and 14JPS02 provide the capability to add job flows using partial promotion

So if you need to promote job flows between SAS 9.1.3 environments here are some pointers that you might find useful:

  • Promote the job flows using the SAS Management Console 9.1 BI Manager plug-in. Last time I tried I couldn’t export job flows from SAS Data Integration Studio 3.4 (even with the latest hotfixes at the time). I last looked many months ago though, so happy to be corrected if that has changed since.
  • Make sure the SAS Management Console 9.1 client installation you will be using (most likely your workstation) has been updated to at least the level of hotfix 913SMC04. I would tend to install the most recent SAS Management Console 9.1 hotfix 913SMC08 instead.
  • This is already in the instructions for 913SMC08, but you also need to ensure the client installation you will be using (most likely your workstation) has been updated to the level of SAS Foundation Services 1.4 hotfix 14JPS02. You might recognize SAS Foundation Services 1.4 from mid-tier installations, but it also contributes the BI Manager plug-in to SAS Management Console 9.1 client installations.
  • Using BI Manager in the source repository track down the folder containing the job flows you want to export and export them into a SAS package (SPK) file. If you can’t find the job flows try looking in the /Shared Data folder. Here is a screenshot showing job flows being exported from SAS Management Console 9.1:
  • Using BI Manager in the target repository import the job flows from the previously exported SAS package (SPK) file. Once imported you will need to re-schedule the flows.

MKDIRMD Macro: Creating Tree Folders in Metadata

Update 10Jul2015: After 5 years, the macro discussed in this blog post is now very dated. If you still have SAS 9.1.3 then it should be ok, but for newer versions of SAS software (such as 9.2, 9.3 and 9.4) there are better options provided as standard by SAS Institute:

1) SAS Platform Object Framework MakeFolder Metadata Utility: see my later blog post

2) SAS 9.4 has the Make Folder (sas-make-folder) command line tool documented in the SAS 9.4 Intelligence Platform: System Administration Guide, Third Edition” under “Batch Tools for Metadata Management”.

The source code for the MKDIRMD macro mentioned below has a check for SAS 9.2 at the end, that should include later versions of SAS (if used with later versions of SAS). However, I would strongly recommend using one of the newer supported methods instead (as mentioned above).

The inspiration for this post came from a question at sasprofessionals.net about code to create tree folders in a SAS® metadata repository. It sounded like a good challenge considering that:

  • Creating a Tree object in metadata is relatively straightforward, but finding the correct parent Tree object to associate it with takes more work. You can’t search by the parent folder name in isolation because there might be multiple folders with the same name at different locations in the tree (e.g. there could be 2 folders both named Reports at /ACME/Sales/Data and /ACME/HR/data). The parent folder could be specified by its object id but that’s a bit tedious. A neater method would be to allow the parent path to be specified in absolute terms (e.g. /ACME/HR/Data/) and get the code to walk down the tree to find the correct unique parent Tree object (e.g. Data under HR under ACME).
  • It should be able to create top level folders in the root of the tree as well as folders at lower depths. Top level folders are represented slightly differently in metadata than lower level folders. They don’t have a ParentTree association but are instead associated with a SoftwareComponent.
  • It should fail if a folder with the same name already exists in the parent folder. The metadata API allows you from create duplicate named folders in the same parent folder.
  • It should check all return codes and fail fast – it is being used to update metadata after all.
  • It would be good if it worked in both SAS 9.1 and SAS 9.2

I had some metadata API fun writing a MKDIRMD macro that satisfied all of the above. Here is an example of it being used:
%mkdirmd(name=ACME, parent=/);
%mkdirmd(name=Sales, parent=/ACME/);
%mkdirmd(name=StoredProcesses, parent=/ACME/Sales/);
%mkdirmd(name=Data, parent=/ACME/Sales/);
%mkdirmd(name=HR, parent=/ACME/);
%mkdirmd(name=StoredProcesses, parent=/ACME/HR/);
%mkdirmd(name=Data, parent=/ACME/HR/);

I have uploaded the MKDIRMD.SAS code I wrote so it can be used by anyone that may have a need for it, or by those who might want to review it as an example program that uses the SAS Open Metadata Interface to query and update metadata. It includes examples of XMLSelect filters on both attribute criteria and association paths. I am publishing this code under the GNU LGPL license so it can be freely used. I have tested the code in both SAS 9.1.3 SP4 and SAS 9.2 M3 but it is provided “as-is” and should be used with care.

Be aware that with any code like this, that updates your metadata, it is strongly recommended that you have a recent backup (that is known to work) and test the code in a non-production environment. I prefer testing things like this in a private administrator environment (Lev9) since even development environments should be considered valuable.

Found the SAS® 9.2 Open Metadata Interface documentation

I find myself referring to the SAS® Open Metadata Interface documentation on a very regular basis and have been keenly awaiting the availability of said documentation for SAS 9.2. I have been scanning the SAS 9.2 Product Documentation A-Z Listing every now and then hoping to see it appear one day but so far it has been absent. I just assumed it hadn’t been released yet. Today, however it occurred to me that perhaps I should try using the support.sas.com search facility instead (how come I didn’t think of that before I wondered). Expecting only hits relating to the SAS 9.1.3 docs, I was very pleasantly surprised to find SAS 9.2 Open Metadata Interface links in the results too.

In case anyone else is looking for this documentation, and to help me find it again, here are the relevant documents I found:

SAS® 9.2 Language Interfaces to Metadata
This document does appear in the A-Z index under Base SAS (and has for a while I think). It can be found here:

SAS® 9.2 Open Metadata Interface: Reference and Usage
This document is not yet in the A-Z index but can be found here (as discovered by searching):

SAS® 9.2 Metadata Model: Reference
This document is not yet in the A-Z index but can be found here (as discovered by searching):

SAS® 9.2 Java Metadata Interface (JavaDoc)
This JavaDoc can be found (and has for a while) in the SAS AppDev Studio focus area at http://support.sas.com/rnd/gendoc/bi92/api/metadata/index.html

Now, of course, I am kicking myself wondering how long all of this documentation has actually been available while I have been waiting for it to appear in the A-Z list :) Can’t dwell on that now though – I have lots of reading ahead.

BTW – does anyone know if there is any published documentation for the SAS 9.2 Management Console Plug-in API available yet?

Metadata Reporting with SAS® Software

I was reminded today of the “Metadata reporting with SAS® software” presentation I did at a SNUG meeting way back in Sep 2007. It seems such a long time ago now, but when I read through it again it is still relevant today. The presentation is mainly about the use of PROC METADATA and the SAS XML Libname Engine (SXLE). It also lists a few metadata reports that I have found useful in the past and were done using this METADATA/SXLE technique:

Sample Report: list of objects currently checked-out (locked) by SAS DI Studio users

Sample Report: reverse chronological list of all check-ins performed by SAS DI Studio users, with links to review the jobs and tables associated with each check-in



Sample Report: list of SAS DI Studio jobs that need to be re-deployed for scheduling

Sample Report: comparison of SAS DI Studio jobs from 2 repositories to identify those that need to be promoted

The presentation is available in PDF format as slides-only as well as slides-with-notes that provide a little more information. The slides show code fragments and so here are a couple of complete, albeit basic, examples that can be used as starting points for further work:

  • ProcMetadataSample1.sas: this is a basic example to extract and print user information (id, name, title and email address) for any user whose name begins with ‘Paul’ (using XMLSELECT)
  • ProcMetadataSample2.sas: extract group information only for those groups that have a user member whose user name begins with ‘SAS’ – this is an example of XMLSELECT filtering based on associations

It was around this time that I had the idea to pursue metadata reporting by creating plug-ins for the SAS Management Console using Java. This ultimately resulted in the creation of our Metacoda Security Plug-ins product used to review and report on security metadata.

Tracking Down the SAS Management Console Log File

Recently I have been updating the documentation for our Metacoda Security Plug-ins relating to their use with SAS® 9.2. In our documentation we mention the SAS Management Console log file as a potential source of support information if help is needed. The SAS Management Console log file can provide clues such as error messages and stack traces. Our software writes information messages to the log file too. It’s a good starting point for troubleshooting.

With SAS 9.1 the location of the SAS Management Console log file was easy to remember. With SAS 9.2 it has moved a little deeper into the file system and I haven’t yet committed its location to memory … which I’m sure will happen soon … perhaps even by the end of this post :).

As I was updating our documentation I was hoping that I could add a reference to a location in the official SAS documentation where people could find out more about the SAS Management Console log file. I couldn’t find any specific information about the log file itself but did see a few references to it in other sections. In some cases the documentation I did find differed a little from my own experiences and so in the rest of this post I will refer to the SAS documentation and also mention where it might differ from my own experience with the software.

The default location for the SAS Management Console log file depends on both the SAS software version you are using as well as the operating system you are running it on.

SAS Management Console 9.1

In my use of SAS Management Console 9.1 I have found that it writes its log to a file named errorLog_SAS_Management_Console.txt in the user’s home directory by default. Potential log file locations for someone with a userid of paul might include:

Windows®
C:\Documents and Settings\paul\errorLog_SAS_Management_Console.txt
UNIX
/home/paul/errorLog_SAS_Management_Console.txt

This differs from SAS® 9.1.3 Management Console User’s Guide, Chapter 2 Setting up SAS Management Console in the section Using the Error Log which has:

This file is named errorlog.txt, and is located by default in the SAS Management Console directory.

SAS Management Console 9.2

With SAS Management Console 9.2 the log file has been renamed to SASMCErrorLog.txt and has relocated a few subdirectories down in the user’s home directory.

I found a SAS documentation reference to the log file location in the SAS® 9.2 Intelligence Platform: System Administration Guide in a section named Troubleshooting a Replication Job. It mentions the following potential locations for SASMCErrorLog.txt:

  • On Windows Vista systems: C:\Users\user\AppData\Roaming\SAS\SASManagementConsole\9.2
  • On other Windows systems: C:\Documents and Settings\user\Application Data\SAS\SASManagementConsole\9.2
  • On UNIX systems: user’s-home-directory/SAS/SASManagementConsole/9.2

The default location and layout of the Windows user profile changed with Windows Vista which explains why the SAS Management Console log file location is different from Windows Vista onwards. Having installed SAS Management Console 9.2 in Windows 7 I have seen that this new log file location applies to Windows 7 too.

The UNIX location as specified in the SAS documentation reference above differs from my own experience with a Linux based installation of SAS Management Console 9.2. On Linux I found SASMCErrorLog.txt was in the location user’s-home-directory/.SASAppData/SASManagementConsole/9.2/

Combining my own experiences with information from the SAS documentation, potential locations for the SAS Management Console 9.2 log file for someone with a userid of paul might include:

Windows 7, Windows Vista
C:\Users\paul\AppData\Roaming\SAS\SASManagementConsole\9.2\SASMCErrorLog.txt
Windows (versions prior to Windows Vista)
C:\Documents and Settings\paul\Application Data\SAS\SASManagementConsole\9.2\SASMCErrorLog.txt
UNIX, Linux
/home/paul/.SASAppData/SASManagementConsole/9.2/SASMCErrorLog.txt

Of course these paths will need to be adjusted if your system administrators have configured operating system installations at your site to store home directories / user profiles in alternative locations.

I would be interested to find out if anyone has found their SAS Management Console log file in any locations other than these.