Nudging SAS Viya Services Timeout

I had been puzzling over why some SAS® Viya™ services were not starting on a machine reboot. Initially I thought the answer appeared in the SAS Viya 3.2 Administration documentation set: see the General Servers and Services: Troubleshooting section.

I found that all the expected services started after:

[root@hostname ~]# /etc/init.d/sas-viya-all-services stop
[root@hostname ~]# rm -f /opt/sas/viya/config/data/consul/checks/*
[root@hostname ~]# /etc/init.d/sas-viya-all-services start
[root@hostname ~]# /etc/init.d/sas-viya-all-services status

However, on further investigation it turned out that it probably wasn’t a problem with those consul/checks files. After another reboot I found that, once again, only a subset of the services had started. Using systemctl to check the status I found the following:

[root@hostname ~]# systemctl status sas-viya-all-services
 sas-viya-all-services.service - start and stop all SAS services
   Loaded: loaded (/usr/lib/systemd/system/sas-viya-all-services.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Mon 2017-04-24 13:33:54 AEST; 31min ago
  Process: 717 ExecStart=/etc/init.d/sas-viya-all-services start (code=killed, signal=TERM)
 Main PID: 717 (code=killed, signal=TERM)
   CGroup: /system.slice/sas-viya-all-services.service

Apr 24 13:33:12 hostname su[12824]: (to sasrabbitmq) root on none
Apr 24 13:33:30 hostname sas-viya-all-services[717]: There are still 1 pending processes
Apr 24 13:33:30 hostname sas-viya-all-services[717]: Starting sas-viya-datatables-default
Apr 24 13:33:30 hostname sas-viya-all-services[717]: Starting sas-viya-deploymentBackup-default
Apr 24 13:33:30 hostname sas-viya-all-services[717]: Starting sas-viya-device-management-default
Apr 24 13:33:30 hostname sas-viya-all-services[717]: Pausing to allow services time to start...
Apr 24 13:33:54 hostname systemd[1]: sas-viya-all-services.service start operation timed out. Terminating.
Apr 24 13:33:54 hostname systemd[1]: Failed to start start and stop all SAS services.
Apr 24 13:33:54 hostname systemd[1]: Unit sas-viya-all-services.service entered failed state.
Apr 24 13:33:54 hostname systemd[1]: sas-viya-all-services.service failed.

So it was due to the amount of time it was taking sas-viya-all-services to start all the services. This is a simple dev/test deployment with everything on one machine, unlike a real deployment where they are much more likely to be distributed over multiple machines. I needed to bump up the timeout for sas-viya-all-services to allow it to complete.

I could see the current timeout settings with:

[root@hostname ~]# systemctl show sas-viya-all-services.service | grep ^Timeout
TimeoutStartUSec=15min
TimeoutStopUSec=15min

… so I bumped the timeout up to 60 minutes to give it more than enough time:

[root@hostname ~]# sed -i s/TimeoutSec=15min/TimeoutSec=60min/ /usr/lib/systemd/system/sas-viya-all-services.service
[root@hostname ~]# systemctl daemon-reload
[root@hostname ~]# systemctl show sas-viya-all-services.service | grep ^Timeout
TimeoutStartUSec=1h
TimeoutStopUSec=1h

… did another reboot and watched the progress with:

[root@hostname ~]# tail -f `ls -1 /opt/sas/viya/config/var/log/all-services/default/all-services*.log | tail -n 1`

… then when complete verified with:

[root@hostname ~]# systemctl status sas-viya-all-services
 sas-viya-all-services.service - start and stop all SAS services
   Loaded: loaded (/usr/lib/systemd/system/sas-viya-all-services.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2017-04-24 15:09:15 AEST; 2min 24s ago
  Process: 722 ExecStart=/etc/init.d/sas-viya-all-services start (code=exited, status=0/SUCCESS)
 Main PID: 722 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sas-viya-all-services.service

Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-backup-agent-default                          00:01:41
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-environmentmanager-default                    00:02:41
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-monitoring-default                            00:02:41
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-sashome-default                               00:02:41
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-sasreportviewer-default                       00:02:47
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-sasthemedesigner-default                      00:02:28
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-sasvisualanalytics-default                    00:02:27
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-viya-sasvisualdatabuilder-default                  00:02:27
Apr 24 15:09:15 hostname sas-viya-all-services[722]: sas-services completed in 00:31:05
Apr 24 15:09:15 hostname systemd[1]: Started start and stop all SAS services.

… or:

[root@hostname ~]# /etc/init.d/sas-viya-all-services status
Getting service info from consul...
  Service                                            Status     Host               Port     PID
  sas-viya-consul-default                            up         N/A                 N/A    3727
  sas-viya-sasdatasvrc-postgres-node0-ct-pg_hba      up         N/A                 N/A    3969
  sas-viya-sasdatasvrc-postgres-node0-ct-postgresql  up         N/A                 N/A    3991
  sas-viya-sasdatasvrc-postgres-pgpool0-ct-pcp       up         N/A                 N/A    4036
  sas-viya-sasdatasvrc-postgres-pgpool0-ct-pgpool    up         N/A                 N/A    4054
  sas-viya-sasdatasvrc-postgres-pgpool0-ct-pool_hba  up         N/A                 N/A    4278
  sas-viya-sasdatasvrc-postgres                      up         N/A                 N/A    4740
  sas-viya-cascontroller-default                     up         N/A                 N/A     930
  sas-viya-httpproxy-default                         up         N/A                 N/A    6290
  sas-viya-rabbitmq-server-default                   up         N/A                 N/A    6039
  sas-viya-sasdatasvrc-postgres-node0                up         N/A                 N/A    4689
  sas-viya-sasstudio-default                         up         N/A                 N/A    1115
  sas-viya-spawner-default                           up         N/A                 N/A     858
...
  sas-viya-sasvisualanalytics-default                up         10.10.10.10       43115   11303
  sas-viya-sasvisualdatabuilder-default              up         10.10.10.10       37424   11334

sas-services completed in 00:00:18

Given the extra time, now all of the SAS Viya services get a chance to start after a machine reboot.