Rgmanager, clustat and clusvcadm are useful tools in cluster land. I recently built a custom resource which I added to one of my service chains. Upon inspecting clustat, I noticed:
[root@server1 ~]# clustat Member Status: Quorate Member Name ID Status ------ ---- ---- ------ server1 1 Online, Local, rgmanager server2 2 Online, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- service:service-main-server1 (server1) failed
Looking at /var/log/messages, I found:
server1 rgmanager: [script] script:shorewall-reload: start of shorewall-reload.sh failed (returned 2) server1 rgmanager: start on script "shorewall-reload" returned 1 (generic error)
This was peculiar, because my script didn’t have an exit code of 2 anywhere. It was due to a syntax error (woops)! Moving on with the syntax error fixed, I had trouble getting the service going again. In the logs I found these:
server1 rgmanager: #68: Failed to start service:service-main-server1; return value: 1 server1 rgmanager: Stopping service service:service-main-server1 server1 rgmanager: #12: RG service:service-main-server1 failed to stop; intervention required server1 rgmanager: Service service:service-main-server1 is failed server1 rgmanager: #13: Service service:service-main-server1 failed to stop cleanly
Running commands like: clusvcadm -e service-main-server1 didn’t help. It turns out that you have to first convince rgmanager that you truly fixed the problem, by first disabling the service. Now you can safely enable it and things should work smoothly:
clusvcadm -d service-main-server1 clusvcadm -e service-main-server1
Hopefully you’ve now got your feet wet with this clustering intro! Remember that you can look in the logs for clues and run clustat -i 1 in a screen session to keep tabs on things.
You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.