Cluster Issue

Why the SP2 Failover failed?

Observations :

1.       After analyzing the logs ,we noticed the below error in cluster logs.  The error what we noticed that the cluster event “get_disk_vg_fs” failed . On further analysis  ,to pinpoint where was the actual issue and why this event failed we further deep-dived the logs . we found that the Cluster services   had issues while activating/mounting the Cluster filesystem /sapmnt/SP2.

   

2.       When we initiated the  SP2 cluster failover , it will un-mount the Filesystems , export the VG’s from node1 and after this it will import all the VG’s and mount the respected Filesystems  on node2 . As per  the logs ,the cluster VG’s  were successfully exported from  node1 and the VG’s  were imported successfully  on  node2   but while mouting the FS(/sapmnt/SP2) it was giving issues .   The cluster failed to mount the /sapmnt/SP2 filesystem  . 

3.       Once we got these details from cluster logs, we investigated further , to know why the cluster was facing issues with /sapmnt/SP2 filesystem during the failover . On further investigation ,we found that /sapmnt/SP2 filesystem was already  NFS -mounted  and also this filesystem was manually mounted on node2    though the normal NFS commands .  That means that cluster was not able to mount the FS since it was already mounted  . 

.

4.       We  verified with  the SAP/DB team in call ,about the requirement of /sapmnt/SP2 filesystem on node2  and upon confirmation, we have un-mounted it. As per the application team this filesystem is needed where SP2 application is running.  we   configured the filesystem  /sapmnt/SP2  as NFS-Crossmount inside cluster  to meet the requirement  and again performed the cluster failover test and application validation . 

Everything was fine .