Callhome SRC E3325009 reported for Version 9

Resolving The Problem

If the issue has to do with Connectivity to one or more managed systems, resolve the connectivity state of "No Connection", "Pending Authentication", or "Failed Authentication" as seen in the lssysconn -r all output.  After resolving the connectivity issues monitor for no more reoccurrences of the SRC reporting.

If the issue is not related to connectivity, the full fix which includes the M940 fixes for E3325009 plus an ipv6 disablement update is now available in the V9.1 M941 Service Pack Release

PTF MH01859 HMC V9 R1.941.0 Service Pack - for 7042 Machine Types or vHMC for x86 hypervisors (5765-HMW)

PTF MH01860 HMC V9 R1.941.0 Service Pack - for 7063 Machine Types or vHMC for PowerVM (5765-HMB)

Another fix is available specific to any configuration that includes Virtual HMC arbitration in HMC V9 R1.941.2:

PTF MH01864 HMC V9 R1.941.2 - for 7042 Machine Types or vHMC for x86 hypervisors (5765-HMW)

PTF MH01865 HMC V9 R1.941.2 - for 7063 Machine Types or vHMC for PowerVM (5765-HMB)

Fixed vHMC MTMS parsing logic for primary HMC arbitration to handle virtual HMCs uniquely to prevent the chprimhmc command error "Failed to get service operation information from peer" and subsequent E3325009 reporting.

Fix Central Download

If E3325009 continues to report the most likely cause is the HMC has a reference to an HMC name or IP address that is no longer valid or reachable. 

Some examples include:

•        The order the HMCs were rebooted caused one HMC to not get refreshed with the new network information. Reboot the HMC reporting the E3325009 one more time. 
•        An older HMC model was replaced with a new model and given the same hostname and IP address
•        An HMC name or IP address was updated in DNS, but the changes have not been made on the HMC itself
•        Port 9920 is blocked in the HMC firewall or in the customer's firewall between the HMCs

FURTHER TROUBLESHOOTING STEPS IF Step 1 and 2 do not mitigate the reporting of E3325009:

1. Verify that all the ipaddresses defined on the HMC are correct and in use:

hmc01>lshmc -n -Fhostname:ipaddr
hmc01:10.1.1.10,11.9.9.9,10.0.0.32

2. If an ipaddress was decommissioned or is not supposed to be in use by the HMC, remove it from the configuration in the "Change Network Settings" menu under HMC Management.  Continue with the additional steps for further network validation before rebooting both HMCs.

Example: hostname is hmc01 and ipaddress is 10.1.1.10
         hostname is hmc02 and ipaddress is 10.1.1.20

2a.  If DNS is not in use, ensure a unique hostname is set, not localhost, and skip to step 3.

2b.  If DNS is in use, verify the dns setup is identical on both HMCs. 

From hmc01:  

hmc01>lshmc -n -Fhostname:domain:nameserver:dns:domainsuffix
hmc01:boulder.ibm.com:10.0.130.50,10.0.128.50:enabled:boulder.ibm.com

From hmc02: 

hmc02>lshmc -n -Fhostname:domain:nameserver:dns:domainsuffix
hmc02:boulder.ibm.com:10.0.130.50,10.0.128.50:enabled:boulder.ibm.com

2c. Check forward and reverse lookup matches on both HMCs for each other.

From hmc01:  

hmc01>host hmc02
hmc02.boulder.ibm.com had address 10.1.1.20

hmc01>host 10.1.1.20
10.1.1.20 in-addr.arpa domain name pointer hmc02.boulder.ibm.com

From hmc02: 

hmc02>host hmc01
hmc01.boulder.ibm.com had address 10.1.1.10

hmc02>host 10.1.1.10
10.1.1.10 in-addr.arpa domain name pointer hmca01.boulder.ibm.com

The reverse lookup shows hmca01 not hmc01, which is not a match.  The DNS records need to be fixed to ensure forward and reverse lookup match.  

3. Verify that port 9920 is open in the firewall for all interfaces on both HMCs with no restriction on the allowed host:

hmc01>lshmc --firewall | grep 9920
interface=eth1,application=FCS,"ports=9920:tcp,udp:9900",allowedhost=0.0.0.0/0.0.0.0
interface=eth2,application=FCS,"ports=9920:tcp,udp:9900",allowedhost=0.0.0.0/0.0.0.0
interface=eth0,application=FCS,"ports=9920:tcp,udp:9900",allowedhost=0.0.0.0/0.0.0.0
interface=eth1,application=FCS,"ports=9920:tcp6,udp6:9900",allowedhost=::/::
interface=eth2,application=FCS,"ports=9920:tcp6,udp6:9900",allowedhost=::/::
interface=eth0,application=FCS,"ports=9920:tcp6,udp6:9900",allowedhost=::/::

Repeat the lshmc --firewall command from hmc02 and verify the same results.

4. Verify 9920 is connected between the HMCs and shows as "ESTABLISHED":

hmc01>cat $(ls -tr /dump/HMCMonitor/netstat* | tail -1) | grep 9920 | grep 10.1.1.20

tcp6       0      0 10.1.1.10:9920          10.1.1.20:37976         ESTABLISHED 7340/java

hmc02>cat $(ls -tr /dump/HMCMonitor/netstat* | tail -1) | grep 9920 | grep 10.1.1.10

tcp6       0      0 10.1.1.20:39998         10.1.1.10:9920        ESTABLISHED 7952/java

5. Verify with the network administrator that no firewall is blocking port 9920 between the HMCs for communication if the 9920 port shows any other status than ESTABLISHED or is not present.

6. Verify each HMC knows who is primary for each managed system after a reboot:

From hmc01:

hmc01>lsprimhmc -m 9009-42A-78XXXXX
is_primary=1,primary_hmc_mtms=VM-Vdxx-xxx/xxxxxxx,primary_hmc_ipaddr=10.1.1.10,primary_hmc_hostname=hmc01

From hmc02:

hmc02>lsprimhmc -m 9009-42A-78XXXXX
is_primary=0,primary_hmc_mtms=VM-Vdxx-xxx/xxxxxxx,primary_hmc_ipaddr=10.1.1.10,primary_hmc_hostname=hmc01

If the output shows good values and good communication, E3325009 may report one time around the HMC rebooted for one or more managed systems due to timing, but will stop reporting on the daily schedule basis. If the output shows blank values from either HMC, then the HMCs are still not communicating with each other and E3325009 will continue reporting every day.  A pedbg -c -q 4 collection from both HMCs is necessary for further debugging of the problem. Contact IBM hardware support for additional assistance. 

Blank value example:

hmc02>lsprimhmc -m 9009-42A-78XXXXX
is_primary=0,primary_hmc_mtms=,primary_hmc_ipaddr=,primary_hmc_hostname=