warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
warning: synonym name HCOLL_ENABLE_MCAST_ALL is used together with the basename HCOLL_ENABLE_MCAST. Basename value will be used.
[b1177:173476:0:173476] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173476:0:173476] ib_mlx5_log.c:139  DCI QP 0xcbfb wqe[36]: SEND s-e [rqpn 0x18348 rlid 159] [va 0x2ab0599d8d00 len 1034 lkey 0x121b5e]
[b1177:173479:0:173479] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173479:0:173479] ib_mlx5_log.c:139  DCI QP 0xcbe3 wqe[42]: SEND s-e [rqpn 0x18045 rlid 159] [va 0x2b4f049d8d00 len 1034 lkey 0x1291db]
[b1177:173500:0:173500] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173500:0:173500] ib_mlx5_log.c:139  DCI QP 0xcbf5 wqe[4]: SEND s-e [rqpn 0x18110 rlid 159] [va 0x2b8bc53d6c80 len 1034 lkey 0xa6c22]
[b1177:173502:0:173502] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173502:0:173502] ib_mlx5_log.c:139  DCI QP 0xcbe4 wqe[36]: SEND s-e [rqpn 0x17cd2 rlid 159] [va 0x2b94a03d8d00 len 1034 lkey 0xc9d61]
[b1177:173517:0:173517] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173517:0:173517] ib_mlx5_log.c:139  DCI QP 0xcbe8 wqe[9]: SEND s-e [rqpn 0x18326 rlid 159] [va 0x2b49aebd8d00 len 1034 lkey 0x12064f]
[b1177:173462:0:173462] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173462:0:173462] ib_mlx5_log.c:139  DCI QP 0xcbda wqe[36]: SEND s-e [rqpn 0x17fa7 rlid 159] [va 0x2b6ed45d8d00 len 1034 lkey 0x7bb3e]
[b1177:173475:0:173475] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173475:0:173475] ib_mlx5_log.c:139  DCI QP 0xcbc5 wqe[42]: SEND s-e [rqpn 0x17cb9 rlid 159] [va 0x2abe17bd8d00 len 1034 lkey 0x11fb40]
[b1177:173482:0:173482] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173482:0:173482] ib_mlx5_log.c:139  DCI QP 0xcbd1 wqe[36]: SEND s-e [rqpn 0x18337 rlid 159] [va 0x2ba3405d8d00 len 1034 lkey 0x1188ec]
[b1177:173497:0:173497] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173497:0:173497] ib_mlx5_log.c:139  DCI QP 0xcbd8 wqe[42]: SEND s-e [rqpn 0x180ea rlid 159] [va 0x2afbd4dd8d00 len 1034 lkey 0xa476c]
[b1177:173499:0:173499] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173499:0:173499] ib_mlx5_log.c:139  DCI QP 0xcbd4 wqe[36]: SEND s-e [rqpn 0x18064 rlid 159] [va 0x2b8ae0bd8d00 len 1034 lkey 0x111a21]
[b1177:173507:0:173507] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173507:0:173507] ib_mlx5_log.c:139  DCI QP 0xcbb1 wqe[42]: SEND s-e [rqpn 0x18179 rlid 159] [va 0x2b28783d8d00 len 1034 lkey 0x1126c4]
[b1177:173520:0:173520] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173520:0:173520] ib_mlx5_log.c:139  DCI QP 0xcbcb wqe[42]: SEND s-e [rqpn 0x18108 rlid 159] [va 0x2b57c1dd8d00 len 1034 lkey 0x114261]
[b1177:173521:0:173521] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173521:0:173521] ib_mlx5_log.c:139  DCI QP 0xcbc1 wqe[4]: SEND s-e [rqpn 0x18210 rlid 159] [va 0x2b9c0d9d6c80 len 1034 lkey 0x7f806]
[b1177:173461:0:173461] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173461:0:173461] ib_mlx5_log.c:139  DCI QP 0xcbfd wqe[36]: SEND s-e [rqpn 0x1831f rlid 159] [va 0x2b566b7d8d00 len 1034 lkey 0x1274ba]
[b1177:173473:0:173473] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173473:0:173473] ib_mlx5_log.c:139  DCI QP 0xcba7 wqe[10]: SEND s-e [rqpn 0x17fb7 rlid 159] [va 0x2ba660fd6c80 len 1034 lkey 0xa6118]
[b1177:173490:0:173490] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173490:0:173490] ib_mlx5_log.c:139  DCI QP 0xcc05 wqe[4]: SEND s-e [rqpn 0x1827c rlid 159] [va 0x2b075abd6c80 len 1034 lkey 0x11080b]
[b1177:173465:0:173465] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173465:0:173465] ib_mlx5_log.c:139  DCI QP 0xcb9d wqe[42]: SEND s-e [rqpn 0x182e8 rlid 159] [va 0x2b7712bd8d00 len 1034 lkey 0xfe5e1]
[b1177:173508:0:173508] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173508:0:173508] ib_mlx5_log.c:139  DCI QP 0xcbf8 wqe[4]: SEND s-e [rqpn 0x180ed rlid 159] [va 0x2b58687d6c80 len 1034 lkey 0x8c02d]
[b1177:173506:0:173506] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173506:0:173506] ib_mlx5_log.c:139  DCI QP 0xcb99 wqe[36]: SEND s-e [rqpn 0x18138 rlid 159] [va 0x2ba9b1dd8d00 len 1034 lkey 0x111d2b]
[b1177:173509:0:173509] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173509:0:173509] ib_mlx5_log.c:139  DCI QP 0xcb92 wqe[36]: SEND s-e [rqpn 0x1834d rlid 159] [va 0x2af89add8d00 len 1034 lkey 0xbddad]
[b1177:173464:0:173464] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173464:0:173464] ib_mlx5_log.c:139  DCI QP 0xcb5f wqe[25]: SEND s-e [rqpn 0x18235 rlid 159] [va 0x2b29f31ba580 len 1034 lkey 0x1265a9]
[b1177:173496:0:173496] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173496:0:173496] ib_mlx5_log.c:139  DCI QP 0xcb4f wqe[25]: SEND s-e [rqpn 0x17fa5 rlid 159] [va 0x2b3a1cdc2780 len 1034 lkey 0x1285cb]
[b1177:173460:0:173460] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173460:0:173460] ib_mlx5_log.c:139  DCI QP 0xcb26 wqe[36]: SEND s-e [rqpn 0x17f9d rlid 159] [va 0x2b3995fd8d00 len 1034 lkey 0xcfdc0]
[b1177:173469:0:173469] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173469:0:173469] ib_mlx5_log.c:139  DCI QP 0xcb82 wqe[19]: SEND s-e [rqpn 0x181ca rlid 159] [va 0x2b2fc15c4800 len 1034 lkey 0x11bf87]
[b1177:173470:0:173470] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173470:0:173470] ib_mlx5_log.c:139  DCI QP 0xcb54 wqe[42]: SEND s-e [rqpn 0x18156 rlid 159] [va 0x2b2b9c7d8d00 len 1034 lkey 0x1260b3]
[b1177:173480:0:173480] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173480:0:173480] ib_mlx5_log.c:139  DCI QP 0xcb1e wqe[42]: SEND s-e [rqpn 0x1800b rlid 159] [va 0x2b288c1d8d00 len 1034 lkey 0xc0152]
[b1177:173487:0:173487] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173487:0:173487] ib_mlx5_log.c:139  DCI QP 0xcb67 wqe[42]: SEND s-e [rqpn 0x18249 rlid 159] [va 0x2abf02bd8d00 len 1034 lkey 0x10bcf2]
[b1177:173516:0:173516] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173516:0:173516] ib_mlx5_log.c:139  DCI QP 0xcaf9 wqe[20]: SEND s-e [rqpn 0x182ff rlid 159] [va 0x2b81be5c2780 len 1034 lkey 0x771c3]
[b1177:173518:0:173518] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173518:0:173518] ib_mlx5_log.c:139  DCI QP 0xcafb wqe[20]: SEND s-e [rqpn 0x181d5 rlid 159] [va 0x2b58545c2780 len 1034 lkey 0x10e33d]
[b1177:173463:0:173463] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173463:0:173463] ib_mlx5_log.c:139  DCI QP 0xcaf8 wqe[36]: SEND s-e [rqpn 0x181dd rlid 159] [va 0x2b4b2cbd8d00 len 1034 lkey 0x1111a1]
[b1177:173492:0:173492] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173492:0:173492] ib_mlx5_log.c:139  DCI QP 0xca94 wqe[42]: SEND s-e [rqpn 0x17cda rlid 159] [va 0x2b50737d8d00 len 1034 lkey 0x11f331]
[b1177:173504:0:173504] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173504:0:173504] ib_mlx5_log.c:139  DCI QP 0xcaef wqe[42]: SEND s-e [rqpn 0x18283 rlid 159] [va 0x2aeb363d8d00 len 1034 lkey 0x1121c0]
[b1177:173472:0:173472] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173472:0:173472] ib_mlx5_log.c:139  DCI QP 0xcaec wqe[36]: SEND s-e [rqpn 0x182c2 rlid 159] [va 0x2b2356fd8d00 len 1034 lkey 0x104240]
[b1177:173523:0:173523] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173523:0:173523] ib_mlx5_log.c:139  DCI QP 0xca91 wqe[26]: SEND s-e [rqpn 0x18231 rlid 159] [va 0x2b5209dc4800 len 1034 lkey 0x76094]
[b1177:173481:0:173481] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173481:0:173481] ib_mlx5_log.c:139  DCI QP 0xca8c wqe[42]: SEND s-e [rqpn 0x18115 rlid 159] [va 0x2b09bcbd8d00 len 1034 lkey 0x10e03a]
[b1177:173478:0:173478] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173478:0:173478] ib_mlx5_log.c:139  DCI QP 0xca75 wqe[9]: SEND s-e [rqpn 0x18214 rlid 159] [va 0x2b2faddd8d00 len 1034 lkey 0x104341]
[b1177:173474:0:173474] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173474:0:173474] ib_mlx5_log.c:139  DCI QP 0xca4a wqe[36]: SEND s-e [rqpn 0x17cbf rlid 159] [va 0x2b1b3d5d8d00 len 1034 lkey 0xc958d]
[b1177:173477:0:173477] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173477:0:173477] ib_mlx5_log.c:139  DCI QP 0xca3c wqe[42]: SEND s-e [rqpn 0x18119 rlid 159] [va 0x2ae8ea9d8d00 len 1034 lkey 0x11fe02]
[b1177:173501:0:173501] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173501:0:173501] ib_mlx5_log.c:139  DCI QP 0xca3e wqe[26]: SEND s-e [rqpn 0x17fb8 rlid 159] [va 0x2aaed5fc4800 len 1034 lkey 0x11151d]
[b1177:173515:0:173515] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173515:0:173515] ib_mlx5_log.c:139  DCI QP 0xca76 wqe[26]: SEND s-e [rqpn 0x18343 rlid 159] [va 0x2b6de0bc8900 len 1034 lkey 0xf4971]
[b1177:173514:0:173514] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173514:0:173514] ib_mlx5_log.c:139  DCI QP 0xca16 wqe[32]: SEND s-e [rqpn 0x18310 rlid 159] [va 0x2aea8d7d2b80 len 1034 lkey 0x121123]
[b1177:173519:0:173519] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173519:0:173519] ib_mlx5_log.c:139  DCI QP 0xca29 wqe[42]: SEND s-e [rqpn 0x181d2 rlid 159] [va 0x2b5061fd8d00 len 1034 lkey 0xae0b0]
[b1177:173468:0:173468] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173468:0:173468] ib_mlx5_log.c:139  DCI QP 0xc9fe wqe[19]: SEND s-e [rqpn 0x181ce rlid 159] [va 0x2ae4fc3c4800 len 1034 lkey 0x1284ca]
[b1177:173512:0:173512] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173512:0:173512] ib_mlx5_log.c:139  DCI QP 0xca15 wqe[22]: SEND s-e [rqpn 0x1817c rlid 159] [va 0x2b0a335ed200 len 1034 lkey 0xb13ca]
[b1177:173491:0:173491] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173491:0:173491] ib_mlx5_log.c:139  DCI QP 0xc9cc wqe[42]: SEND s-e [rqpn 0x1822e rlid 159] [va 0x2b0b11fd8d00 len 1034 lkey 0xaf6c1]
[b1177:173505:0:173505] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173505:0:173505] ib_mlx5_log.c:139  DCI QP 0xc9e7 wqe[36]: SEND s-e [rqpn 0x1808f rlid 159] [va 0x2b8e7d5d8d00 len 1034 lkey 0x6c468]
[b1177:173510:0:173510] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173510:0:173510] ib_mlx5_log.c:139  DCI QP 0xc9dd wqe[36]: SEND s-e [rqpn 0x180aa rlid 159] [va 0x2ace14bd8d00 len 1034 lkey 0xac39c]
[b1177:173513:0:173513] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173513:0:173513] ib_mlx5_log.c:139  DCI QP 0xc97d wqe[26]: SEND s-e [rqpn 0x18262 rlid 159] [va 0x2b75dadc6880 len 1034 lkey 0x11e4d4]
[b1177:173488:0:173488] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173488:0:173488] ib_mlx5_log.c:139  DCI QP 0xc93f wqe[19]: SEND s-e [rqpn 0x1827f rlid 159] [va 0x2afb5e3c4800 len 1034 lkey 0x12c4d7]
[b1177:173498:0:173498] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173498:0:173498] ib_mlx5_log.c:139  DCI QP 0xc95c wqe[36]: SEND s-e [rqpn 0x180b5 rlid 159] [va 0x2b7bf91d8d00 len 1034 lkey 0x119333]
[b1177:173400:0:173400] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173400:0:173400] ib_mlx5_log.c:139  DCI QP 0xc925 wqe[19]: SEND s-e [rqpn 0x17cf8 rlid 159] [va 0x2aee63fc4800 len 1034 lkey 0x108993]
[b1177:173402:0:173402] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173402:0:173402] ib_mlx5_log.c:139  DCI QP 0xc94b wqe[10]: SEND s-e [rqpn 0x18004 rlid 159] [va 0x2b99351d6c80 len 1034 lkey 0xa9e8a]
[b1177:173435:0:173435] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173435:0:173435] ib_mlx5_log.c:139  DCI QP 0xc930 wqe[10]: SEND s-e [rqpn 0x17e9b rlid 159] [va 0x2ae5fbdd6c80 len 1034 lkey 0xbac2e]
[b1177:173489:0:173489] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173489:0:173489] ib_mlx5_log.c:139  DCI QP 0xc903 wqe[42]: SEND s-e [rqpn 0x18323 rlid 159] [va 0x2b314fdd8d00 len 1034 lkey 0xc3568]
[b1177:173494:0:173494] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173494:0:173494] ib_mlx5_log.c:139  DCI QP 0xc8fb wqe[42]: SEND s-e [rqpn 0x17ccc rlid 159] [va 0x2aed073d8d00 len 1034 lkey 0x11b878]
[b1177:173411:0:173411] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173411:0:173411] ib_mlx5_log.c:139  DCI QP 0xc8d8 wqe[42]: SEND s-e [rqpn 0x180a6 rlid 159] [va 0x2b3e6f9d8d00 len 1034 lkey 0xc607a]
[b1177:173425:0:173425] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173425:0:173425] ib_mlx5_log.c:139  DCI QP 0xc900 wqe[42]: SEND s-e [rqpn 0x17cce rlid 159] [va 0x2aeb9fbd8d00 len 1034 lkey 0xfe9e4]
[b1177:173484:0:173484] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173484:0:173484] ib_mlx5_log.c:139  DCI QP 0xc8c4 wqe[19]: SEND s-e [rqpn 0x182ae rlid 159] [va 0x2b142a5c4800 len 1034 lkey 0x120712]
[b1177:173424:0:173424] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173424:0:173424] ib_mlx5_log.c:139  DCI QP 0xc88c wqe[19]: SEND s-e [rqpn 0x18025 rlid 159] [va 0x2ba2c6bc4800 len 1034 lkey 0x110785]
[b1177:173427:0:173427] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173427:0:173427] ib_mlx5_log.c:139  DCI QP 0xc884 wqe[42]: SEND s-e [rqpn 0x17d4d rlid 159] [va 0x2ae127dd8d00 len 1034 lkey 0x119cfb]
[b1177:173471:0:173471] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173471:0:173471] ib_mlx5_log.c:139  DCI QP 0xc87f wqe[26]: SEND s-e [rqpn 0x1833e rlid 159] [va 0x2acfe11c4800 len 1034 lkey 0xc0d56]
[b1177:173423:0:173423] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173423:0:173423] ib_mlx5_log.c:139  DCI QP 0xc8b8 wqe[3]: SEND s-e [rqpn 0x17d1c rlid 159] [va 0x2af3ad7d8d00 len 1034 lkey 0x1118a7]
[b1177:173466:0:173466] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173466:0:173466] ib_mlx5_log.c:139  DCI QP 0xc86b wqe[36]: SEND s-e [rqpn 0x181f4 rlid 159] [va 0x2adde63d8d00 len 1034 lkey 0xb05c5]
[b1177:173495:0:173495] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173495:0:173495] ib_mlx5_log.c:139  DCI QP 0xc871 wqe[36]: SEND s-e [rqpn 0x18332 rlid 159] [va 0x2ab7a8dd8d00 len 1034 lkey 0xceaa4]
[b1177:173503:0:173503] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173503:0:173503] ib_mlx5_log.c:139  DCI QP 0xc87d wqe[42]: SEND s-e [rqpn 0x1829f rlid 159] [va 0x2b7be6dd8d00 len 1034 lkey 0x112930]
[b1177:173421:0:173421] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173421:0:173421] ib_mlx5_log.c:139  DCI QP 0xc848 wqe[3]: SEND s-e [rqpn 0x17d58 rlid 159] [va 0x2af46bdd8d00 len 1034 lkey 0x11c494]
[b1177:173416:0:173416] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173416:0:173416] ib_mlx5_log.c:139  DCI QP 0xc83a wqe[39]: SEND s-e [rqpn 0x18023 rlid 159] [va 0x2b625fbd8d00 len 1034 lkey 0x91270]
[b1177:173419:0:173419] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173419:0:173419] ib_mlx5_log.c:139  DCI QP 0xc853 wqe[42]: SEND s-e [rqpn 0x17cfd rlid 159] [va 0x2b7bb87d8d00 len 1034 lkey 0x1267ab]
[b1177:173511:0:173511] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173511:0:173511] ib_mlx5_log.c:139  DCI QP 0xc81c wqe[42]: SEND s-e [rqpn 0x18181 rlid 159] [va 0x2b8d9e9d8d00 len 1034 lkey 0xbb42f]
[b1177:173457:0:173457] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173457:0:173457] ib_mlx5_log.c:139  DCI QP 0xc815 wqe[41]: SEND s-e [rqpn 0x1807b rlid 159] [va 0x2b13f93d8d00 len 1034 lkey 0x121c5c]
[b1177:173485:0:173485] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173485:0:173485] ib_mlx5_log.c:139  DCI QP 0xc7f2 wqe[42]: SEND s-e [rqpn 0x18259 rlid 159] [va 0x2ba0927d8d00 len 1034 lkey 0x11fc41]
[b1177:173408:0:173408] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173408:0:173408] ib_mlx5_log.c:139  DCI QP 0xc7e0 wqe[36]: SEND s-e [rqpn 0x17d44 rlid 159] [va 0x2b032e5d8d00 len 1034 lkey 0x100902]
[b1177:173410:0:173410] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173410:0:173410] ib_mlx5_log.c:139  DCI QP 0xc7d4 wqe[9]: SEND s-e [rqpn 0x18067 rlid 159] [va 0x2ae82b7d8d00 len 1034 lkey 0x11c393]
[b1177:173401:0:173401] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173401:0:173401] ib_mlx5_log.c:139  DCI QP 0xc784 wqe[42]: SEND s-e [rqpn 0x17fa2 rlid 159] [va 0x2b15e1dd8d00 len 1034 lkey 0xa954e]
[b1177:173403:0:173403] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173403:0:173403] ib_mlx5_log.c:139  DCI QP 0xc7cb wqe[36]: SEND s-e [rqpn 0x17d24 rlid 159] [va 0x2b8324bd8d00 len 1034 lkey 0x6e0a2]
[b1177:173414:0:173414] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173414:0:173414] ib_mlx5_log.c:139  DCI QP 0xc7cf wqe[9]: SEND s-e [rqpn 0x17fbe rlid 159] [va 0x2aac8ffd8d00 len 1034 lkey 0x123b69]
[b1177:173426:0:173426] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173426:0:173426] ib_mlx5_log.c:139  DCI QP 0xc78b wqe[42]: SEND s-e [rqpn 0x17ced rlid 159] [va 0x2b2a30dd8d00 len 1034 lkey 0xa6d24]
[b1177:173433:0:173433] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173433:0:173433] ib_mlx5_log.c:139  DCI QP 0xc7ad wqe[42]: SEND s-e [rqpn 0x17d08 rlid 159] [va 0x2b65a13d8d00 len 1034 lkey 0x11f43f]
[b1177:173455:0:173455] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173455:0:173455] ib_mlx5_log.c:139  DCI QP 0xc776 wqe[9]: SEND s-e [rqpn 0x17cbd rlid 159] [va 0x2b2dc7fd8d00 len 1034 lkey 0xb4bed]
[b1177:173397:0:173397] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173397:0:173397] ib_mlx5_log.c:139  DCI QP 0xc76d wqe[36]: SEND s-e [rqpn 0x17d27 rlid 159] [va 0x2b0a461d8d00 len 1034 lkey 0x84023]
[b1177:173412:0:173412] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173412:0:173412] ib_mlx5_log.c:139  DCI QP 0xc739 wqe[36]: SEND s-e [rqpn 0x17d48 rlid 159] [va 0x2b634d7d8d00 len 1034 lkey 0xc05cf]
[b1177:173439:0:173439] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173439:0:173439] ib_mlx5_log.c:139  DCI QP 0xc756 wqe[42]: SEND s-e [rqpn 0x17d38 rlid 159] [va 0x2b00235d8d00 len 1034 lkey 0xbdc41]
[b1177:173444:0:173444] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173444:0:173444] ib_mlx5_log.c:139  DCI QP 0xc746 wqe[36]: SEND s-e [rqpn 0x17ccb rlid 159] [va 0x2b5793dd8d00 len 1034 lkey 0x125a9d]
[b1177:173447:0:173447] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173447:0:173447] ib_mlx5_log.c:139  DCI QP 0xc74c wqe[36]: SEND s-e [rqpn 0x17cb7 rlid 159] [va 0x2aca5d1d8d00 len 1034 lkey 0x117dd0]
[b1177:173413:0:173413] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173413:0:173413] ib_mlx5_log.c:139  DCI QP 0xc713 wqe[42]: SEND s-e [rqpn 0x17d1f rlid 159] [va 0x2b9d7ebd8d00 len 1034 lkey 0x125ca7]
[b1177:173428:0:173428] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173428:0:173428] ib_mlx5_log.c:139  DCI QP 0xc720 wqe[42]: SEND s-e [rqpn 0x17d53 rlid 159] [va 0x2b0a717d8d00 len 1034 lkey 0x120d17]
[b1177:173432:0:173432] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173432:0:173432] ib_mlx5_log.c:139  DCI QP 0xc6fa wqe[25]: SEND s-e [rqpn 0x17d34 rlid 159] [va 0x2b852d3ba580 len 1034 lkey 0x103433]
[b1177:173398:0:173398] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173398:0:173398] ib_mlx5_log.c:139  DCI QP 0xc6cb wqe[3]: SEND s-e [rqpn 0x17d5e rlid 159] [va 0x2b727cbd8d00 len 1034 lkey 0xab462]
[b1177:173422:0:173422] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173422:0:173422] ib_mlx5_log.c:139  DCI QP 0xc6e5 wqe[19]: SEND s-e [rqpn 0x17f64 rlid 159] [va 0x2b2f7f7c8900 len 1034 lkey 0xad181]
[b1177:173430:0:173430] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173430:0:173430] ib_mlx5_log.c:139  DCI QP 0xc6e9 wqe[3]: SEND s-e [rqpn 0x17d16 rlid 159] [va 0x2af22d9d8d00 len 1034 lkey 0xba026]
[b1177:173415:0:173415] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173415:0:173415] ib_mlx5_log.c:139  DCI QP 0xc6b3 wqe[36]: SEND s-e [rqpn 0x181a3 rlid 159] [va 0x2ada445d8d00 len 1034 lkey 0x113241]
[b1177:173458:0:173458] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173458:0:173458] ib_mlx5_log.c:139  DCI QP 0xc6ba wqe[3]: SEND s-e [rqpn 0x17cb0 rlid 159] [va 0x2b9a42bd8d00 len 1034 lkey 0xfea68]
[b1177:173409:0:173409] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173409:0:173409] ib_mlx5_log.c:139  DCI QP 0xc67f wqe[9]: SEND s-e [rqpn 0x17fc7 rlid 159] [va 0x2b7b471d8d00 len 1034 lkey 0xfdd4b]
[b1177:173418:0:173418] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173418:0:173418] ib_mlx5_log.c:139  DCI QP 0xc68f wqe[33]: SEND s-e [rqpn 0x17d54 rlid 159] [va 0x2b2536fd8d00 len 1034 lkey 0xbf24e]
[b1177:173396:0:173396] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173396:0:173396] ib_mlx5_log.c:139  DCI QP 0xc66a wqe[42]: SEND s-e [rqpn 0x17f34 rlid 159] [va 0x2b05699d8d00 len 1034 lkey 0x10d21d]
[b1177:173406:0:173406] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173406:0:173406] ib_mlx5_log.c:139  DCI QP 0xc69b wqe[42]: SEND s-e [rqpn 0x17d5b rlid 159] [va 0x2ae8f33d8d00 len 1034 lkey 0x60b56]
[b1177:173434:0:173434] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173434:0:173434] ib_mlx5_log.c:139  DCI QP 0xc65d wqe[9]: SEND s-e [rqpn 0x17d2f rlid 159] [va 0x2ac478dd8d00 len 1034 lkey 0xfe766]
[b1177:173442:0:173442] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173442:0:173442] ib_mlx5_log.c:139  DCI QP 0xc638 wqe[36]: SEND s-e [rqpn 0x17cd6 rlid 159] [va 0x2ace1afd8d00 len 1034 lkey 0xb48eb]
[b1177:173449:0:173449] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173449:0:173449] ib_mlx5_log.c:139  DCI QP 0xc64f wqe[25]: SEND s-e [rqpn 0x17cc3 rlid 159] [va 0x2b98e07c2780 len 1034 lkey 0x10baf0]
[b1177:173448:0:173448] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173448:0:173448] ib_mlx5_log.c:139  DCI QP 0xc627 wqe[25]: SEND s-e [rqpn 0x17cc1 rlid 159] [va 0x2aacfe5ba580 len 1034 lkey 0x10db2d]
[b1177:173405:0:173405] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173405:0:173405] ib_mlx5_log.c:139  DCI QP 0xc5f1 wqe[26]: SEND s-e [rqpn 0x17f3e rlid 159] [va 0x2b51193c4800 len 1034 lkey 0xafdc2]
[b1177:173420:0:173420] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173420:0:173420] ib_mlx5_log.c:139  DCI QP 0xc609 wqe[26]: SEND s-e [rqpn 0x17cf6 rlid 159] [va 0x2abe27dc4800 len 1034 lkey 0xc536f]
[b1177:173429:0:173429] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173429:0:173429] ib_mlx5_log.c:139  DCI QP 0xc5f2 wqe[36]: SEND s-e [rqpn 0x17d50 rlid 159] [va 0x2abf303d8d00 len 1034 lkey 0x11f2e7]
[b1177:173453:0:173453] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173453:0:173453] ib_mlx5_log.c:139  DCI QP 0xc626 wqe[36]: SEND s-e [rqpn 0x17cf2 rlid 159] [va 0x2b0faf7d8d00 len 1034 lkey 0xaca76]
[b1177:173437:0:173437] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173437:0:173437] ib_mlx5_log.c:139  DCI QP 0xc5d8 wqe[26]: SEND s-e [rqpn 0x17d1e rlid 159] [va 0x2ac3719c4800 len 1034 lkey 0x100600]
[b1177:173407:0:173407] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173407:0:173407] ib_mlx5_log.c:139  DCI QP 0xc5b3 wqe[26]: SEND s-e [rqpn 0x17d2b rlid 159] [va 0x2b480f1c4800 len 1034 lkey 0x10d522]
[b1177:173445:0:173445] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173445:0:173445] ib_mlx5_log.c:139  DCI QP 0xc5b4 wqe[36]: SEND s-e [rqpn 0x17cd4 rlid 159] [va 0x2b03293d8d00 len 1034 lkey 0x120b4d]
[b1177:173454:0:173454] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173454:0:173454] ib_mlx5_log.c:139  DCI QP 0xc5d2 wqe[25]: SEND s-e [rqpn 0x17f84 rlid 159] [va 0x2ae5373c2780 len 1034 lkey 0xa651e]
[b1177:173431:0:173431] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173431:0:173431] ib_mlx5_log.c:139  DCI QP 0xc56f wqe[9]: SEND s-e [rqpn 0x17d57 rlid 159] [va 0x2b87de3d8d00 len 1034 lkey 0x127be2]
[b1177:173440:0:173440] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173440:0:173440] ib_mlx5_log.c:139  DCI QP 0xc57e wqe[36]: SEND s-e [rqpn 0x17d21 rlid 159] [va 0x2b3e53bd8d00 len 1034 lkey 0xbefbd]
[b1177:173452:0:173452] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173452:0:173452] ib_mlx5_log.c:139  DCI QP 0xc5b2 wqe[19]: SEND s-e [rqpn 0x17f3a rlid 159] [va 0x2b11883c4800 len 1034 lkey 0x74ee7]
[b1177:173459:0:173459] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173459:0:173459] ib_mlx5_log.c:139  DCI QP 0xc55a wqe[26]: SEND s-e [rqpn 0x17ce4 rlid 159] [va 0x2ac0df1c4800 len 1034 lkey 0x11b979]
[b1177:173436:0:173436] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173436:0:173436] ib_mlx5_log.c:139  DCI QP 0xc559 wqe[42]: SEND s-e [rqpn 0x17d02 rlid 159] [va 0x2ae7439d8d00 len 1034 lkey 0x127ec6]
[b1177:173404:0:173404] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173404:0:173404] ib_mlx5_log.c:139  DCI QP 0xc53e wqe[26]: SEND s-e [rqpn 0x17f54 rlid 159] [va 0x2b24883c4800 len 1034 lkey 0xe8eed]
[b1177:173417:0:173417] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173417:0:173417] ib_mlx5_log.c:139  DCI QP 0xc518 wqe[3]: SEND s-e [rqpn 0x17ce7 rlid 159] [va 0x2b966cdd8d00 len 1034 lkey 0x91708]
[b1177:173438:0:173438] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173438:0:173438] ib_mlx5_log.c:139  DCI QP 0xc512 wqe[42]: SEND s-e [rqpn 0x1804a rlid 159] [va 0x2b7f80dd8d00 len 1034 lkey 0x100ba9]
[b1177:173441:0:173441] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173441:0:173441] ib_mlx5_log.c:139  DCI QP 0xc50b wqe[36]: SEND s-e [rqpn 0x17ce0 rlid 159] [va 0x2b5c027d8d00 len 1034 lkey 0xff684]
[b1177:173446:0:173446] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173446:0:173446] ib_mlx5_log.c:139  DCI QP 0xc514 wqe[13]: SEND s-e [rqpn 0x17cd0 rlid 159] [va 0x2b25c59be680 len 1034 lkey 0xaa18c]
[b1177:173450:0:173450] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173450:0:173450] ib_mlx5_log.c:139  DCI QP 0xc4ff wqe[13]: SEND s-e [rqpn 0x17d1b rlid 159] [va 0x2ac3f27ba580 len 1034 lkey 0xffd8b]
[b1177:173443:0:173443] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173443:0:173443] ib_mlx5_log.c:139  DCI QP 0xc4f9 wqe[42]: SEND s-e [rqpn 0x17d3d rlid 159] [va 0x2b8d9bdd8d00 len 1034 lkey 0x11303f]
[b1177:173451:0:173451] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173451:0:173451] ib_mlx5_log.c:139  DCI QP 0xc4fe wqe[8]: SEND s-e [rqpn 0x17cb4 rlid 159] [va 0x2afe4e5b8500 len 1034 lkey 0x12a2a3]
[b1177:173456:0:173456] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173456:0:173456] ib_mlx5_log.c:139  DCI QP 0xc4fc wqe[42]: SEND s-e [rqpn 0x1804d rlid 159] [va 0x2ac51e5d8d00 len 1034 lkey 0x11c510]
[b1177:173483:0:173483] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173483:0:173483] ib_mlx5_log.c:139  DCI QP 0xc9a5 wqe[42]: SEND s-e [rqpn 0x17cc8 rlid 159] [va 0x2b8f88dd8d00 len 1034 lkey 0xaebb9]
[b1177:173399:0:173399] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173399:0:173399] ib_mlx5_log.c:139  DCI QP 0xc9b8 wqe[42]: SEND s-e [rqpn 0x17d0f rlid 159] [va 0x2b82b8fd8d00 len 1034 lkey 0x11f93c]
[b1177:173467:0:173467] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173467:0:173467] ib_mlx5_log.c:139  DCI QP 0xc9a2 wqe[36]: SEND s-e [rqpn 0x182ea rlid 159] [va 0x2abc781d8d00 len 1034 lkey 0x562e6]
[b1177:173486:0:173486] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173486:0:173486] ib_mlx5_log.c:139  DCI QP 0xc983 wqe[42]: SEND s-e [rqpn 0x1830e rlid 159] [va 0x2b42299d8d00 len 1034 lkey 0xfe4e0]
[b1177:173493:0:173493] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173493:0:173493] ib_mlx5_log.c:139  DCI QP 0xc99a wqe[36]: SEND s-e [rqpn 0x1812e rlid 159] [va 0x2ac00c3d8d00 len 1034 lkey 0x11633f]
[b1177:173522:0:173522] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1177:173522:0:173522] ib_mlx5_log.c:139  DCI QP 0xc980 wqe[42]: SEND s-e [rqpn 0x18113 rlid 159] [va 0x2b41b2dd8d00 len 1034 lkey 0x43f5a]
[b1155:150662:0:150662] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150662:0:150662] ib_mlx5_log.c:139  DCI QP 0xc3a9 wqe[1]: SEND s-e [rqpn 0x17ccb rlid 159] [va 0x2b6d789d4c00 len 2058 lkey 0x88162]
[b1155:150614:0:150614] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150614:0:150614] ib_mlx5_log.c:139  DCI QP 0xc39b wqe[1]: SEND s-e [rqpn 0x17f34 rlid 159] [va 0x2acb97dd4c00 len 2058 lkey 0xae1d2]
[b1155:150628:0:150628] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150628:0:150628] ib_mlx5_log.c:139  DCI QP 0xc3b7 wqe[1]: SEND s-e [rqpn 0x18067 rlid 159] [va 0x2ae4b5ff5400 len 2058 lkey 0x9f984]
[b1155:150663:0:150663] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150663:0:150663] ib_mlx5_log.c:139  DCI QP 0xc386 wqe[14]: SEND s-e [rqpn 0x17cd4 rlid 159] [va 0x2b59b55a1f80 len 2058 lkey 0xb232f]
[b1155:150622:0:150622] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150622:0:150622] ib_mlx5_log.c:139  DCI QP 0xc3d6 wqe[1]: SEND s-e [rqpn 0x17f54 rlid 159] [va 0x2b0cf89f5400 len 2058 lkey 0x7fb28]
[b1155:150623:0:150623] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150623:0:150623] ib_mlx5_log.c:139  DCI QP 0xc38d wqe[44]: SEND s-e [rqpn 0x17f3e rlid 159] [va 0x2ae116df5400 len 2058 lkey 0xafa03]
[b1155:150624:0:150624] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150624:0:150624] ib_mlx5_log.c:139  DCI QP 0xc379 wqe[45]: SEND s-e [rqpn 0x17d5b rlid 159] [va 0x2b017e5f5400 len 2058 lkey 0xaeae3]
[b1155:150637:0:150637] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150637:0:150637] ib_mlx5_log.c:139  DCI QP 0xc388 wqe[14]: SEND s-e [rqpn 0x17cfd rlid 159] [va 0x2b9f243f5400 len 2058 lkey 0xacfb1]
[b1155:150651:0:150651] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150651:0:150651] ib_mlx5_log.c:139  DCI QP 0xc394 wqe[2]: SEND s-e [rqpn 0x17d08 rlid 159] [va 0x2ae2c5fd4c00 len 2058 lkey 0x832d2]
[b1155:150630:0:150630] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150630:0:150630] ib_mlx5_log.c:139  DCI QP 0xc37f wqe[20]: SEND s-e [rqpn 0x17d48 rlid 159] [va 0x2b1fadfa4000 len 2058 lkey 0x788ae]
[b1155:150634:0:150634] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150634:0:150634] ib_mlx5_log.c:139  DCI QP 0xc33d wqe[14]: SEND s-e [rqpn 0x18023 rlid 159] [va 0x2b002ffa8100 len 2058 lkey 0x2e4545]
[b1155:150644:0:150644] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150644:0:150644] ib_mlx5_log.c:139  DCI QP 0xc316 wqe[14]: SEND s-e [rqpn 0x17ced rlid 159] [va 0x2ae7eabb2380 len 2058 lkey 0xb222e]
[b1155:150666:0:150666] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150666:0:150666] ib_mlx5_log.c:139  DCI QP 0xc341 wqe[5]: SEND s-e [rqpn 0x17cc1 rlid 159] [va 0x2ae19fbd4c00 len 2058 lkey 0xb0821]
[b1155:150669:0:150669] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150669:0:150669] ib_mlx5_log.c:139  DCI QP 0xc339 wqe[17]: SEND s-e [rqpn 0x17cb4 rlid 159] [va 0x2b60b0df5400 len 2058 lkey 0x9f281]
[b1155:150629:0:150629] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150629:0:150629] ib_mlx5_log.c:139  DCI QP 0xc30b wqe[2]: SEND s-e [rqpn 0x180a6 rlid 159] [va 0x2abd497d4c00 len 2058 lkey 0xaf3f4]
[b1155:150631:0:150631] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150631:0:150631] ib_mlx5_log.c:139  DCI QP 0xc2f7 wqe[29]: SEND s-e [rqpn 0x17d1f rlid 159] [va 0x2b78e9ba1f80 len 2058 lkey 0x251501]
[b1155:150636:0:150636] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150636:0:150636] ib_mlx5_log.c:139  DCI QP 0xc2f6 wqe[57]: SEND s-e [rqpn 0x17d54 rlid 159] [va 0x2b60791d4c00 len 2058 lkey 0x2f749c]
[b1155:150615:0:150615] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150615:0:150615] ib_mlx5_log.c:139  DCI QP 0xc2c0 wqe[57]: SEND s-e [rqpn 0x17d27 rlid 159] [va 0x2b290bdd4c00 len 2058 lkey 0xa9294]
[b1155:150626:0:150626] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150626:0:150626] ib_mlx5_log.c:139  DCI QP 0xc2b7 wqe[54]: SEND s-e [rqpn 0x17d44 rlid 159] [va 0x2af7cd3d2b80 len 2058 lkey 0x9be73]
[b1155:150633:0:150633] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150633:0:150633] ib_mlx5_log.c:139  DCI QP 0xc2d7 wqe[29]: SEND s-e [rqpn 0x181a3 rlid 159] [va 0x2aed501a1f80 len 2058 lkey 0xe0892]
[b1155:150653:0:150653] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150653:0:150653] ib_mlx5_log.c:139  DCI QP 0xc2da wqe[17]: SEND s-e [rqpn 0x17e9b rlid 159] [va 0x2b466a5f5400 len 2058 lkey 0xb1030]
[b1155:150627:0:150627] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150627:0:150627] ib_mlx5_log.c:139  DCI QP 0xc297 wqe[5]: SEND s-e [rqpn 0x17fc7 rlid 159] [va 0x2b58131d4c00 len 2058 lkey 0x842d6]
[b1155:150632:0:150632] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150632:0:150632] ib_mlx5_log.c:139  DCI QP 0xc2b2 wqe[17]: SEND s-e [rqpn 0x17fbe rlid 159] [va 0x2ab0ccfc0700 len 2058 lkey 0x25211d]
[b1155:150635:0:150635] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150635:0:150635] ib_mlx5_log.c:139  DCI QP 0xc24f wqe[17]: SEND s-e [rqpn 0x17ce7 rlid 159] [va 0x2b0df83f5400 len 2058 lkey 0x9d8fa]
[b1155:150638:0:150638] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150638:0:150638] ib_mlx5_log.c:139  DCI QP 0xc2e0 wqe[29]: SEND s-e [rqpn 0x17cf6 rlid 159] [va 0x2b67535b6480 len 2058 lkey 0xad3b5]
[b1155:150650:0:150650] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150650:0:150650] ib_mlx5_log.c:139  DCI QP 0xc28a wqe[1]: SEND s-e [rqpn 0x17d34 rlid 159] [va 0x2ace959d4c00 len 2058 lkey 0xadecf]
[b1155:150660:0:150660] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150660:0:150660] ib_mlx5_log.c:139  DCI QP 0xc2b5 wqe[17]: SEND s-e [rqpn 0x17cd6 rlid 159] [va 0x2af8a73f5400 len 2058 lkey 0x88465]
[b1155:150643:0:150643] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150643:0:150643] ib_mlx5_log.c:139  DCI QP 0xc247 wqe[17]: SEND s-e [rqpn 0x17cce rlid 159] [va 0x2b9d3b1a8100 len 2058 lkey 0xb1b1f]
[b1155:150645:0:150645] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150645:0:150645] ib_mlx5_log.c:139  DCI QP 0xc249 wqe[17]: SEND s-e [rqpn 0x17d4d rlid 159] [va 0x2adbe1dcca00 len 2058 lkey 0xad9c2]
[b1155:150656:0:150656] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150656:0:150656] ib_mlx5_log.c:139  DCI QP 0xc24d wqe[14]: SEND s-e [rqpn 0x1804a rlid 159] [va 0x2b6cd89a8100 len 2058 lkey 0xaf801]
[b1155:150674:0:150674] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150674:0:150674] ib_mlx5_log.c:139  DCI QP 0xc23e wqe[29]: SEND s-e [rqpn 0x1804d rlid 159] [va 0x2adf1f1ca980 len 2058 lkey 0xae0d1]
[b1155:150642:0:150642] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150642:0:150642] ib_mlx5_log.c:139  DCI QP 0xc20b wqe[2]: SEND s-e [rqpn 0x18025 rlid 159] [va 0x2b638ddd4c00 len 2058 lkey 0xb1434]
[b1155:150618:0:150618] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150618:0:150618] ib_mlx5_log.c:139  DCI QP 0xc20d wqe[5]: SEND s-e [rqpn 0x17cf8 rlid 159] [va 0x2b02837d4c00 len 2058 lkey 0xac9a3]
[b1155:150620:0:150620] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150620:0:150620] ib_mlx5_log.c:139  DCI QP 0xc1ff wqe[54]: SEND s-e [rqpn 0x18004 rlid 159] [va 0x2b915efd4c00 len 2058 lkey 0xb1131]
[b1155:150648:0:150648] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150648:0:150648] ib_mlx5_log.c:139  DCI QP 0xc205 wqe[14]: SEND s-e [rqpn 0x17d16 rlid 159] [va 0x2ab8be1f5400 len 2058 lkey 0xb1d21]
[b1155:150667:0:150667] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150667:0:150667] ib_mlx5_log.c:139  DCI QP 0xc1cc wqe[30]: SEND s-e [rqpn 0x17cc3 rlid 159] [va 0x2b151abc4800 len 2058 lkey 0xb0213]
[b1155:150619:0:150619] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150619:0:150619] ib_mlx5_log.c:139  DCI QP 0xc1b0 wqe[2]: SEND s-e [rqpn 0x17fa2 rlid 159] [va 0x2add459d4c00 len 2058 lkey 0xaebe4]
[b1155:150625:0:150625] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150625:0:150625] ib_mlx5_log.c:139  DCI QP 0xc1a3 wqe[1]: SEND s-e [rqpn 0x17d2b rlid 159] [va 0x2b0e70dd0b00 len 2058 lkey 0xac6a0]
[b1155:150654:0:150654] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150654:0:150654] ib_mlx5_log.c:139  DCI QP 0xc1b1 wqe[42]: SEND s-e [rqpn 0x17d02 rlid 159] [va 0x2b1c457f5400 len 2058 lkey 0x9b371]
[b1155:150655:0:150655] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150655:0:150655] ib_mlx5_log.c:139  DCI QP 0xc15c wqe[54]: SEND s-e [rqpn 0x17d1e rlid 159] [va 0x2b3c6d7d4c00 len 2058 lkey 0xb0a22]
[b1155:150668:0:150668] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150668:0:150668] ib_mlx5_log.c:139  DCI QP 0xc176 wqe[5]: SEND s-e [rqpn 0x17d1b rlid 159] [va 0x2b1396fd4c00 len 2058 lkey 0x9e57f]
[b1155:150670:0:150670] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150670:0:150670] ib_mlx5_log.c:139  DCI QP 0xc150 wqe[26]: SEND s-e [rqpn 0x17f3a rlid 159] [va 0x2ae32e5a1f80 len 2058 lkey 0x1342fc]
[b1155:150673:0:150673] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150673:0:150673] ib_mlx5_log.c:139  DCI QP 0xc1d4 wqe[17]: SEND s-e [rqpn 0x17cbd rlid 159] [va 0x2b4f493a1f80 len 2058 lkey 0xe6922]
[b1155:150675:0:150675] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150675:0:150675] ib_mlx5_log.c:139  DCI QP 0xc172 wqe[17]: SEND s-e [rqpn 0x1807b rlid 159] [va 0x2b7fa47a8100 len 2058 lkey 0xb2732]
[b1155:150616:0:150616] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150616:0:150616] ib_mlx5_log.c:139  DCI QP 0xc119 wqe[2]: SEND s-e [rqpn 0x17d5e rlid 159] [va 0x2b780a1d4c00 len 2058 lkey 0xadcc5]
[b1155:150617:0:150617] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150617:0:150617] ib_mlx5_log.c:139  DCI QP 0xc118 wqe[30]: SEND s-e [rqpn 0x17d0f rlid 159] [va 0x2af4e51b6480 len 2058 lkey 0xb1f23]
[b1155:150621:0:150621] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150621:0:150621] ib_mlx5_log.c:139  DCI QP 0xc117 wqe[42]: SEND s-e [rqpn 0x17d24 rlid 159] [va 0x2aedbd9f7480 len 2058 lkey 0xaf5fe]
[b1155:150649:0:150649] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150649:0:150649] ib_mlx5_log.c:139  DCI QP 0xc128 wqe[15]: SEND s-e [rqpn 0x17d57 rlid 159] [va 0x2aacf41f5400 len 2058 lkey 0xb1e22]
[b1155:150661:0:150661] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150661:0:150661] ib_mlx5_log.c:139  DCI QP 0xc129 wqe[2]: SEND s-e [rqpn 0x17d3d rlid 159] [va 0x2b7d621d4c00 len 2058 lkey 0xad7c0]
[b1155:150664:0:150664] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150664:0:150664] ib_mlx5_log.c:139  DCI QP 0xc10c wqe[42]: SEND s-e [rqpn 0x17cd0 rlid 159] [va 0x2b77a3bf5400 len 2058 lkey 0xb293c]
[b1155:150658:0:150658] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150658:0:150658] ib_mlx5_log.c:139  DCI QP 0xc0d9 wqe[2]: SEND s-e [rqpn 0x17d21 rlid 159] [va 0x2b4c2f1d4c00 len 2058 lkey 0x8163c]
[b1155:150676:0:150676] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150676:0:150676] ib_mlx5_log.c:139  DCI QP 0xc0f9 wqe[29]: SEND s-e [rqpn 0x17cb0 rlid 159] [va 0x2b98675b0300 len 2058 lkey 0xaf700]
[b1155:150640:0:150640] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150640:0:150640] ib_mlx5_log.c:139  DCI QP 0xc091 wqe[29]: SEND s-e [rqpn 0x17f64 rlid 159] [va 0x2af1429a6080 len 2058 lkey 0x6bf303]
[b1155:150683:0:150683] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150683:0:150683] ib_mlx5_log.c:139  DCI QP 0xc0ce wqe[14]: SEND s-e [rqpn 0x182e8 rlid 159] [va 0x2b48cdbf5400 len 2058 lkey 0x87c55]
[b1155:150691:0:150691] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150691:0:150691] ib_mlx5_log.c:139  DCI QP 0xc0be wqe[44]: SEND s-e [rqpn 0x182c2 rlid 159] [va 0x2b2357dd4c00 len 2058 lkey 0xae2d3]
[b1155:150704:0:150704] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150704:0:150704] ib_mlx5_log.c:139  DCI QP 0xc0a1 wqe[17]: SEND s-e [rqpn 0x18259 rlid 159] [va 0x2ae19a9a1f80 len 2058 lkey 0x133aeb]
[b1155:150705:0:150705] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150705:0:150705] ib_mlx5_log.c:139  DCI QP 0xc094 wqe[29]: SEND s-e [rqpn 0x1830e rlid 159] [va 0x2b67847a1f80 len 2058 lkey 0x86834]
[b1155:150641:0:150641] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150641:0:150641] ib_mlx5_log.c:139  DCI QP 0xc064 wqe[29]: SEND s-e [rqpn 0x17d1c rlid 159] [va 0x2b17eafa6080 len 2058 lkey 0x9b6599]
[b1155:150652:0:150652] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150652:0:150652] ib_mlx5_log.c:139  DCI QP 0xc03b wqe[1]: SEND s-e [rqpn 0x17d2f rlid 159] [va 0x2b923f7d4c00 len 2058 lkey 0xae6df]
[b1155:150657:0:150657] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150657:0:150657] ib_mlx5_log.c:139  DCI QP 0xc00a wqe[2]: SEND s-e [rqpn 0x17d38 rlid 159] [va 0x2ad5b4dd2b80 len 2058 lkey 0xaf2f3]
[b1155:150677:0:150677] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150677:0:150677] ib_mlx5_log.c:139  DCI QP 0xc058 wqe[17]: SEND s-e [rqpn 0x17ce4 rlid 159] [va 0x2ab0421c4800 len 2058 lkey 0xb191d]
[b1155:150682:0:150682] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150682:0:150682] ib_mlx5_log.c:139  DCI QP 0xc014 wqe[1]: SEND s-e [rqpn 0x18235 rlid 159] [va 0x2b05babf5400 len 2058 lkey 0x9f180]
[b1155:150703:0:150703] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150703:0:150703] ib_mlx5_log.c:139  DCI QP 0xc062 wqe[17]: SEND s-e [rqpn 0x182ae rlid 159] [va 0x2b086f9a1f80 len 2058 lkey 0x8b46fc]
[b1155:150706:0:150706] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150706:0:150706] ib_mlx5_log.c:139  DCI QP 0xc065 wqe[29]: SEND s-e [rqpn 0x18249 rlid 159] [va 0x2ae266dbc600 len 2058 lkey 0xa5390]
[b1155:150647:0:150647] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150647:0:150647] ib_mlx5_log.c:139  DCI QP 0xbffa wqe[17]: SEND s-e [rqpn 0x17d50 rlid 159] [va 0x2ab9d83a6080 len 2058 lkey 0xac49e]
[b1155:150701:0:150701] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150701:0:150701] ib_mlx5_log.c:139  DCI QP 0xbfff wqe[1]: SEND s-e [rqpn 0x18337 rlid 159] [va 0x2b07e15d2b80 len 2058 lkey 0xb0c24]
[b1155:150708:0:150708] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150708:0:150708] ib_mlx5_log.c:139  DCI QP 0xc02d wqe[17]: SEND s-e [rqpn 0x18323 rlid 159] [va 0x2b773a7a8100 len 2058 lkey 0x130b8e]
[b1155:150693:0:150693] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150693:0:150693] ib_mlx5_log.c:139  DCI QP 0xbfdd wqe[14]: SEND s-e [rqpn 0x17cbf rlid 159] [va 0x2b6fcd1f7480 len 2058 lkey 0x87a53]
[b1155:150646:0:150646] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150646:0:150646] ib_mlx5_log.c:139  DCI QP 0xbfb9 wqe[45]: SEND s-e [rqpn 0x17d53 rlid 159] [va 0x2b4103bf7480 len 2058 lkey 0xb0f2f]
[b1155:150671:0:150671] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150671:0:150671] ib_mlx5_log.c:139  DCI QP 0xbf7f wqe[29]: SEND s-e [rqpn 0x17cf2 rlid 159] [va 0x2b357919ff00 len 2058 lkey 0xb2b3e]
[b1155:150680:0:150680] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150680:0:150680] ib_mlx5_log.c:139  DCI QP 0xbf9e wqe[45]: SEND s-e [rqpn 0x17fa7 rlid 159] [va 0x2b7fa37f7480 len 2058 lkey 0xaff10]
[b1155:150686:0:150686] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150686:0:150686] ib_mlx5_log.c:139  DCI QP 0xbfdc wqe[44]: SEND s-e [rqpn 0x181ce rlid 159] [va 0x2b811a9f7480 len 2058 lkey 0x9c675]
[b1155:150727:0:150727] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150727:0:150727] ib_mlx5_log.c:139  DCI QP 0xbfaf wqe[14]: SEND s-e [rqpn 0x180ed rlid 159] [va 0x2ab9b61f7480 len 2058 lkey 0xb051e]
[b1155:150721:0:150721] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150721:0:150721] ib_mlx5_log.c:139  DCI QP 0xbf72 wqe[14]: SEND s-e [rqpn 0x17cd2 rlid 159] [va 0x2abb9ffa6080 len 2058 lkey 0x9a56f]
[b1155:150685:0:150685] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150685:0:150685] ib_mlx5_log.c:139  DCI QP 0xbf6c wqe[15]: SEND s-e [rqpn 0x182ea rlid 159] [va 0x2b9dfe5f7480 len 2058 lkey 0xad2b4]
[b1155:150702:0:150702] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150702:0:150702] ib_mlx5_log.c:139  DCI QP 0xbf4a wqe[17]: SEND s-e [rqpn 0x17cc8 rlid 159] [va 0x2b5c6e5c2780 len 2058 lkey 0xeeffe]
[b1155:150710:0:150710] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150710:0:150710] ib_mlx5_log.c:139  DCI QP 0xbf22 wqe[14]: SEND s-e [rqpn 0x1822e rlid 159] [va 0x2b86efba6080 len 2058 lkey 0xb516d]
[b1155:150717:0:150717] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150717:0:150717] ib_mlx5_log.c:139  DCI QP 0xbf60 wqe[3]: SEND s-e [rqpn 0x180b5 rlid 159] [va 0x2b9940dd2b80 len 2058 lkey 0xaeff0]
[b1155:150725:0:150725] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150725:0:150725] ib_mlx5_log.c:139  DCI QP 0xbf50 wqe[14]: SEND s-e [rqpn 0x18138 rlid 159] [va 0x2b87f63f7480 len 2058 lkey 0xb1a1e]
[b1155:150694:0:150694] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150694:0:150694] ib_mlx5_log.c:139  DCI QP 0xbf11 wqe[2]: SEND s-e [rqpn 0x17cb9 rlid 159] [va 0x2aea51fd2b80 len 2058 lkey 0xaf0f1]
[b1155:150698:0:150698] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150698:0:150698] ib_mlx5_log.c:139  DCI QP 0xbf13 wqe[29]: SEND s-e [rqpn 0x18045 rlid 159] [va 0x2b1f6899ff00 len 2058 lkey 0x8754f]
[b1155:150681:0:150681] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150681:0:150681] ib_mlx5_log.c:139  DCI QP 0xbe9d wqe[29]: SEND s-e [rqpn 0x181dd rlid 159] [va 0x2aaf051b4400 len 2058 lkey 0x9f683]
[b1155:150690:0:150690] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150690:0:150690] ib_mlx5_log.c:139  DCI QP 0xbe88 wqe[27]: SEND s-e [rqpn 0x1833e rlid 159] [va 0x2ba3cf9b2380 len 2058 lkey 0x88061]
[b1155:150692:0:150692] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150692:0:150692] ib_mlx5_log.c:139  DCI QP 0xbef3 wqe[3]: SEND s-e [rqpn 0x17fb7 rlid 159] [va 0x2b27221d2b80 len 2058 lkey 0xaccae]
[b1155:150695:0:150695] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150695:0:150695] ib_mlx5_log.c:139  DCI QP 0xbf05 wqe[17]: SEND s-e [rqpn 0x18348 rlid 159] [va 0x2b4f3b99ff00 len 2058 lkey 0x12fe76]
[b1155:150699:0:150699] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150699:0:150699] ib_mlx5_log.c:139  DCI QP 0xbed3 wqe[14]: SEND s-e [rqpn 0x1800b rlid 159] [va 0x2ab4ed3a6080 len 2058 lkey 0xad1b3]
[b1155:150700:0:150700] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150700:0:150700] ib_mlx5_log.c:139  DCI QP 0xbeda wqe[17]: SEND s-e [rqpn 0x18115 rlid 159] [va 0x2b475adc2780 len 2058 lkey 0xb2c3f]
[b1155:150728:0:150728] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150728:0:150728] ib_mlx5_log.c:139  DCI QP 0xbec2 wqe[29]: SEND s-e [rqpn 0x1834d rlid 159] [va 0x2b5db899ff00 len 2058 lkey 0x6c338b]
[b1155:150639:0:150639] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150639:0:150639] ib_mlx5_log.c:139  DCI QP 0xbe41 wqe[14]: SEND s-e [rqpn 0x17d58 rlid 159] [va 0x2acbd99a6080 len 2058 lkey 0xb3c53]
[b1155:150659:0:150659] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150659:0:150659] ib_mlx5_log.c:139  DCI QP 0xbe6e wqe[2]: SEND s-e [rqpn 0x17ce0 rlid 159] [va 0x2b916f7d2b80 len 2058 lkey 0xb0e2e]
[b1155:150672:0:150672] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150672:0:150672] ib_mlx5_log.c:139  DCI QP 0xbe3f wqe[14]: SEND s-e [rqpn 0x17f84 rlid 159] [va 0x2b2d3b79ff00 len 2058 lkey 0xdfc7f]
[b1155:150697:0:150697] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150697:0:150697] ib_mlx5_log.c:139  DCI QP 0xbe6f wqe[14]: SEND s-e [rqpn 0x18214 rlid 159] [va 0x2b002c39de80 len 2058 lkey 0x87750]
[b1155:150711:0:150711] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150711:0:150711] ib_mlx5_log.c:139  DCI QP 0xbea5 wqe[14]: SEND s-e [rqpn 0x17cda rlid 159] [va 0x2b4be2bf7480 len 2058 lkey 0x9e27e]
[b1155:150713:0:150713] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150713:0:150713] ib_mlx5_log.c:139  DCI QP 0xbe78 wqe[42]: SEND s-e [rqpn 0x17ccc rlid 159] [va 0x2ae23fdd0b00 len 2058 lkey 0xb0314]
[b1155:150678:0:150678] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150678:0:150678] ib_mlx5_log.c:139  DCI QP 0xbded wqe[14]: SEND s-e [rqpn 0x17f9d rlid 159] [va 0x2b8ff939ff00 len 2058 lkey 0x1338e8]
[b1155:150679:0:150679] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150679:0:150679] ib_mlx5_log.c:139  DCI QP 0xbdfe wqe[14]: SEND s-e [rqpn 0x1831f rlid 159] [va 0x2b86a35fb580 len 2058 lkey 0xb2430]
[b1155:150684:0:150684] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150684:0:150684] ib_mlx5_log.c:139  DCI QP 0xbdc7 wqe[1]: SEND s-e [rqpn 0x181f4 rlid 159] [va 0x2b3fb51d2b80 len 2058 lkey 0xaf4f5]
[b1155:150687:0:150687] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150687:0:150687] ib_mlx5_log.c:139  DCI QP 0xbde9 wqe[15]: SEND s-e [rqpn 0x181ca rlid 159] [va 0x2ad948ff7480 len 2058 lkey 0x87e5f]
[b1155:150712:0:150712] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150712:0:150712] ib_mlx5_log.c:139  DCI QP 0xbe0d wqe[29]: SEND s-e [rqpn 0x1812e rlid 159] [va 0x2afc337a6080 len 2058 lkey 0xaf902]
[b1155:150737:0:150737] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150737:0:150737] ib_mlx5_log.c:139  DCI QP 0xbdc4 wqe[17]: SEND s-e [rqpn 0x181d5 rlid 159] [va 0x2b3f3cb9ff00 len 2058 lkey 0xe9265]
[b1155:150738:0:150738] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150738:0:150738] ib_mlx5_log.c:139  DCI QP 0xbe1f wqe[14]: SEND s-e [rqpn 0x181d2 rlid 159] [va 0x2b122259ff00 len 2058 lkey 0x7d704]
[b1155:150742:0:150742] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150742:0:150742] ib_mlx5_log.c:139  DCI QP 0xbe32 wqe[29]: SEND s-e [rqpn 0x18231 rlid 159] [va 0x2b1ca5da6080 len 2058 lkey 0x8480a]
[b1155:150696:0:150696] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150696:0:150696] ib_mlx5_log.c:139  DCI QP 0xbd45 wqe[29]: SEND s-e [rqpn 0x18119 rlid 159] [va 0x2ac89f79ff00 len 2058 lkey 0x252521]
[b1155:150709:0:150709] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150709:0:150709] ib_mlx5_log.c:139  DCI QP 0xbdb0 wqe[29]: SEND s-e [rqpn 0x1827c rlid 159] [va 0x2afd6939ff00 len 2058 lkey 0x7cffd]
[b1155:150724:0:150724] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150724:0:150724] ib_mlx5_log.c:139  DCI QP 0xbd4f wqe[2]: SEND s-e [rqpn 0x1808f rlid 159] [va 0x2b1fd4dd2b80 len 2058 lkey 0xeced1]
[b1155:150730:0:150730] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150730:0:150730] ib_mlx5_log.c:139  DCI QP 0xbd61 wqe[17]: SEND s-e [rqpn 0x18181 rlid 159] [va 0x2ad19db9ff00 len 2058 lkey 0x84404]
[b1155:150741:0:150741] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150741:0:150741] ib_mlx5_log.c:139  DCI QP 0xbd8a wqe[17]: SEND s-e [rqpn 0x18113 rlid 159] [va 0x2b69b6baa180 len 2058 lkey 0x2512fe]
[b1155:150665:0:150665] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150665:0:150665] ib_mlx5_log.c:139  DCI QP 0xbce7 wqe[29]: SEND s-e [rqpn 0x17cb7 rlid 159] [va 0x2afe9b7aa180 len 2058 lkey 0xb0720]
[b1155:150715:0:150715] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150715:0:150715] ib_mlx5_log.c:139  DCI QP 0xbcd0 wqe[1]: SEND s-e [rqpn 0x17fa5 rlid 159] [va 0x2accb4bd2b80 len 2058 lkey 0x87b54]
[b1155:150716:0:150716] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150716:0:150716] ib_mlx5_log.c:139  DCI QP 0xbcc1 wqe[54]: SEND s-e [rqpn 0x180ea rlid 159] [va 0x2b087dfd2b80 len 2058 lkey 0xb0112]
[b1155:150719:0:150719] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150719:0:150719] ib_mlx5_log.c:139  DCI QP 0xbcb7 wqe[1]: SEND s-e [rqpn 0x18110 rlid 159] [va 0x2b22925d2b80 len 2058 lkey 0x87d5e]
[b1155:150723:0:150723] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150723:0:150723] ib_mlx5_log.c:139  DCI QP 0xbd41 wqe[2]: SEND s-e [rqpn 0x18283 rlid 159] [va 0x2b0e90bd2b80 len 2058 lkey 0x9c074]
[b1155:150740:0:150740] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150740:0:150740] ib_mlx5_log.c:139  DCI QP 0xbccc wqe[29]: SEND s-e [rqpn 0x18210 rlid 159] [va 0x2ac7651a6080 len 2058 lkey 0x88364]
[b1155:150707:0:150707] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150707:0:150707] ib_mlx5_log.c:139  DCI QP 0xbcb1 wqe[2]: SEND s-e [rqpn 0x1827f rlid 159] [va 0x2b66f3bd2b80 len 2058 lkey 0xacdaf]
[b1155:150732:0:150732] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150732:0:150732] ib_mlx5_log.c:139  DCI QP 0xbcc3 wqe[17]: SEND s-e [rqpn 0x18262 rlid 159] [va 0x2acf6ef9ff00 len 2058 lkey 0xedbe3]
[b1155:150735:0:150735] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150735:0:150735] ib_mlx5_log.c:139  DCI QP 0xbcae wqe[17]: SEND s-e [rqpn 0x182ff rlid 159] [va 0x2ae03919ff00 len 2058 lkey 0xe9e80]
[b1155:150736:0:150736] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150736:0:150736] ib_mlx5_log.c:139  DCI QP 0xbca7 wqe[17]: SEND s-e [rqpn 0x18326 rlid 159] [va 0x2b412739ff00 len 2058 lkey 0x86733]
[b1155:150689:0:150689] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150689:0:150689] ib_mlx5_log.c:139  DCI QP 0xbc97 wqe[18]: SEND s-e [rqpn 0x18156 rlid 159] [va 0x2b27f49f7480 len 2058 lkey 0xb061f]
[b1155:150718:0:150718] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150718:0:150718] ib_mlx5_log.c:139  DCI QP 0xbca8 wqe[17]: SEND s-e [rqpn 0x18064 rlid 159] [va 0x2acff4ff7480 len 2058 lkey 0xaeeef]
[b1155:150720:0:150720] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150720:0:150720] ib_mlx5_log.c:139  DCI QP 0xbc96 wqe[3]: SEND s-e [rqpn 0x17fb8 rlid 159] [va 0x2b7b57dd2b80 len 2058 lkey 0xafe0f]
[b1155:150722:0:150722] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150722:0:150722] ib_mlx5_log.c:139  DCI QP 0xbca4 wqe[2]: SEND s-e [rqpn 0x1829f rlid 159] [va 0x2b0ce01d2b80 len 2058 lkey 0x88263]
[b1155:150726:0:150726] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150726:0:150726] ib_mlx5_log.c:139  DCI QP 0xbca5 wqe[2]: SEND s-e [rqpn 0x18179 rlid 159] [va 0x2ace7d9d2b80 len 2058 lkey 0xafc05]
[b1155:150731:0:150731] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150731:0:150731] ib_mlx5_log.c:139  DCI QP 0xbc93 wqe[15]: SEND s-e [rqpn 0x1817c rlid 159] [va 0x2afbe7df7480 len 2058 lkey 0xaece5]
[b1155:150734:0:150734] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150734:0:150734] ib_mlx5_log.c:139  DCI QP 0xbc94 wqe[15]: SEND s-e [rqpn 0x18343 rlid 159] [va 0x2afc2e7e5000 len 2058 lkey 0xa2785]
[b1155:150714:0:150714] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150714:0:150714] ib_mlx5_log.c:139  DCI QP 0xbc8e wqe[2]: SEND s-e [rqpn 0x18332 rlid 159] [va 0x2b81f95d2b80 len 2058 lkey 0xb181c]
[b1155:150729:0:150729] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150729:0:150729] ib_mlx5_log.c:139  DCI QP 0xbc84 wqe[17]: SEND s-e [rqpn 0x180aa rlid 159] [va 0x2b1515bc0700 len 2058 lkey 0x1325c0]
[b1155:150733:0:150733] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150733:0:150733] ib_mlx5_log.c:139  DCI QP 0xbc75 wqe[5]: SEND s-e [rqpn 0x18310 rlid 159] [va 0x2b5799dd2b80 len 2058 lkey 0x8183e]
[b1155:150739:0:150739] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1155:150739:0:150739] ib_mlx5_log.c:139  DCI QP 0xbc91 wqe[29]: SEND s-e [rqpn 0x18108 rlid 159] [va 0x2ac4991ba580 len 2058 lkey 0xac8a2]
==== backtrace (tid: 173435) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173415) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173431) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE5DF0D0630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE5DF313377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE5DF314A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE5ED0928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE5ED096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE5ED0970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE5ED3D3593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE5ED3F3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE5E7D992EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE5E7D60EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE5E2C6E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EECBCE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EEC24846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EECC24A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EEC4F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EEC4FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EECC0C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE5EECBD015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE5EEBF4B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE5DED3386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE5DED6C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE5DED22C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE5DEC973D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE5DF2FF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173426) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADA2787A630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADA27ABD377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADA27ABEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ADA318108B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ADA31814F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ADA318150A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ADA31B74593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ADA31B94D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ADA311432EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADA3110AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADA2B418934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ADA33466E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ADA333CE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ADA3346C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ADA333F955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADA333F9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADA3346AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ADA33467015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADA3339EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADA274DD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADA275166AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADA274CCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADA274413D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADA27AA9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173439) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B87C1640630  Unknown               Unknown  Unknown
libc-2.17.so       00002B87C1883377  gsignal               Unknown  Unknown
libc-2.17.so       00002B87C1884A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B87CB5D68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B87CB5DAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B87CB5DB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B87CB93A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B87CB95AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B87CAF092EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B87CAED0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B87C51DE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B87D522CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B87D5194846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B87D52324A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B87D51BF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B87D51BFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B87D5230C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B87D522D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B87D5164B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B87C12A386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B87C12DC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B87C1292C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B87C12073D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B87C186F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173425) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173421) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2A140A6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2A142E9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2A142EAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2A1E03C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2A1E040F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2A1E0410A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2A1E3A0593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2A1E3C0D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2A1D96F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2A1D936EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2A17C44934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FC92E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FBFA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FC984A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FC2555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FC25EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FC96C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A1FC93015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2A1FBCAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2A13D0986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2A13D426AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2A13CF8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2A13C6D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2A142D5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173422) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173468) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173471) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173406) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B000685F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0006AA2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0006AA3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B00188928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0018896F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B00188970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0018BC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0018BE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B000FD272EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B000FCEEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B000A3FD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B001A44CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B001A3B4846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B001A4524A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B001A3DF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B001A3DFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B001A450C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B001A44D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B001A384B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B00064C286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B00064FB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B00064B1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B00064263D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0006A8E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173473) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173418) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEB82E8B630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEB830CE377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEB830CFA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEB90EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEB90EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEB90EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEB91224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEB91244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEB908232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEB8BF1AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEB86A29934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEB92A78E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEB929E0846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB92A7E4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB92A0B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB92A0BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB92A7CC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB92A79015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEB929B0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEB82AEE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEB82B276AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEB82ADDC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEB82A523D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEB830BA545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173491) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173437) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF44F18C630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF44F3CF377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF44F3D0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF45D2F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF45D2F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF45D2F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF45D624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF45D644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF45CC232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF457E1BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF452D2A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EE5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EDC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EE654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EDF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EDF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EE63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF45EE60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF457F04B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF44EDEF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF44EE286AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF44EDDEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF44ED533D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF44F3BB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2F62A73630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2F62CB6377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2F62CB7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2F74AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2F74AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2F74AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2F74E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2F74E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2F744232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2F6BF02EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2F66611934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2F7665FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2F765C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2F766654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2F765F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2F765F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2F76663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2F76660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2F6BFEBB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2F626D686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2F6270F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2F626C5C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2F6263A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2F62CA2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE4DF6AD630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE4DF8F0377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE4DF8F1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE4E96448B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE4E9648F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE4E96490A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE4E99A8593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE4E99C8D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE4E8F772EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE4E8F3EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE4E324B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB29AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB202846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB2A04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB22D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB22DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB29EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4EB29B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE4EB1D2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE4DF31086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE4DF3496AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE4DF2FFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE4DF2743D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE4DF8DC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACFC4525630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACFC4768377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACFC4769A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACFCE4BB8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACFCE4BFF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACFCE4C00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACFCE81F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACFCE83FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACFCDDEE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACFCDDB5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACFC80C3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD8114E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD807C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD811A4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD80A755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD80A7EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD8118C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFD8115015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACFD804CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACFC418886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACFC41C16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACFC4177C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACFC40EC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACFC4754545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173440) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173479) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173443) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE8D66FA630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE8D693D377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE8D693EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE8E86EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE8E86F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE8E86F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE8E8A23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE8E8A43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE8E80222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE8DFF8AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE8DA298934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA2EFE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA257846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA2F54A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA28255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA282EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA2F3C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8EA2F0015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE8EA227B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE8D635D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE8D63966AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE8D634CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE8D62C13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE8D6929545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173436) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173442) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173410) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173441) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173472) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173412) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173419) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173416) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173424) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173486) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173438) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173430) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173429) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173420) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173432) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173508) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173423) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173434) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173417) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA64437D630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA6445C0377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA6445C1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA64E3138B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA64E317F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA64E3180A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA64E677593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA64E697D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA64DC462EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA64DC0DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA647F1B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA6580B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA65801D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA6580BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA65804855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA658048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA6580B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA6580B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA64FEA1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA643FE086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA6440196AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA643FCFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA643F443D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA6445AC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173427) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
==== backtrace (tid: 173433) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B251A1C9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B251A40C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B251A40DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B252C2658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B252C269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B252C26A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B252C599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B252C5B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2523A922EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2523A59EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B251DD67934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B252DDD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B252DD3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B252DDDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B252DD6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B252DD67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B252DDD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B252DDD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2523FCCB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2519E2C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2519E656AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2519E1BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2519D903D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B251A3F8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0AF5283630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0AF54C6377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0AF54C7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0AFF2198B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0AFF21DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0AFF21E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0AFF57D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0AFF59DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0AFEB4C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0AFEB13EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0AF8E21934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08F15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08E7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08F1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08EA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08EA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08F19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0B08F16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0AFFF47B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0AF4EE686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0AF4F1F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0AF4ED5C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0AF4E4A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0AF54B2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173428) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC354C11630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC354E54377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC354E55A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC35EBA78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC35EBABF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC35EBAC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC35EF0B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC35EF2BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC35E4DA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC35E4A1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC3587AF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC3688CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC368833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3688D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC36885E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC36885EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3688CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3688CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC35FF1FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC35487486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC3548AD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC354863C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC3547D83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC354E40545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173414) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173413) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3E36EB6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3E370F9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3E370FAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3E44EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3E44EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3E44EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3E45224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3E45244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3E448232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3E3FF45EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3E3AA54934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46AAAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46A12846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46AB04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46A3D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46A3DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46AAEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E46AAB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3E469E2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3E36B1986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3E36B526AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3E36B08C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3E36A7D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3E370E5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8D7F017630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8D7F25A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8D7F25BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8D8D0928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8D8D096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8D8D0970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8D8D3C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8D8D3E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8D87CDF2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8D87CA6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8D82BB5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EC01E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EB69846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EC074A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EB9455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EB94EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EC05C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D8EC02015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8D87FEDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8D7EC7A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8D7ECB36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8D7EC69C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8D7EBDE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8D7F246545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4EE7CC0630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4EE7F03377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4EE7F04A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4EF1C568B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4EF1C5AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4EF1C5B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4EF1FBA593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4EF1FDAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4EF15892EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4EF1550EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4EEB85E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF38ACE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF3814846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF38B24A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF383F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF383FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF38B0C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4EF38AD015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4EF37E4B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4EE792386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4EE795C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4EE7912C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4EE78873D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4EE7EEF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE726BE5630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE726E28377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE726E29A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE738B7F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE738B83F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE738B840A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE738EE3593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE738F03D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE7384B22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE738479EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE72A783934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A7D2E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A73A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A7D84A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A76555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A765EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A7D6C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE73A7D3015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE73A70AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE72684886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE7268816AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE726837C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE7267AC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE726E14545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173480) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173481) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
[b1170:179126:0:179126] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179126:0:179126] ib_mlx5_log.c:139  DCI QP 0x18c08 wqe[23]: SEND s-e [rqpn 0x1831f rlid 159] [va 0x2b93a65c4800 len 522 lkey 0x155c12]
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5BE5A40630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5BE5C83377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5BE5C84A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5BEF9D78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5BEF9DBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5BEF9DC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5BEFD3B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5BEFD5BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5BEF30A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5BEF2D1EE4  mca_pml_ucx_progr     Unknown  Unknown
[b1170:179130:0:179130] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179130:0:179130] ib_mlx5_log.c:139  DCI QP 0x18c02 wqe[23]: SEND s-e [rqpn 0x182e8 rlid 159] [va 0x2ba3b57c4800 len 522 lkey 0x1547f6]
[b1170:179145:0:179145] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179145:0:179145] ib_mlx5_log.c:139  DCI QP 0x18c0d wqe[25]: SEND s-e [rqpn 0x1800b rlid 159] [va 0x2b5fdebbc600 len 522 lkey 0x1609e6]
libopen-pal.so.40  00002B5BE95DE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF9641E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF95A9846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF96474A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF95D455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF95D4EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF9645C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BF9642015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5BF9579B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5BE56A386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5BE56DC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5BE5692C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5BE56073D7  PMPI_Init_f08         Unknown  Unknown
[b1170:179187:0:179187] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179187:0:179187] ib_mlx5_log.c:139  DCI QP 0x18c04 wqe[25]: SEND s-e [rqpn 0x18113 rlid 159] [va 0x2ab6953c4800 len 522 lkey 0x15fbd3]
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5BE5C6F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
[b1170:179136:0:179136] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179136:0:179136] ib_mlx5_log.c:139  DCI QP 0x18c0c wqe[5]: SEND s-e [rqpn 0x1833e rlid 159] [va 0x2aff603d8d00 len 522 lkey 0x1540f2]
[b1170:179151:0:179151] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179151:0:179151] ib_mlx5_log.c:139  DCI QP 0x18bc0 wqe[23]: SEND s-e [rqpn 0x1830e rlid 159] [va 0x2b027d7c6880 len 522 lkey 0x15bc85]
[b1170:179157:0:179157] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179157:0:179157] ib_mlx5_log.c:139  DCI QP 0x18b96 wqe[23]: SEND s-e [rqpn 0x17cda rlid 159] [va 0x2b92493c4800 len 522 lkey 0x15a672]
[b1170:179167:0:179167] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179167:0:179167] ib_mlx5_log.c:139  DCI QP 0x18bf7 wqe[23]: SEND s-e [rqpn 0x17cd2 rlid 159] [va 0x2b713fdc4800 len 522 lkey 0x157e37]
[b1170:179175:0:179175] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179175:0:179175] ib_mlx5_log.c:139  DCI QP 0x18bd2 wqe[23]: SEND s-e [rqpn 0x180aa rlid 159] [va 0x2b452bbc4800 len 522 lkey 0x158f54]
[b1170:179180:0:179180] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179180:0:179180] ib_mlx5_log.c:139  DCI QP 0x18c0b wqe[43]: SEND s-e [rqpn 0x18343 rlid 159] [va 0x2ad2121d8d00 len 522 lkey 0x1548f7]
[b1170:179181:0:179181] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179181:0:179181] ib_mlx5_log.c:139  DCI QP 0x18bfd wqe[43]: SEND s-e [rqpn 0x182ff rlid 159] [va 0x2b0e2c3d8d00 len 522 lkey 0x159656]
[b1170:179139:0:179139] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179139:0:179139] ib_mlx5_log.c:139  DCI QP 0x18b6d wqe[25]: SEND s-e [rqpn 0x17cbf rlid 159] [va 0x2b975d9c2780 len 522 lkey 0x15daa7]
[b1170:179152:0:179152] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179152:0:179152] ib_mlx5_log.c:139  DCI QP 0x18b63 wqe[25]: SEND s-e [rqpn 0x18249 rlid 159] [va 0x2b76ef3c4800 len 522 lkey 0x15a973]
[b1170:179168:0:179168] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179168:0:179168] ib_mlx5_log.c:139  DCI QP 0x18b73 wqe[26]: SEND s-e [rqpn 0x1829f rlid 159] [va 0x2ab3b11c4800 len 522 lkey 0x1546f5]
[b1170:179171:0:179171] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179171:0:179171] ib_mlx5_log.c:139  DCI QP 0x18b5a wqe[25]: SEND s-e [rqpn 0x18138 rlid 159] [va 0x2b9bcf5e2f80 len 522 lkey 0x15ca96]
[b1170:179172:0:179172] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179172:0:179172] ib_mlx5_log.c:139  DCI QP 0x18c14 wqe[25]: SEND s-e [rqpn 0x18179 rlid 159] [va 0x2b2e0bfc6880 len 522 lkey 0x158c52]
[b1170:179176:0:179176] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179176:0:179176] ib_mlx5_log.c:139  DCI QP 0x18b7a wqe[23]: SEND s-e [rqpn 0x18181 rlid 159] [va 0x2ab9803c4800 len 522 lkey 0x155d13]
[b1170:179138:0:179138] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179138:0:179138] ib_mlx5_log.c:139  DCI QP 0x18b57 wqe[23]: SEND s-e [rqpn 0x17fb7 rlid 159] [va 0x2ae2b6bc4800 len 522 lkey 0x156117]
[b1170:179149:0:179149] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179149:0:179149] ib_mlx5_log.c:139  DCI QP 0x18b3d wqe[35]: SEND s-e [rqpn 0x182ae rlid 159] [va 0x2b0c3f5d8d00 len 522 lkey 0x15a167]
[b1170:179150:0:179150] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179150:0:179150] ib_mlx5_log.c:139  DCI QP 0x18b41 wqe[15]: SEND s-e [rqpn 0x18259 rlid 159] [va 0x2af2bd9be680 len 522 lkey 0x1607e4]
[b1170:179159:0:179159] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179159:0:179159] ib_mlx5_log.c:139  DCI QP 0x18b54 wqe[26]: SEND s-e [rqpn 0x17ccc rlid 159] [va 0x2b1f05dc4800 len 522 lkey 0x158142]
[b1170:179134:0:179134] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179134:0:179134] ib_mlx5_log.c:139  DCI QP 0x18b2f wqe[13]: SEND s-e [rqpn 0x181ca rlid 159] [va 0x2b3d553d8d00 len 522 lkey 0x157632]
[b1170:179160:0:179160] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179160:0:179160] ib_mlx5_log.c:139  DCI QP 0x18b26 wqe[27]: SEND s-e [rqpn 0x18332 rlid 159] [va 0x2b39193c4800 len 522 lkey 0x158645]
[b1170:179182:0:179182] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179182:0:179182] ib_mlx5_log.c:139  DCI QP 0x18b3c wqe[23]: SEND s-e [rqpn 0x18326 rlid 159] [va 0x2b08bcfc4800 len 522 lkey 0x15e3b5]
[b1170:179133:0:179133] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179133:0:179133] ib_mlx5_log.c:139  DCI QP 0x18abd wqe[13]: SEND s-e [rqpn 0x181ce rlid 159] [va 0x2b2b0e9d6c80 len 522 lkey 0x157531]
[b1170:179135:0:179135] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179135:0:179135] ib_mlx5_log.c:139  DCI QP 0x18ad1 wqe[25]: SEND s-e [rqpn 0x18156 rlid 159] [va 0x2b31b57c4800 len 522 lkey 0x156b25]
[b1170:179137:0:179137] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179137:0:179137] ib_mlx5_log.c:139  DCI QP 0x18ac7 wqe[23]: SEND s-e [rqpn 0x182c2 rlid 159] [va 0x2abc729c4800 len 522 lkey 0x15b581]
[b1170:179147:0:179147] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179147:0:179147] ib_mlx5_log.c:139  DCI QP 0x18aba wqe[25]: SEND s-e [rqpn 0x18337 rlid 159] [va 0x2b5bf6bb8500 len 522 lkey 0x15f5c6]
[b1170:179169:0:179169] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179169:0:179169] ib_mlx5_log.c:139  DCI QP 0x18b07 wqe[23]: SEND s-e [rqpn 0x18283 rlid 159] [va 0x2b8cb07c6880 len 522 lkey 0x15c794]
[b1170:179177:0:179177] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179177:0:179177] ib_mlx5_log.c:139  DCI QP 0x18ae5 wqe[35]: SEND s-e [rqpn 0x1817c rlid 159] [va 0x2b90195d8d00 len 522 lkey 0x15b478]
[b1170:179127:0:179127] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179127:0:179127] ib_mlx5_log.c:139  DCI QP 0x18a8b wqe[26]: SEND s-e [rqpn 0x17fa7 rlid 159] [va 0x2adff0bc4800 len 522 lkey 0x1601d7]
[b1170:179140:0:179140] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179140:0:179140] ib_mlx5_log.c:139  DCI QP 0x18a93 wqe[23]: SEND s-e [rqpn 0x17cb9 rlid 159] [va 0x2b3d091c4800 len 522 lkey 0x15dba8]
[b1170:179178:0:179178] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179178:0:179178] ib_mlx5_log.c:139  DCI QP 0x18a64 wqe[6]: SEND s-e [rqpn 0x18262 rlid 159] [va 0x2b87ccbd8d00 len 522 lkey 0x158041]
[b1170:179146:0:179146] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179146:0:179146] ib_mlx5_log.c:139  DCI QP 0x18a3a wqe[15]: SEND s-e [rqpn 0x18115 rlid 159] [va 0x2b4ff21ba580 len 522 lkey 0x15fad2]
[b1170:179153:0:179153] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179153:0:179153] ib_mlx5_log.c:139  DCI QP 0x18a50 wqe[13]: SEND s-e [rqpn 0x1827f rlid 159] [va 0x2acd2fbd8d00 len 522 lkey 0x15e5b6]
[b1170:179166:0:179166] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179166:0:179166] ib_mlx5_log.c:139  DCI QP 0x18a45 wqe[13]: SEND s-e [rqpn 0x17fb8 rlid 159] [va 0x2b3d2efd8d00 len 522 lkey 0x158e53]
[b1170:179174:0:179174] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179174:0:179174] ib_mlx5_log.c:139  DCI QP 0x18a25 wqe[23]: SEND s-e [rqpn 0x1834d rlid 159] [va 0x2b09293c2780 len 522 lkey 0x159255]
[b1170:179125:0:179125] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179125:0:179125] ib_mlx5_log.c:139  DCI QP 0x189f4 wqe[25]: SEND s-e [rqpn 0x17f9d rlid 159] [va 0x2b52eabc2780 len 522 lkey 0x15f7c7]
[b1170:179128:0:179128] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179128:0:179128] ib_mlx5_log.c:139  DCI QP 0x189ce wqe[25]: SEND s-e [rqpn 0x181dd rlid 159] [va 0x2ac59c9c4800 len 522 lkey 0x15fcd4]
[b1170:179154:0:179154] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179154:0:179154] ib_mlx5_log.c:139  DCI QP 0x189cc wqe[15]: SEND s-e [rqpn 0x18323 rlid 159] [va 0x2b37641c2780 len 522 lkey 0x160de7]
[b1170:179162:0:179162] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179162:0:179162] ib_mlx5_log.c:139  DCI QP 0x18a29 wqe[23]: SEND s-e [rqpn 0x180ea rlid 159] [va 0x2ae0137c4800 len 522 lkey 0x15b377]
[b1170:179164:0:179164] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179164:0:179164] ib_mlx5_log.c:139  DCI QP 0x189da wqe[23]: SEND s-e [rqpn 0x18064 rlid 159] [va 0x2b30d4bc6880 len 522 lkey 0x155605]
[b1170:179144:0:179144] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179144:0:179144] ib_mlx5_log.c:139  DCI QP 0x18a18 wqe[16]: SEND s-e [rqpn 0x18045 rlid 159] [va 0x2b3a34dba580 len 522 lkey 0x15e9b7]
[b1170:179155:0:179155] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179155:0:179155] ib_mlx5_log.c:139  DCI QP 0x189ca wqe[25]: SEND s-e [rqpn 0x1827c rlid 159] [va 0x2b92fabc8900 len 522 lkey 0x15f3c5]
[b1170:179186:0:179186] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179186:0:179186] ib_mlx5_log.c:139  DCI QP 0x1898b wqe[23]: SEND s-e [rqpn 0x18210 rlid 159] [va 0x2b5733bc4800 len 522 lkey 0x1600d6]
[b1170:179131:0:179131] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179131:0:179131] ib_mlx5_log.c:139  DCI QP 0x189a3 wqe[15]: SEND s-e [rqpn 0x181f4 rlid 159] [va 0x2b94f19be680 len 522 lkey 0x15d5a3]
[b1170:179132:0:179132] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179132:0:179132] ib_mlx5_log.c:139  DCI QP 0x1897d wqe[23]: SEND s-e [rqpn 0x182ea rlid 159] [va 0x2b98cc7c8900 len 522 lkey 0x155103]
[b1170:179163:0:179163] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179163:0:179163] ib_mlx5_log.c:139  DCI QP 0x18971 wqe[23]: SEND s-e [rqpn 0x180b5 rlid 159] [va 0x2b67af1c4800 len 522 lkey 0x158a48]
[b1170:179170:0:179170] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179170:0:179170] ib_mlx5_log.c:139  DCI QP 0x18986 wqe[25]: SEND s-e [rqpn 0x1808f rlid 159] [va 0x2ac4bc3c2780 len 522 lkey 0x158b51]
[b1170:179179:0:179179] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179179:0:179179] ib_mlx5_log.c:139  DCI QP 0x18989 wqe[5]: SEND s-e [rqpn 0x18310 rlid 159] [va 0x2b3d181d6c80 len 522 lkey 0x156622]
[b1170:179143:0:179143] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179143:0:179143] ib_mlx5_log.c:139  DCI QP 0x1894d wqe[23]: SEND s-e [rqpn 0x18214 rlid 159] [va 0x2b59e1bc4800 len 522 lkey 0x15c187]
[b1170:179158:0:179158] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179158:0:179158] ib_mlx5_log.c:139  DCI QP 0x1894b wqe[26]: SEND s-e [rqpn 0x1812e rlid 159] [va 0x2aef937c2780 len 522 lkey 0x154b01]
[b1170:179165:0:179165] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179165:0:179165] ib_mlx5_log.c:139  DCI QP 0x18916 wqe[25]: SEND s-e [rqpn 0x18110 rlid 159] [va 0x2b03c4fba580 len 522 lkey 0x15ba83]
[b1170:179183:0:179183] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179183:0:179183] ib_mlx5_log.c:139  DCI QP 0x18936 wqe[5]: SEND s-e [rqpn 0x181d5 rlid 159] [va 0x2b0fd6fd8d00 len 522 lkey 0x159757]
[b1170:179161:0:179161] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179161:0:179161] ib_mlx5_log.c:139  DCI QP 0x18931 wqe[43]: SEND s-e [rqpn 0x17fa5 rlid 159] [va 0x2b8d0add8d00 len 522 lkey 0x157934]
[b1170:179148:0:179148] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179148:0:179148] ib_mlx5_log.c:139  DCI QP 0x188cf wqe[15]: SEND s-e [rqpn 0x17cc8 rlid 159] [va 0x2abe6d7ba580 len 522 lkey 0x15f9d1]
[b1170:179173:0:179173] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179173:0:179173] ib_mlx5_log.c:139  DCI QP 0x188be wqe[23]: SEND s-e [rqpn 0x180ed rlid 159] [va 0x2ad0bb7c4800 len 522 lkey 0x15ae76]
[b1170:179069:0:179069] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179069:0:179069] ib_mlx5_log.c:139  DCI QP 0x18912 wqe[13]: SEND s-e [rqpn 0x17f54 rlid 159] [va 0x2b1fa45d8d00 len 522 lkey 0x156924]
[b1170:179082:0:179082] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179082:0:179082] ib_mlx5_log.c:139  DCI QP 0x188fe wqe[25]: SEND s-e [rqpn 0x17ce7 rlid 159] [va 0x2b9978db8500 len 522 lkey 0x15d7a5]
[b1170:179111:0:179111] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179111:0:179111] ib_mlx5_log.c:139  DCI QP 0x188f3 wqe[5]: SEND s-e [rqpn 0x17cd0 rlid 159] [va 0x2b2a971d8d00 len 522 lkey 0x156723]
[b1170:179141:0:179141] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179141:0:179141] ib_mlx5_log.c:139  DCI QP 0x188b8 wqe[15]: SEND s-e [rqpn 0x18348 rlid 159] [va 0x2b81959ba580 len 522 lkey 0x15f2c4]
[b1170:179103:0:179103] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179103:0:179103] ib_mlx5_log.c:139  DCI QP 0x188b5 wqe[26]: SEND s-e [rqpn 0x1804a rlid 159] [va 0x2b58157c4800 len 522 lkey 0x158746]
[b1170:179156:0:179156] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179156:0:179156] ib_mlx5_log.c:139  DCI QP 0x188b2 wqe[23]: SEND s-e [rqpn 0x1822e rlid 159] [va 0x2b04853c8900 len 522 lkey 0x15eab8]
[b1170:179185:0:179185] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179185:0:179185] ib_mlx5_log.c:139  DCI QP 0x18886 wqe[26]: SEND s-e [rqpn 0x18108 rlid 159] [va 0x2aefcc9c4800 len 522 lkey 0x1602d8]
[b1170:179071:0:179071] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179071:0:179071] ib_mlx5_log.c:139  DCI QP 0x1886b wqe[23]: SEND s-e [rqpn 0x17d5b rlid 159] [va 0x0 len 522 lkey 0x8895]
[b1170:179101:0:179101] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179101:0:179101] ib_mlx5_log.c:139  DCI QP 0x18856 wqe[25]: SEND s-e [rqpn 0x17d02 rlid 159] [va 0x0 len 522 lkey 0x8593]
[b1170:179188:0:179188] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179188:0:179188] ib_mlx5_log.c:139  DCI QP 0x18885 wqe[5]: SEND s-e [rqpn 0x18231 rlid 159] [va 0x0 len 522 lkey 0x8996]
[b1170:179113:0:179113] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179113:0:179113] ib_mlx5_log.c:139  DCI QP 0x1885c wqe[13]: SEND s-e [rqpn 0x17cc1 rlid 159] [va 0x0 len 522 lkey 0x8794]
[b1170:179094:0:179094] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179094:0:179094] ib_mlx5_log.c:139  DCI QP 0x1880b wqe[25]: SEND s-e [rqpn 0x17d50 rlid 159] [va 0x0 len 522 lkey 0x7d8f]
[b1170:179085:0:179085] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179085:0:179085] ib_mlx5_log.c:139  DCI QP 0x18812 wqe[13]: SEND s-e [rqpn 0x17cf6 rlid 159] [va 0x0 len 522 lkey 0x7f90]
[b1170:179095:0:179095] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179095:0:179095] ib_mlx5_log.c:139  DCI QP 0x1881c wqe[25]: SEND s-e [rqpn 0x17d16 rlid 159] [va 0x0 len 522 lkey 0x8391]
[b1170:179123:0:179123] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179123:0:179123] ib_mlx5_log.c:139  DCI QP 0x1883c wqe[23]: SEND s-e [rqpn 0x17cb0 rlid 159] [va 0x0 len 522 lkey 0x8492]
[b1170:179124:0:179124] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179124:0:179124] ib_mlx5_log.c:139  DCI QP 0x187ed wqe[6]: SEND s-e [rqpn 0x17ce4 rlid 159] [va 0x0 len 522 lkey 0x7a8e]
[b1170:179086:0:179086] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179086:0:179086] ib_mlx5_log.c:139  DCI QP 0x187e6 wqe[26]: SEND s-e [rqpn 0x17d58 rlid 159] [va 0x0 len 522 lkey 0x778d]
[b1170:179122:0:179122] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179122:0:179122] ib_mlx5_log.c:139  DCI QP 0x187d6 wqe[23]: SEND s-e [rqpn 0x1807b rlid 159] [va 0x0 len 522 lkey 0x748c]
[b1170:179129:0:179129] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179129:0:179129] ib_mlx5_log.c:139  DCI QP 0x187ba wqe[5]: SEND s-e [rqpn 0x18235 rlid 159] [va 0x0 len 522 lkey 0x6c8a]
[b1170:179088:0:179088] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179088:0:179088] ib_mlx5_log.c:139  DCI QP 0x1878a wqe[23]: SEND s-e [rqpn 0x17d1c rlid 159] [va 0x0 len 522 lkey 0x6687]
[b1170:179184:0:179184] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179184:0:179184] ib_mlx5_log.c:139  DCI QP 0x187ac wqe[23]: SEND s-e [rqpn 0x181d2 rlid 159] [va 0x0 len 522 lkey 0x6889]
[b1170:179067:0:179067] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179067:0:179067] ib_mlx5_log.c:139  DCI QP 0x1877f wqe[25]: SEND s-e [rqpn 0x18004 rlid 159] [va 0x0 len 522 lkey 0x6486]
[b1170:179077:0:179077] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179077:0:179077] ib_mlx5_log.c:139  DCI QP 0x1879e wqe[15]: SEND s-e [rqpn 0x17d48 rlid 159] [va 0x0 len 522 lkey 0x6788]
[b1170:179120:0:179120] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179120:0:179120] ib_mlx5_log.c:139  DCI QP 0x187cf wqe[22]: SEND s-e [rqpn 0x17cbd rlid 159] [va 0x0 len 522 lkey 0x6d8b]
[b1170:179087:0:179087] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179087:0:179087] ib_mlx5_log.c:139  DCI QP 0x1875a wqe[5]: SEND s-e [rqpn 0x17f64 rlid 159] [va 0x0 len 522 lkey 0x6284]
[b1170:179109:0:179109] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179109:0:179109] ib_mlx5_log.c:139  DCI QP 0x1871b wqe[26]: SEND s-e [rqpn 0x17ccb rlid 159] [va 0x0 len 522 lkey 0x5a7f]
[b1170:179116:0:179116] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179116:0:179116] ib_mlx5_log.c:139  DCI QP 0x18731 wqe[42]: SEND s-e [rqpn 0x17cb4 rlid 159] [va 0x0 len 522 lkey 0x5b80]
[b1170:179072:0:179072] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179072:0:179072] ib_mlx5_log.c:139  DCI QP 0x186c8 wqe[43]: SEND s-e [rqpn 0x17d2b rlid 159] [va 0x0 len 522 lkey 0x567b]
[b1170:179093:0:179093] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179093:0:179093] ib_mlx5_log.c:139  DCI QP 0x186f1 wqe[23]: SEND s-e [rqpn 0x17d53 rlid 159] [va 0x0 len 522 lkey 0x587d]
[b1170:179106:0:179106] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179106:0:179106] ib_mlx5_log.c:139  DCI QP 0x1874a wqe[26]: SEND s-e [rqpn 0x17ce0 rlid 159] [va 0x0 len 522 lkey 0x6083]
[b1170:179107:0:179107] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179107:0:179107] ib_mlx5_log.c:139  DCI QP 0x18779 wqe[15]: SEND s-e [rqpn 0x17cd6 rlid 159] [va 0x0 len 522 lkey 0x6385]
[b1170:179119:0:179119] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179119:0:179119] ib_mlx5_log.c:139  DCI QP 0x1872d wqe[43]: SEND s-e [rqpn 0x17f84 rlid 159] [va 0x0 len 522 lkey 0x5e81]
[b1170:179062:0:179062] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179062:0:179062] ib_mlx5_log.c:139  DCI QP 0x186ad wqe[23]: SEND s-e [rqpn 0x17d27 rlid 159] [va 0x0 len 522 lkey 0x5479]
[b1170:179064:0:179064] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179064:0:179064] ib_mlx5_log.c:139  DCI QP 0x18739 wqe[23]: SEND s-e [rqpn 0x17d0f rlid 159] [va 0x0 len 522 lkey 0x5f82]
[b1170:179073:0:179073] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179073:0:179073] ib_mlx5_log.c:139  DCI QP 0x186ba wqe[23]: SEND s-e [rqpn 0x17d44 rlid 159] [va 0x0 len 522 lkey 0x557a]
[b1170:179074:0:179074] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179074:0:179074] ib_mlx5_log.c:139  DCI QP 0x186d8 wqe[23]: SEND s-e [rqpn 0x17fc7 rlid 159] [va 0x0 len 522 lkey 0x577c]
[b1170:179076:0:179076] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179076:0:179076] ib_mlx5_log.c:139  DCI QP 0x186fe wqe[25]: SEND s-e [rqpn 0x180a6 rlid 159] [va 0x0 len 522 lkey 0x597e]
[b1170:179097:0:179097] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179097:0:179097] ib_mlx5_log.c:139  DCI QP 0x186a3 wqe[5]: SEND s-e [rqpn 0x17d34 rlid 159] [va 0x0 len 522 lkey 0x5378]
[b1170:179096:0:179096] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179096:0:179096] ib_mlx5_log.c:139  DCI QP 0x1868d wqe[26]: SEND s-e [rqpn 0x17d57 rlid 159] [va 0x0 len 522 lkey 0x5177]
[b1170:179098:0:179098] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179098:0:179098] ib_mlx5_log.c:139  DCI QP 0x1863c wqe[22]: SEND s-e [rqpn 0x17d08 rlid 159] [va 0x0 len 522 lkey 0x4972]
[b1170:179102:0:179102] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179102:0:179102] ib_mlx5_log.c:139  DCI QP 0x18672 wqe[13]: SEND s-e [rqpn 0x17d1e rlid 159] [va 0x0 len 522 lkey 0x5076]
[b1170:179104:0:179104] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179104:0:179104] ib_mlx5_log.c:139  DCI QP 0x18676 wqe[22]: SEND s-e [rqpn 0x17d38 rlid 159] [va 0x0 len 522 lkey 0x4f75]
[b1170:179108:0:179108] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179108:0:179108] ib_mlx5_log.c:139  DCI QP 0x18646 wqe[23]: SEND s-e [rqpn 0x17d3d rlid 159] [va 0x0 len 522 lkey 0x4a73]
[b1170:179118:0:179118] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179118:0:179118] ib_mlx5_log.c:139  DCI QP 0x1865f wqe[23]: SEND s-e [rqpn 0x17cf2 rlid 159] [va 0x0 len 522 lkey 0x4b74]
[b1170:179061:0:179061] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179061:0:179061] ib_mlx5_log.c:139  DCI QP 0x1862c wqe[26]: SEND s-e [rqpn 0x17f34 rlid 159] [va 0x0 len 522 lkey 0x4771]
[b1170:179078:0:179078] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179078:0:179078] ib_mlx5_log.c:139  DCI QP 0x18624 wqe[26]: SEND s-e [rqpn 0x17d1f rlid 159] [va 0x0 len 522 lkey 0x4570]
[b1170:179063:0:179063] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179063:0:179063] ib_mlx5_log.c:139  DCI QP 0x185fb wqe[25]: SEND s-e [rqpn 0x17d5e rlid 159] [va 0x0 len 522 lkey 0x426e]
[b1170:179070:0:179070] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179070:0:179070] ib_mlx5_log.c:139  DCI QP 0x185ea wqe[5]: SEND s-e [rqpn 0x17f3e rlid 159] [va 0x0 len 522 lkey 0x406d]
[b1170:179100:0:179100] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179100:0:179100] ib_mlx5_log.c:139  DCI QP 0x18612 wqe[22]: SEND s-e [rqpn 0x17e9b rlid 159] [va 0x0 len 522 lkey 0x446f]
[b1170:179075:0:179075] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179075:0:179075] ib_mlx5_log.c:139  DCI QP 0x185d4 wqe[23]: SEND s-e [rqpn 0x18067 rlid 159] [va 0x0 len 522 lkey 0x3d6b]
[b1170:179080:0:179080] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179080:0:179080] ib_mlx5_log.c:139  DCI QP 0x185b8 wqe[15]: SEND s-e [rqpn 0x181a3 rlid 159] [va 0x0 len 522 lkey 0x396a]
[b1170:179115:0:179115] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179115:0:179115] ib_mlx5_log.c:139  DCI QP 0x185e1 wqe[43]: SEND s-e [rqpn 0x17d1b rlid 159] [va 0x0 len 522 lkey 0x3f6c]
[b1170:179083:0:179083] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179083:0:179083] ib_mlx5_log.c:139  DCI QP 0x18517 wqe[14]: SEND s-e [rqpn 0x17d54 rlid 159] [va 0x0 len 522 lkey 0x3366]
[b1170:179110:0:179110] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179110:0:179110] ib_mlx5_log.c:139  DCI QP 0x18571 wqe[22]: SEND s-e [rqpn 0x17cd4 rlid 159] [va 0x0 len 522 lkey 0x3467]
[b1170:179112:0:179112] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179112:0:179112] ib_mlx5_log.c:139  DCI QP 0x185a3 wqe[22]: SEND s-e [rqpn 0x17cb7 rlid 159] [va 0x0 len 522 lkey 0x3869]
[b1170:179114:0:179114] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179114:0:179114] ib_mlx5_log.c:139  DCI QP 0x1857c wqe[42]: SEND s-e [rqpn 0x17cc3 rlid 159] [va 0x0 len 522 lkey 0x3568]
[b1170:179065:0:179065] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179065:0:179065] ib_mlx5_log.c:139  DCI QP 0x1850d wqe[5]: SEND s-e [rqpn 0x17cf8 rlid 159] [va 0x0 len 522 lkey 0x3265]
[b1170:179090:0:179090] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179090:0:179090] ib_mlx5_log.c:139  DCI QP 0x1851b wqe[25]: SEND s-e [rqpn 0x17cce rlid 159] [va 0x0 len 522 lkey 0x2c5f]
[b1170:179117:0:179117] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179117:0:179117] ib_mlx5_log.c:139  DCI QP 0x18513 wqe[5]: SEND s-e [rqpn 0x17f3a rlid 159] [va 0x0 len 522 lkey 0x2f62]
[b1170:179079:0:179079] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179079:0:179079] ib_mlx5_log.c:139  DCI QP 0x18505 wqe[15]: SEND s-e [rqpn 0x17fbe rlid 159] [va 0x0 len 522 lkey 0x2b5e]
[b1170:179081:0:179081] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179081:0:179081] ib_mlx5_log.c:139  DCI QP 0x1850b wqe[15]: SEND s-e [rqpn 0x18023 rlid 159] [va 0x0 len 522 lkey 0x2d60]
==== backtrace (tid: 173409) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
[b1170:179084:0:179084] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179084:0:179084] ib_mlx5_log.c:139  DCI QP 0x18500 wqe[15]: SEND s-e [rqpn 0x17cfd rlid 159] [va 0x0 len 522 lkey 0x2659]
[b1170:179089:0:179089] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179089:0:179089] ib_mlx5_log.c:139  DCI QP 0x184fe wqe[35]: SEND s-e [rqpn 0x18025 rlid 159] [va 0x0 len 522 lkey 0x275a]
[b1170:179091:0:179091] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179091:0:179091] ib_mlx5_log.c:139  DCI QP 0x184fb wqe[23]: SEND s-e [rqpn 0x17ced rlid 159] [va 0x0 len 522 lkey 0x3164]
[b1170:179099:0:179099] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179099:0:179099] ib_mlx5_log.c:139  DCI QP 0x184fa wqe[26]: SEND s-e [rqpn 0x17d2f rlid 159] [va 0x0 len 522 lkey 0x295c]
[b1170:179092:0:179092] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179092:0:179092] ib_mlx5_log.c:139  DCI QP 0x184f6 wqe[23]: SEND s-e [rqpn 0x17d4d rlid 159] [va 0x0 len 522 lkey 0x2e61]
[b1170:179068:0:179068] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179068:0:179068] ib_mlx5_log.c:139  DCI QP 0x184ee wqe[23]: SEND s-e [rqpn 0x17d24 rlid 159] [va 0x0 len 522 lkey 0x285b]
[b1170:179105:0:179105] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179105:0:179105] ib_mlx5_log.c:139  DCI QP 0x184f3 wqe[23]: SEND s-e [rqpn 0x17d21 rlid 159] [va 0x0 len 522 lkey 0x3063]
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
[b1170:179142:0:179142] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179142:0:179142] ib_mlx5_log.c:139  DCI QP 0x184ea wqe[15]: SEND s-e [rqpn 0x18119 rlid 159] [va 0x0 len 522 lkey 0x2457]
[b1170:179066:0:179066] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179066:0:179066] ib_mlx5_log.c:139  DCI QP 0x184d9 wqe[23]: SEND s-e [rqpn 0x17fa2 rlid 159] [va 0x0 len 522 lkey 0x2a5d]
[b1170:179121:0:179121] ib_mlx5_log.c:139  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[b1170:179121:0:179121] ib_mlx5_log.c:139  DCI QP 0x184d5 wqe[23]: SEND s-e [rqpn 0x1804d rlid 159] [va 0x0 len 522 lkey 0x2558]
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACDFE392630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACDFE5D5377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACDFE5D6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACE104918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACE10495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACE104960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACE107C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACE107E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACE07C5B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACE07C22EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACE01F30934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACE12000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACE11F68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE120064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE11F9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE11F93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE12004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE12001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACE07F69B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACDFDFF586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACDFE02E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACDFDFE4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACDFDF593D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACDFE5C1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7B9BB05630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B9BD48377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7B9BD49A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7BA5A9B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7BA5A9FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7BA5AA00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7BA5DFF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7BA5E1FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7BA53CE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7BA5395EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7B9F6A3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA76F1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA7659846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA76F74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA768455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA7684EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA76F5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BA76F2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7BA7629B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7B9B76886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7B9B7A16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7B9B757C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7B9B6CC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B9BD34545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE80EA6E630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE80ECB1377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE80ECB2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE820AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE820AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE820AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE820E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE820E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE8204232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE817EFDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE81260C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE82265FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE8225C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8226654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8225F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8225F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE822663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE822660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE817FE6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE80E6D186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE80E70A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE80E6C0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE80E6353D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE80EC9D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6330B7F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6330DC2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6330DC3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B633AB158B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B633AB19F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B633AB1A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B633AE79593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B633AE99D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B633A4482EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B633A40FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B633471D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6344770E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B63446D8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B63447764A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B634470355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6344703EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6344774C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6344771015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B63446A8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B63307E286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B633081B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B63307D1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B63307463D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6330DAE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173455) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173469) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6242E8A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B62430CD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B62430CEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6250EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6250EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6250EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6251224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6251244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B62508232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B624BF1AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6246A28934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6252A78E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B62529E0846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6252A7E4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6252A0B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6252A0BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6252A7CC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6252A79015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B62529B0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6242AED86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6242B266AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6242ADCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6242A513D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B62430B9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173470) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B233A26C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B233A4AF377  gsignal               Unknown  Unknown
libc-2.17.so       00002B233A4B0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B234C2658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B234C269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B234C26A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B234C599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B234C5B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2343B352EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2343AFCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B233DE0A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B234DE56E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B234DDBE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B234DE5C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B234DDE955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B234DDE9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B234DE5AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B234DE57015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B234DD8EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2339ECF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2339F086AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2339EBEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2339E333D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B233A49B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173465) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7F640AF630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7F642F2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7F642F3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7F6E0458B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7F6E049F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7F6E04A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7F6E3A9593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7F6E3C9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7F6D9782EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7F6D93FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7F67C4D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FC9BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FC03846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FCA14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FC2E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FC2EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FC9FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F6FC9C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7F6FBD3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7F63D1286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7F63D4B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7F63D01C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7F63C763D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7F642DE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA2A9F62630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA2AA1A5377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA2AA1A6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA2BC04F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA2BC053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA2BC0540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA2BC383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA2BC3A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA2B382B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA2B37F2EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA2ADB00934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDBBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDB26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDBC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDB5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDB51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDBC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA2BDBBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA2B3F7BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA2A9BC586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA2A9BFE6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA2A9BB4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA2A9B293D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA2AA191545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173411) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173474) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173475) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABF136CC630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABF1390F377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABF13910A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABF1D6628B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABF1D666F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABF1D6670A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABF1D9C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABF1D9E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABF1CF952EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABF1CF5CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABF1726A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F2B8E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F220846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F2BE4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F24B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F24BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F2BCC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABF1F2B9015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABF1F1F0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABF1332F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABF133686AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABF1331EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABF132933D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABF138FB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABE0B0FB630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABE0B33E377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABE0B33FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABE190928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABE19096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABE190970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABE193F6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABE19416D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABE13DC42EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABE13D8BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABE0EC99934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1ACE8E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1AC50846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1ACEE4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1AC7B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1AC7BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1ACECC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE1ACE9015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABE1AC20B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABE0AD5E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABE0AD976AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABE0AD4DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABE0ACC23D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABE0B32A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF390AB8630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF390CFB377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF390CFCA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF39AA4E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF39AA52F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF39AA530A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF39ADB2593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF39ADD2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF39A3812EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF39A348EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF394656934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A46C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A462E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A46CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A465955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A4659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A46CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF3A46C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF39BFCBB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF39071B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF3907546AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF39070AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF39067F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF390CE7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF210C9C630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF210EDF377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF210EE0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF21AC328B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF21AC36F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF21AC370A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF21AF96593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF21AFB6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF21A5652EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF21A52CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF21483A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF2248CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF224833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2248D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF22485E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF22485EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2248CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2248CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF21BFAAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF2108FF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF2109386AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF2108EEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF2108633D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF210ECB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173408) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC45C0E2630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC45C325377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC45C326A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC4660788B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC46607CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC46607D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC4663DC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC4663FCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC4659AB2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC465972EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC45FC80934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC467CCEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC467C36846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC467CD44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC467C6155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC467C61EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC467CD2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC467CCF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC467C06B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC45BD4586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC45BD7E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC45BD34C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC45BCA93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC45C311545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B420CD6A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B420CFAD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B420CFAEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4216D008B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4216D04F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4216D050A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4217064593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4217084D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B42166332EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B42165FAEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4210908934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B422095CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B42208C4846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B42209624A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B42208EF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B42208EFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4220960C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B422095D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4220894B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B420C9CD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B420CA066AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B420C9BCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B420C9313D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B420CF99545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173467) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B85105E1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8510824377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8510825A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B851A5778B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B851A57BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B851A57C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B851A8DB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B851A8FBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8519EAA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8519E71EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B851417F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B85242BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8524222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B85242C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B852424D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B852424DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B85242BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B85242BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B851BF00B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B851024486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B851027D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8510233C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B85101A83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8510810545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173463) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173404) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173405) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173407) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B584BA7E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B584BCC1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B584BCC2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5855A148B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5855A18F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5855A190A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5855D78593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5855D98D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B58553472EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B585530EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B584F61C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B585766AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B58575D2846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B58576704A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B58575FD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B58575FDEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B585766EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B585766B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B58575A2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B584B6E186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B584B71A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B584B6D0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B584B6453D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B584BCAD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173464) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173452) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173466) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173461) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173460) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173462) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173476) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173477) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173478) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173482) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173483) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173489) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B965000C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B965024F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9650250A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9659FA28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9659FA6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9659FA70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B965A306593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B965A326D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B96598D52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B965989CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9653BAA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B965BBF8E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B965BB60846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B965BBFE4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B965BB8B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B965BB8BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B965BBFCC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B965BBF9015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B965BB30B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B964FC6F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B964FCA86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B964FC5EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B964FBD33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B965023B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173487) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173445) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173488) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173403) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173484) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173485) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173490) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE10B0AB630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE10B2EE377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE10B2EFA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE1190928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE119096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE1190970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE1193C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE1193E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE113D742EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE113D3BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE10EC49934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE11ACA2E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE11AC0A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE11ACA84A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE11AC3555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE11AC35EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE11ACA6C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE11ACA3015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE11ABDAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE10AD0E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE10AD476AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE10ACFDC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE10AC723D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE10B2DA545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173397) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173400) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173401) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173402) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173396) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173398) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173399) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B658460E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6584851377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6584852A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B658E5A48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B658E5A8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B658E5A90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B658E908593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B658E928D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B658DED72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B658DE9EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B65881AC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B65982BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6598222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B65982C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B659824D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B659824DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B65982BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B65982BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B658FF2DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B658427186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B65842AA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6584260C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B65841D53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B658483D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173449) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9D61DDB630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9D6201E377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9D6201FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9D7404F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9D74053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9D740540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9D6BD70593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9D6BD90D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9D6B6A42EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9D6B66BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9D65979934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9D759CAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9D75932846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D759D04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D7595D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D7595DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D759CEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D759CB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9D75902B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9D61A3E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9D61A776AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9D61A2DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9D619A23D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9D6200A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AAC73315630  Unknown               Unknown  Unknown
libc-2.17.so       00002AAC73558377  gsignal               Unknown  Unknown
libc-2.17.so       00002AAC73559A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AAC812F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AAC812F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AAC812F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AAC81624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AAC81644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AAC80C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AAC7BFA4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AAC76EB3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82F00E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82E68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82F064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82E9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82E93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82F04C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AAC82F01015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AAC82E38B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AAC72F7886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AAC72FB16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AAC72F67C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AAC72EDC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AAC73544545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0A54A3B630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0A54C7E377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0A54C7FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0A5E9D18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0A5E9D5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0A5E9D60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0A5ED35593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0A5ED55D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0A5E3042EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0A5E2CBEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0A585D9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0A686C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0A6862E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A686CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A6865955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A68659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A686CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A686C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0A5FF4EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0A5469E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0A546D76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0A5468DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0A546023D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0A54C6A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173501) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173454) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173502) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B286F547630  Unknown               Unknown  Unknown
libc-2.17.so       00002B286F78A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B286F78BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B28794DD8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B28794E1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B28794E20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2879841593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2879861D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2878E102EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2878DD7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B28730E5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B287B133E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B287B09B846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B287B1394A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B287B0C655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B287B0C6EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B287B137C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B287B134015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B287B06BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B286F1AA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B286F1E36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B286F199C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B286F10E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B286F776545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B099FDD1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B09A0014377  gsignal               Unknown  Unknown
libc-2.17.so       00002B09A0015A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B09A9D678B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B09A9D6BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B09A9D6C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B09AA0CB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B09AA0EBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B09A969A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B09A9661EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B09A396F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB9BDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB925846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB9C34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB95055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB950EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB9C1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B09AB9BE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B09AB8F5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B099FA3486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B099FA6D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B099FA23C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B099F9983D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B09A0000545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173448) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173453) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173446) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173497) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7B2A3F8630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B2A63B377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7B2A63CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7B3C4918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7B3C495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7B3C4960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7B3C7C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7B3C7E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7B33CC22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7B33C89EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7B2DF96934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3E000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3DF68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3E0064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3DF9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3DF93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3E004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3E001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7B33FD1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7B2A05B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7B2A0946AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7B2A04AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7B29FBF3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B2A627545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2FA487D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2FA4AC0377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2FA4AC1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2FAE8138B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2FAE817F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2FAE8180A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2FAEB77593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2FAEB97D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2FAE1462EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2FAE10DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2FA841B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB84C1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB8429846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB84C74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB845455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB8454EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB84C5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FB84C2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2FAFF95B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2FA44E086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2FA45196AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2FA44CFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2FA44443D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2FA4AAC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150742) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2B7F9B7630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2B7FBFA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2B7FBFBA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2B8994E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2B89952F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2B899530A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2B89CB2593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2B89CD2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2B892812EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2B89248EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2B83555934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B5A4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B50C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B5AA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B53755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B537EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B5A8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B8B5A5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2B8B4DCB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2B7F61A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2B7F6536AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2B7F609C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2B7F57E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2B7FBE6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2DAB2D6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2DAB519377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2DAB51AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2DB92F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2DB92F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2DB92F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2DB9624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2DB9644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2DB8C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2DB3F65EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2DAEE74934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAECEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAE36846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAED44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAE6155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAE61EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAED2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DBAECF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2DBAE06B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2DAAF3986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2DAAF726AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2DAAF28C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2DAAE9D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2DAB505545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B76F5DCD630  Unknown               Unknown  Unknown
libc-2.17.so       00002B76F6010377  gsignal               Unknown  Unknown
libc-2.17.so       00002B76F6011A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B770804F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7708053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B77080540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B76FFD62593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B76FFD82D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B76FF6962EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B76FF65DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B76F996B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B77099BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7709922846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B77099C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B770994D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B770994DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B77099BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B77099BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B77098F2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B76F5A3086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B76F5A696AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B76F5A1FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B76F59943D7  PMPI_Init_f08         Unknown  Unknown
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B76F5FFC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3E52D2E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3E52F71377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3E52F72A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3E60CC48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3E60CC8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3E60CC90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3E61028593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3E61048D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3E608232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3E5BDBDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3E568CC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3E6291AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3E62882846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E629204A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E628AD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E628ADEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E6291EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3E6291B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3E62852B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3E5299186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3E529CA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3E52980C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3E528F53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3E52F5D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173447) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173493) ====
==== backtrace (tid: 173492) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABC5B4DC630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABC5B71F377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABC5B720A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABC654728B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABC65476F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABC654770A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABC657D6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABC657F6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABC64DA52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABC64D6CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABC5F07A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABC670C8E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABC67030846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC670CE4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC6705B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC6705BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC670CCC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC670C9015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABC67000B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABC5B13F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABC5B1786AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABC5B12EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABC5B0A33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABC5B70B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABDFAF80630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABDFB1C3377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABDFB1C4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABE08F1B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABE08F1FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABE08F200A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABE0927F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABE0929FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABE0884E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABE08815EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABDFEB1E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AB6EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AAD6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AB744A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AB0155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AB01EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AB72C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE0AB6F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABE0AAA6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABDFABE386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABDFAC1C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABDFABD2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABDFAB473D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABDFB1AF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B246B78E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B246B9D1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B246B9D2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B24757258B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2475729F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B247572A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2475A89593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2475AA9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B24750582EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B247501FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B246F32C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B247737BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B24772E3846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B24773814A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B247730E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B247730EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B247737FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B247737C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B24772B3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B246B3F186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B246B42A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B246B3E0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B246B3553D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B246B9BD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B50FC66E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B50FC8B1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B50FC8B2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B51066058B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5106609F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B510660A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5106969593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5106989D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5105F382EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5105EFFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B510020C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B51102BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5110222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B51102C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B511024D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B511024DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B51102BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B51102BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5107F8EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B50FC2D186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B50FC30A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B50FC2C0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B50FC2353D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B50FC89D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B47F24B8630  Unknown               Unknown  Unknown
libc-2.17.so       00002B47F26FB377  gsignal               Unknown  Unknown
libc-2.17.so       00002B47F26FCA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B48044918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4804495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B48044960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B48047C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B48047E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B47FBD812EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B47FBD48EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B47F6056934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B48060A4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B480600C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B48060AA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B480603755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4806037EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B48060A8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B48060A5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4805FDCB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B47F211B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B47F21546AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B47F210AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B47F207F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B47F26E7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173503) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173500) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0311907630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0311B4A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0311B4BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B031B89D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B031B8A1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B031B8A20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B031BC01593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B031BC21D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B031B1D02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B031B197EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B03154A5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B03255F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B032555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B03255FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B032558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0325589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B03255FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B03255F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B031BEE8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B031156A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B03115A36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0311559C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B03114CE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0311B36545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4B0FED2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4B10115377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4B10116A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4B19E688B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4B19E6CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4B19E6D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4B1A1CC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4B1A1ECD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4B1979B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4B19762EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4B13A70934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BABEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BA26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BAC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BA5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BA51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BAC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4B1BABF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4B1B9F6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4B0FB3586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4B0FB6E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4B0FB24C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4B0FA993D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4B10101545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1B20990630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1B20BD3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1B20BD4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1B2A9268B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1B2A92AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1B2A92B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1B2AC8A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1B2ACAAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1B2A2592EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1B2A220EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1B2452E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1B346C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1B3462E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1B346CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1B3465955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1B34659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1B346CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1B346C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1B2BEA3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1B205F386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1B2062C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1B205E2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1B205573D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1B20BBF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B116B78E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B116B9D1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B116B9D2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B11757248B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1175728F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B11757290A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1175A88593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1175AA8D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B11750572EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B117501EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B116F32C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B117737AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B11772E2846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B11773804A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B117730D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B117730DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B117737EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B117737B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B11772B2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B116B3F186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B116B42A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B116B3E0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B116B3553D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B116B9BD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6EB78CA630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6EB7B0D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6EB7B0EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6EC18608B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6EC1864F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6EC18650A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6EC1BC4593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6EC1BE4D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B6EC11932EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B6EC115AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6EBB468934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC34B6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC341E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC34BC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC344955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC3449EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC34BAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6EC34B7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B6EC33EEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6EB752D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6EB75666AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6EB751CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6EB74913D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6EB7AF9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173457) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173458) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150729) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
==== backtrace (tid: 173494) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173495) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173499) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1C89133630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1C89376377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1C89377A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1C930CA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1C930CEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1C930CF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1C9342E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1C9344ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1C929FD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1C929C4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1C8CCD1934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CD21E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CC89846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CD274A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CCB455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CCB4EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CD25C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C9CD22015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1C9CC59B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1C88D9686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1C88DCF6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1C88D85C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1C88CFA3D7  PMPI_Init_f08         Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1C89362545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173496) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADDC9624630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADDC9867377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADDC9868A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ADDD35BA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ADDD35BEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ADDD35BF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ADDD391E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ADDD393ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ADDD2EED2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADDD2EB4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADDCD1C2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD213E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD17B846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD2194A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD1A655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD1A6EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD217C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ADDDD214015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADDDD14BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADDC928786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADDC92C06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADDC9276C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADDC91EB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADDC9853545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173498) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173456) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B29D640D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B29D6650377  gsignal               Unknown  Unknown
libc-2.17.so       00002B29D6651A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B29E84918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B29E8495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B29E84960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B29E87C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B29E87E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B29DFCD62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B29DFC9DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B29D9FAB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B29EA000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B29E9F68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B29EA0064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B29E9F9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B29E9F93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B29EA004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B29EA001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B29DFFE4B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B29D607086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B29D60A96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B29D605FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B29D5FD43D7  PMPI_Init_f08         Unknown  Unknown
==== backtrace (tid: 150710) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B29D663C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
==== backtrace (tid: 173504) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173506) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
==== backtrace (tid: 150696) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
==== backtrace (tid: 173507) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173444) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150631) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
==== backtrace (tid: 173450) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173451) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173459) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173505) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8F6BFEC630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8F6C22F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8F6C230A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8F75F828B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8F75F86F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8F75F870A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8F762E6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8F76306D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8F758B52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8F7587CEE4  mca_pml_ucx_progr     Unknown  Unknown
==== backtrace (tid: 150694) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
libopen-pal.so.40  00002B8F6FB8A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77BD8E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77B40846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77BDE4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77B6B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77B6BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77BDCC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8F77BD9015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8F77B10B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8F6BC4F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8F6BC886AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8F6BC3EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8F6BBB33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8F6C21B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B39791E4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3979427377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3979428A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B398317A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B398317EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B398317F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B39834DE593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B39834FED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3982AAD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3982A74EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B397CD82934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B398CF15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B398CE7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B398CF1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B398CEA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B398CEA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B398CF19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B398CF16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3983EA8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3978E4786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3978E806AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3978E36C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3978DAB3D7  PMPI_Init_f08         Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3979413545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB03CCC2630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB03CF05377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB03CF06A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB046C588B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB046C5CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB046C5D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB046FBC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB046FDCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB04658B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB046552EE4  mca_pml_ucx_progr     Unknown  Unknown
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
libopen-pal.so.40  00002AB040860934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB0508CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB050833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0508D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB05085E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB05085EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0508CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0508CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB047FD0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB03C92586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB03C95E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB03C914C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB03C8893D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB03CEF1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE8CDD27630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE8CDF6A377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE8CDF6BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE8D7CBD8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE8D7CC1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE8D7CC20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE8E002B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE8E004BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE8D75F02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE8D75B7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE8D18C5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E1913E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E187B846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E19194A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E18A655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E18A6EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E1917C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE8E1914015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE8E184BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE8CD98A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE8CD9C36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE8CD979C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE8CD8EE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE8CDF56545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B99184DB630  Unknown               Unknown  Unknown
libc-2.17.so       00002B991871E377  gsignal               Unknown  Unknown
libc-2.17.so       00002B991871FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B99224718B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9922475F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B99224760A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B99227D5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B99227F5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9921DA42EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9921D6BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B991C079934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B992C0CEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B992C036846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B992C0D44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B992C06155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B992C061EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B992C0D2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B992C0CF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B992C006B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B991813E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B99181776AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B991812DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B99180A23D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B991870A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA3237E3630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA323A26377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA323A27A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA32D7798B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA32D77DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA32D77E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA32DADD593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA32DAFDD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA32D0AC2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA32D073EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA327381934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F3CFE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F337846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F3D54A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F36255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F362EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F3D3C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA32F3D0015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA32F307B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA32344686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA32347F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA323435C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA3233AA3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA323A12545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2F90FD1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2F91214377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2F91215A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2F9AF678B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2F9AF6BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2F9AF6C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2F9B2CB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2F9B2EBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2F9A89A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2F9A861EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2F94B6F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4CF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4C5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4CFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4C8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4C87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4CF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2FA4CF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2F9BEB6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2F90C3486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2F90C6D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2F90C23C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2F90B983D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2F91200545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B725FDCD630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7260010377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7260011A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7269D638B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7269D67F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7269D680A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B726A0C7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B726A0E7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B72696962EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B726965DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B726396B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B726B9B9E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B726B921846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B726B9BF4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B726B94C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B726B94CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B726B9BDC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B726B9BA015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B726B8F1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B725FA3086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B725FA696AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B725FA1FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B725F9943D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B725FFFC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B15C510A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B15C534D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B15C534EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B15CF0A08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B15CF0A4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B15CF0A50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B15CF404593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B15CF424D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B15CE9D32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B15CE99AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B15C8CA8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8D0DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8C75846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8D134A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8CA055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8CA0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8D11C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B15D8D0E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B15D8C45B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B15C4D6D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B15C4DA66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B15C4D5CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B15C4CD13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B15C5339545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B054CCD8630  Unknown               Unknown  Unknown
libc-2.17.so       00002B054CF1B377  gsignal               Unknown  Unknown
libc-2.17.so       00002B054CF1CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0556C6F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0556C73F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0556C740A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0556FD3593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0556FF3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B05565A22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0556569EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0550876934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B05608CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0560833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B05608D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B056085E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B056085EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B05608CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B05608CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0557FE7B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B054C93B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B054C9746AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B054C92AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B054C89F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B054CF07545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B073DE5E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B073E0A1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B073E0A2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B075004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0750053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B07500540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0750383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B07503A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B07477272EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B07476EEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B07419FC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0751A4FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B07519B7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0751A554A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B07519E255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B07519E2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0751A53C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0751A50015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0751987B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B073DAC186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B073DAFA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B073DAB0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B073DA253D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B073E08D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFB4175B630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFB4199E377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFB4199FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFB4B6F18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFB4B6F5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFB4B6F60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFB4BA55593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFB4BA75D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFB4B0242EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFB4AFEBEE4  mca_pml_ucx_progr     Unknown  Unknown
==== backtrace (tid: 150637) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
libopen-pal.so.40  00002AFB452F9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFB553DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFB55345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFB553E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFB5537055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFB55370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFB553E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFB553DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFB4BF57B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFB413BE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFB413F76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFB413ADC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFB413223D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFB4198A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B829C33E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B829C581377  gsignal               Unknown  Unknown
libc-2.17.so       00002B829C582A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B82A62D48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B82A62D8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B82A62D90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B82A6638593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B82A6658D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B82A5C072EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B82A5BCEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B829FEDC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B82B00B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B82B001D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B82B00BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B82B004855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B82B0048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B82B00B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B82B00B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B82A7E62B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B829BFA186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B829BFDA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B829BF90C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B829BF053D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B829C56D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0A2941C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0A2965F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0A29660A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0A333B28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0A333B6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0A333B70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0A33716593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0A33736D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0A32CE52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0A32CACEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0A2CFBA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3D00BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3CF73846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3D0114A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3CF9E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3CF9EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3D00FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A3D00C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0A3CF43B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0A2907F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0A290B86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0A2906EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0A28FE33D7  PMPI_Init_f08         Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0A2964B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B030C6B9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B030C8FC377  gsignal               Unknown  Unknown
libc-2.17.so       00002B030C8FDA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B031664F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0316653F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B03166540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B03169B3593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B03169D3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0315F822EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0315F49EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0310257934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B03202BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0320222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B03202C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B032024D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B032024DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B03202BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B03202BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0317FD8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B030C31C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B030C3556AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B030C30BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B030C2803D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B030C8E8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B140D7C4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B140DA07377  gsignal               Unknown  Unknown
libc-2.17.so       00002B140DA08A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B141775A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B141775EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B141775F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1417ABE593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1417ADED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B141708D2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1417054EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1411362934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B14213DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1421345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B14213E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B142137055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1421370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B14213E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B14213DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1417FC0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B140D42786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B140D4606AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B140D416C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B140D38B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B140D9F3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B564EA04630  Unknown               Unknown  Unknown
libc-2.17.so       00002B564EC47377  gsignal               Unknown  Unknown
libc-2.17.so       00002B564EC48A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5660AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5660AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5660AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5660E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5660E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B56604232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5657E93EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B56525A2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B566265FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B56625C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B56626654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B56625F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B56625F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5662663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5662660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5657F7CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B564E66786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B564E6A06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B564E656C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B564E5CB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B564EC33545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B14F8FA7630  Unknown               Unknown  Unknown
libc-2.17.so       00002B14F91EA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B14F91EBA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1502F3D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1502F41F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1502F420A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B15032A1593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B15032C1D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B15028702EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1502837EE4  mca_pml_ucx_progr     Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEE4720D630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEE47450377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEE47451A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEE552F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEE552F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEE552F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEE55624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEE55644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEE54C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEE4FE9CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEE4ADAB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56E5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56DC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56E654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56DF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56DF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56E63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEE56E60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEE4FF85B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEE46E7086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEE46EA96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEE46E5FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEE46DD43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEE4743C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
libopen-pal.so.40  00002B14FCB45934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B150CB94E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B150CAFC846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B150CB9A4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B150CB2755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B150CB27EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B150CB98C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B150CB95015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B150CACCB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B14F8C0A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B14F8C436AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B14F8BF9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B14F8B6E3D7  PMPI_Init_f08         Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8307E9B630  Unknown               Unknown  Unknown
libc-2.17.so       00002B83080DE377  gsignal               Unknown  Unknown
libc-2.17.so       00002B83080DFA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8311E318B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8311E35F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8311E360A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8312195593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B83121B5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B83117642EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B831172BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B830BA39934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8313A87E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B83139EF846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8313A8D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8313A1A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8313A1AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8313A8BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8313A88015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B83139BFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8307AFE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8307B376AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8307AEDC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8307A623D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B83080CA545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B31331A6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B31333E9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B31333EAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B31412F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B31412F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B31412F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3141624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3141644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3140C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B313BE35EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3136D44934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3142E5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3142DC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3142E654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3142DF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3142DF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3142E63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3142E60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B313BF1EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3132E0986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3132E426AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3132DF8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3132D6D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B14F91D6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B31333D5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150740) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABEE5F61630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABEE61A4377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABEE61A5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABEF804F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABEF8053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABEF80540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABEF8383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABEF83A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABEEF82A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABEEF7F1EE4  mca_pml_ucx_progr     Unknown  Unknown
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
libopen-pal.so.40  00002ABEE9AFF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABEF9BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABEEFF7AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABEE5BC486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABEE5BFD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABEE5BB3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABEE5B283D7  PMPI_Init_f08         Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABEE6190545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA075B72630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA075DB5377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA075DB6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA07FB088B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA07FB0CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA07FB0D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA08801E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA08803ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA07F43B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA07F402EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA079710934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA089859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA0897C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA08985F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA0897EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA0897ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA08985DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA08985A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA07FEF0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA0757D586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA07580E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA0757C4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA0757393D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA075DA1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B98C3ADF630  Unknown               Unknown  Unknown
libc-2.17.so       00002B98C3D22377  gsignal               Unknown  Unknown
libc-2.17.so       00002B98C3D23A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B98CDA758B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B98CDA79F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B98CDA7A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B98CDDD9593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B98CDDF9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B98CD3A82EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B98CD36FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B98C767D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF6CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF633846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF6D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF65E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF65EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF6CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B98CF6CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B98CF603B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B98C374286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B98C377B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B98C3731C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B98C36A63D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B98C3D0E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 173510) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173511) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
==== backtrace (tid: 150702) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
==== backtrace (tid: 173513) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
==== backtrace (tid: 150706) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150645) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B86D2E30630  Unknown               Unknown  Unknown
libc-2.17.so       00002B86D3073377  gsignal               Unknown  Unknown
libc-2.17.so       00002B86D3074A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B86E0EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B86E0EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B86E0EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B86E1224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B86E1244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B86E08232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B86DBEBFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B86D69CE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B86E2A5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B86E29C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B86E2A654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B86E29F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B86E29F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B86E2A63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B86E2A60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B86DBFA8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B86D2A9386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B86D2ACC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B86D2A82C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B86D29F73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B86D305F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150634) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150741) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150697) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150644) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
==== backtrace (tid: 173509) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B78CCED3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B78CD116377  gsignal               Unknown  Unknown
libc-2.17.so       00002B78CD117A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B78D6E698B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B78D6E6DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B78D6E6E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B78D71CD593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B78D71EDD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B78D679C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B78D6763EE4  mca_pml_ucx_progr     Unknown  Unknown
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
libopen-pal.so.40  00002B78D0A71934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0AE7E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0A4F846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0AED4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0A7A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0A7AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0AEBC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B78E0AE8015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B78D7FC5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B78CCB3686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B78CCB6F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B78CCB25C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B78CCA9A3D7  PMPI_Init_f08         Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B78CD102545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC8829FA630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC882C3D377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC882C3EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC894AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC894AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC894AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC894E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC894E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC8944232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC88BE8AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC886598934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC89665FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC8965C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8966654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8965F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8965F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC896663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC896660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC88BF74B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC88265D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC8826966AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC88264CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC8825C13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC882C29545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEA352FD630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEA35540377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEA35541A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEA3F2938B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEA3F297F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEA3F2980A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEA3F5F7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEA3F617D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEA3EBC62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEA3EB8DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEA38E9B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48F15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48E7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48F1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48EA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48EA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48F19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA48F16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEA3FFC1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEA34F6086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEA34F996AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEA34F4FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEA34EC43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEA3552C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150643) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150640) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9F07656630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9F07899377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9F0789AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9F115EC8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9F115F0F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9F115F10A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9F11950593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9F11970D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9F10F1F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9F10EE6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9F0B1F4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9F13242E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9F131AA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F132484A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F131D555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F131D5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F13246C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F13243015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9F1317AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9F072B986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9F072F26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9F072A8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9F0721D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9F07885545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150735) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
==== backtrace (tid: 173512) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173514) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173515) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173519) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173521) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173516) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173518) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173523) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173520) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173522) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 173517) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150633) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150664) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150739) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC748451630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC748694377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC748695A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC7523E78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC7523EBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC7523EC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC75274B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC75276BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC751D1A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC751CE1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC74BFEF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C0B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C01D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C0BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C04855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C0B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC75C0B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC753F75B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC7480B486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC7480ED6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC7480A3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC7480183D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC748680545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150709) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AAEB93A4630  Unknown               Unknown  Unknown
libc-2.17.so       00002AAEB95E7377  gsignal               Unknown  Unknown
libc-2.17.so       00002AAEB95E8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AAEC333A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AAEC333EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AAEC333F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AAEC369E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AAEC36BED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AAEC2C6D2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AAEC2C34EE4  mca_pml_ucx_progr     Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
libopen-pal.so.40  00002AAEBCF42934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCF90E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCEF8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCF964A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCF2355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCF23EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCF94C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AAECCF91015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AAECCEC8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AAEB900786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AAEB90406AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AAEB8FF6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AAEB8F6B3D7  PMPI_Init_f08         Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AAEB95D3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150672) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150666) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150657) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150734) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150690) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150651) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150665) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150730) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150667) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150649) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150639) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150737) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150728) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150676) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150698) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150731) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150695) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150700) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150685) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150683) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150733) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150732) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150674) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150708) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5C5191D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5C51B60377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5C51B61A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5C5B8B38B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5C5B8B7F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5C5B8B80A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5C5BC17593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5C5BC37D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5C5B1E62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5C5B1ADEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5C554BB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5C655F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5C6555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5C655FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5C6558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5C65589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5C655FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5C655F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5C5BEFEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5C5158086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5C515B96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5C5156FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5C514E43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5C51B4C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150703) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADBC505C630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADBC529F377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADBC52A0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ADBCEFF28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ADBCEFF6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ADBCEFF70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ADBCF356593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ADBCF376D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ADBCE9252EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADBCE8ECEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADBC8BFA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8CF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8C5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8CFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8C8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8C87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8CF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ADBD8CF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADBCFF41B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADBC4CBF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADBC4CF86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADBC4CAEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADBC4C233D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADBC528B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE24A143630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE24A386377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE24A387A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE25C2658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE25C269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE25C26A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE25C599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE25C5B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE253A0C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE2539D3EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE24DCE1934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DDD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DD3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DDDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DD6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DD67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DDD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE25DDD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE253F46B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE249DA686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE249DDF6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE249D95C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE249D0A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE24A372545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150701) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6999EB5630  Unknown               Unknown  Unknown
libc-2.17.so       00002B699A0F8377  gsignal               Unknown  Unknown
libc-2.17.so       00002B699A0F9A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B69AC04F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B69AC053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B69AC0540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B69AC383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B69AC3A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B69A377E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B69A3745EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B699DA53934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADBBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADB26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADBC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADB5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADB51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADBC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B69ADBBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B69A3ECEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6999B1886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6999B516AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6999B07C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6999A7C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B699A0E4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0013250630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0013493377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0013494A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B00212F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B00212F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B00212F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0021624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0021644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0020C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B001BEDFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0016DEE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0022E5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0022DC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0022E654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0022DF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0022DF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0022E63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0022E60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B001BFC8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0012EB386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0012EEC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0012EA2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0012E173D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B001347F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B948377E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B94839C1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B94839C2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B948D7148B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B948D718F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B948D7190A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B948DA78593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B948DA98D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B948D0472EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B948D00EEE4  mca_pml_ucx_progr     Unknown  Unknown
==== backtrace (tid: 150704) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
libopen-pal.so.40  00002B948731C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B948F36AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B948F2D2846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B948F3704A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B948F2FD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B948F2FDEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B948F36EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B948F36B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B948F2A2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B94833E186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B948341A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B94833D0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B94833453D7  PMPI_Init_f08         Unknown  Unknown
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B94839AD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE51A632630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE51A875377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE51A876A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE52C6EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE52C6F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE52C6F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE52CA23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE52CA43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE52C0222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE523EC2EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE51E1D0934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E25EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E1C6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E2644A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E1F155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E1F1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E262C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE52E25F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE523FABB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE51A29586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE51A2CE6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE51A284C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE51A1F93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE51A861545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B000F60A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B000F84D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B000F84EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B00195A08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B00195A4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B00195A50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0019904593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0019924D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0018ED32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0018E9AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B00131A8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B001B1F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B001B15E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B001B1FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B001B18955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B001B189EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B001B1FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B001B1F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B001B12EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B000F26D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B000F2A66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B000F25CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B000F1D13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B000F839545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150718) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150707) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150699) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE7CDFA1630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE7CE1E4377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE7CE1E5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE7E004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE7E0053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE7E00540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE7E0383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE7E03A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE7D786A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE7D7831EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE7D1B3F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE7E1BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE7D7FBAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE7CDC0486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE7CDC3D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE7CDBF3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE7CDB683D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE7CE1D0545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9D1E467630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9D1E6AA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9D1E6ABA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9D304918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9D30495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9D304960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9D307C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9D307E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9D27D302EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9D27CF7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9D22005934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9D3205FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9D31FC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D320654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D31FF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D31FF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D32063C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D32060015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9D31F97B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9D1E0CA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9D1E1036AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9D1E0B9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9D1E02E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9D1E696545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150679) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150705) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AACE180C630  Unknown               Unknown  Unknown
libc-2.17.so       00002AACE1A4F377  gsignal               Unknown  Unknown
libc-2.17.so       00002AACE1A50A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AACEB7A28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AACEB7A6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AACEB7A70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AACEBB06593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AACEBB26D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AACEB0D52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AACEB09CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AACE53AA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AACF540AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AACF5372846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AACF54104A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AACF539D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AACF539DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AACF540EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AACF540B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AACF5342B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AACE146F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AACE14A86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AACE145EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AACE13D33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AACE1A3B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B25A8D3B630  Unknown               Unknown  Unknown
libc-2.17.so       00002B25A8F7E377  gsignal               Unknown  Unknown
libc-2.17.so       00002B25A8F7FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B25B2CD18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B25B2CD5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B25B2CD60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B25B3035593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B25B3055D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B25B26042EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B25B25CBEE4  mca_pml_ucx_progr     Unknown  Unknown
==== backtrace (tid: 150682) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
libopen-pal.so.40  00002B25AC8D9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC92AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC892846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC9304A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC8BD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC8BDEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC92EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B25BC92B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B25BC862B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B25A899E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B25A89D76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B25A898DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B25A89023D7  PMPI_Init_f08         Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B25A8F6A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150638) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150727) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150738) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
==== backtrace (tid: 150642) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150736) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFBB7FCA630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFBB820D377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFBB820EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFBC1F608B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFBC1F64F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFBC1F650A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFBC22C4593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFBC22E4D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFBC18932EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFBC185AEE4  mca_pml_ucx_progr     Unknown  Unknown
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
libopen-pal.so.40  00002AFBBBB68934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3BB6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3B1E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3BBC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3B4955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3B49EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3BBAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBC3BB7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFBC3AEEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFBB7C2D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFBB7C666AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFBB7C1CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFBB7B913D7  PMPI_Init_f08         Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFBB81F9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150726) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150636) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150687) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC47C508630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC47C74B377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC47C74CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC48649E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC4864A2F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC4864A30A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC486802593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC486822D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC485DD12EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC485D98EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC4800A6934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC490100E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC490068846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC4901064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC49009355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC490093EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC490104C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC490101015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC490038B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC47C16B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC47C1A46AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC47C15AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC47C0CF3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC47C737545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150632) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF125C0A630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF125E4D377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF125E4EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF12FBA08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF12FBA4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF12FBA50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF13801E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF13803ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF12F4D32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF12F49AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF1297A8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF139859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF1397C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF13985F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF1397EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF1397ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF13985DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF13985A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF12FF88B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF12586D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF1258A66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF12585CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF1257D13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF125E39545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150725) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0F92A91630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0F92CD4377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0F92CD5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0FA4AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0FA4AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0FA4AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0FA4E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0FA4E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0FA44232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0F9BF20EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0F9662F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA668CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA65F4846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA66924A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA661F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA661FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA6690C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FA668D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0FA65C4B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0F926F486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0F9272D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0F926E3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0F926583D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0F92CC0545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150630) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150692) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
==== backtrace (tid: 150691) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABFEF741630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABFEF984377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABFEF985A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABFF96D78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABFF96DBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABFF96DC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABFF9A3B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABFF9A5BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABFF900A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABFF8FD1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABFF32DF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB32DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB295846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB3334A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB2C055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB2C0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB331C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABFFB32E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABFFB265B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABFEF3A486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABFEF3DD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABFEF393C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABFEF3083D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABFEF970545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACA403E6630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACA40629377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACA4062AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACA4A37C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACA4A380F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACA4A3810A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACA4A6E0593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACA4A700D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACA49CAF2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACA49C76EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACA43F84934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACA540B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACA5401D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACA540BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACA5404855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACA54048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACA540B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACA540B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACA4BF0AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACA4004986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACA400826AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACA40038C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACA3FFAD3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACA40615545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC0C2456630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC0C2699377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC0C269AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC0D44918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC0D4495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC0D44960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC0D47C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC0D47E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC0CBD1F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC0CBCE6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC0C5FF4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D604BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D5FB3846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D60514A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D5FDE55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D5FDEEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D604FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC0D604C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC0D5F83B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC0C20B986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC0C20F26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC0C20A8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC0C201D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC0C2685545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150641) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFC119D0630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFC11C13377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFC11C14A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFC1B9668B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFC1B96AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFC1B96B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFC1BCCA593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFC1BCEAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFC1B2992EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFC1B260EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFC1556E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFC255F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC255FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC25589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC255FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC255F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFC1BFB1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFC1163386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFC1166C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFC11622C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFC115973D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFC11BFF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7BCA061630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7BCA2A4377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7BCA2A5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7BDC04F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7BDC053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7BDC0540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7BDC383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7BDC3A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7BD392A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7BD38F1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7BCDBFF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDC4FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDBB7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDC554A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDBE255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDBE2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDC53C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BDDC50015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7BDDB87B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7BC9CC486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7BC9CFD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7BC9CB3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7BC9C283D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7BCA290545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFD4C67A630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFD4C8BD377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFD4C8BEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFD566108B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFD56614F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFD566150A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFD56974593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFD56994D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFD55F432EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFD55F0AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFD50218934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFD602BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFD60222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFD602C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFD6024D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFD6024DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFD602BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFD602BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFD57F99B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFD4C2DD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFD4C3166AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFD4C2CCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFD4C2413D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFD4C8A9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC501810630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC501A53377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC501A54A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC50B7A68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC50B7AAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC50B7AB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC50BB0A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC50BB2AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC50B0D92EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC50B0A0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC5053AE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC51540AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC515372846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC5154104A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC51539D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC51539DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC51540EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC51540B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC515342B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC50147386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC5014AC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC501462C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC5013D73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC501A3F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B13DC788630  Unknown               Unknown  Unknown
libc-2.17.so       00002B13DC9CB377  gsignal               Unknown  Unknown
libc-2.17.so       00002B13DC9CCA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B13E671E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B13E6722F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B13E67230A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B13E6A82593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B13E6AA2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B13E60512EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B13E6018EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B13E0326934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B13F04C1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B13F0429846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B13F04C74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B13F045455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B13F0454EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B13F04C5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B13F04C2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B13E7EA0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B13DC3EB86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B13DC4246AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B13DC3DAC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B13DC34F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B13DC9B7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7786F0C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B778714F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7787150A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7794EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7794EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7794EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7795224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7795244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B77948232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B778FF9BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B778AAAA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7796B00E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7796A68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7796B064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7796A9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7796A93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7796B04C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7796B01015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7796A38B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7786B6F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7786BA86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7786B5EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7786AD33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B778713B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150635) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AED33508630  Unknown               Unknown  Unknown
libc-2.17.so       00002AED3374B377  gsignal               Unknown  Unknown
libc-2.17.so       00002AED3374CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AED3D49E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AED3D4A2F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AED3D4A30A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AED3D802593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AED3D822D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AED3CDD12EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AED3CD98EE4  mca_pml_ucx_progr     Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB78C002630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB78C245377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB78C246A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB795F988B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB795F9CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB795F9D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB7962FC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB79631CD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB7958CB2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB795892EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AED370A6934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F0F4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F05C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F0FA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F08755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F087EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F0F8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AED3F0F5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AED3F02CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AED3316B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AED331A46AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AED3315AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AED330CF3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AED33737545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
libopen-pal.so.40  00002AB78FBA0934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB797BEEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB797B56846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB797BF44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB797B8155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB797B81EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB797BF2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB797BEF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB797B26B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB78BC6586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB78BC9E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB78BC54C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB78BBC93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB78C231545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5056A56630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5056C99377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5056C9AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5068AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5068AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5068AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5068E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5068E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B50684232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B505FEE5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B505A5F4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B506A65FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B506A5C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B506A6654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B506A5F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B506A5F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B506A663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B506A660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B505FFCEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B50566B986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B50566F26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B50566A8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B505661D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5056C85545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150628) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B285B5BA630  Unknown               Unknown  Unknown
libc-2.17.so       00002B285B7FD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B285B7FEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B28655508B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2865554F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B28655550A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B28658B4593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B28658D4D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2864E832EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2864E4AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B285F158934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B28671A6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B286710E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B28671AC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B286713955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2867139EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B28671AAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B28671A7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B28670DEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B285B21D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B285B2566AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B285B20CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B285B1813D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B285B7E9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8E60982630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8E60BC5377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8E60BC6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8E6A9188B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8E6A91CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8E6A91D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8E6AC7C593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8E6AC9CD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8E6A24B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8E6A212EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8E64520934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8E7456EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8E744D6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8E745744A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8E7450155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8E74501EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8E74572C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8E7456F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8E744A6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8E605E586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8E6061E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8E605D4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8E605493D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8E60BB1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150658) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7BDC556630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7BDC799377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7BDC79AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7BE64ED8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7BE64F1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7BE64F20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7BE6851593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7BE6871D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7BE5E202EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7BE5DE7EE4  mca_pml_ucx_progr     Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AD180E60630  Unknown               Unknown  Unknown
libc-2.17.so       00002AD1810A3377  gsignal               Unknown  Unknown
libc-2.17.so       00002AD1810A4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AD18ADF68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AD18ADFAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AD18ADFB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AD18B15A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AD18B17AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AD18A7292EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AD18A6F0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7BE00F4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF0146E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF00AE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF014C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF00D955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF00D9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF014AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7BF0147015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7BF007EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7BDC1B986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7BDC1F26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7BDC1A8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7BDC11D3D7  PMPI_Init_f08         Unknown  Unknown
libopen-pal.so.40  00002AD1849FE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AD194AE7E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AD194A4F846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AD194AED4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AD194A7A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD194A7AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD194AEBC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AD194AE8015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AD18BF52B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AD180AC386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AD180AFC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AD180AB2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AD180A273D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7BDC785545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AD18108F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5D9BC5F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5D9BEA2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5D9BEA3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5DA5BF58B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5DA5BF9F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5DA5BFA0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5DA5F59593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5DA5F79D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5DA55282EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5DA54EFEE4  mca_pml_ucx_progr     Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3A00176630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3A003B9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3A003BAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3A0A10C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3A0A110F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3A0A1110A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3A0A470593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3A0A490D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3A09A3F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3A09A06EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5D9F7FD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA784BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA77B3846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA78514A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA77DE55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA77DEEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA784FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5DA784C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5DA7783B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5D9B8C286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5D9B8FB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5D9B8B1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5D9B8263D7  PMPI_Init_f08         Unknown  Unknown
libopen-pal.so.40  00002B3A03D14934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BD62E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BCCA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BD684A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BCF555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BCF5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BD66C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A0BD63015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3A0BC9AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B39FFDD986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B39FFE126AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B39FFDC8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B39FFD3D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5D9BE8E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3A003A5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFE3188D630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE31AD0377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFE31AD1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFE3B8248B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFE3B828F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFE3B8290A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFE3BB88593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFE3BBA8D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFE3B1572EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFE3B11EEE4  mca_pml_ucx_progr     Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3F1FF3C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3F2017F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3F20180A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3F29ED28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3F29ED6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3F29ED70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3F2A236593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3F2A256D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3F298052EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3F297CCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFE3542B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFE4547EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFE453E6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE454844A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE4541155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE45411EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE45482C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE4547F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFE453B6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFE314F086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFE315296AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFE314DFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFE314543D7  PMPI_Init_f08         Unknown  Unknown
libopen-pal.so.40  00002B3F23ADA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BB28E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BA90846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BB2E4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BABB55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BABBEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BB2CC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3F2BB29015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3F2BA60B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3F1FB9F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3F1FBD86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3F1FB8EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3F1FB033D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE31ABC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3F2016B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFBCB0D1630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFBCB314377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFBCB315A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFBD90928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFBD9096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFBD90970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFBD93D3593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFBD93F3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFBD3D992EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFBD3D60EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFBCEC6F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDACBCE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDAC24846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDACC24A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDAC4F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDAC4FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDACC0C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFBDACBD015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFBDABF4B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFBCAD3486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFBCAD6D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFBCAD23C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFBCAC983D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFBCB300545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150677) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8BA860C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8BA884F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8BA8850A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8BB25A28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8BB25A6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8BB25A70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8BB2906593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8BB2926D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8BB1ED52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8BB1E9CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8BAC1AA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC2BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC2C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC24D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC24DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC2BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8BBC2BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8BB3F2BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8BA826F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8BA82A86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8BA825EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8BA81D33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8BA883B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1F4BD4F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1F4BF92377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1F4BF93A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1F55CE68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1F55CEAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1F55CEB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1F5604A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1F5606AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1F556192EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1F555E0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1F4F8ED934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1F5793CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1F578A4846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F579424A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F578CF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F578CFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F57940C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F5793D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1F57874B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1F4B9B286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1F4B9EB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1F4B9A1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1F4B9163D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1F4BF7E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4F1EC89630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4F1EECC377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4F1EECDA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4F2CC928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4F2CC96F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4F2CC970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4F2CFC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4F2CFE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4F27D522EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4F27D19EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4F22827934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E879E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E7E1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E87F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E80C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E80CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E87DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F2E87A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4F2E7B1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4F1E8EC86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4F1E9256AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4F1E8DBC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4F1E8503D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4F1EEB8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA995069630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA9952AC377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA9952ADA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA99EFFF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA99F003F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA99F0040A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA99F363593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA99F383D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA99E9322EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA99E8F9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA998C07934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8CF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8C5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8CFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8C8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8C87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8CF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA9A8CF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA99FF4EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA994CCC86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA994D056AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA994CBBC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA994C303D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA995298545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9A25EB9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9A260FC377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9A260FDA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9A3804F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9A38053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9A380540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9A38383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9A383A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9A2F7822EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9A2F749EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9A29A57934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9A39BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9A2FED2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9A25B1C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9A25B556AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9A25B0BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9A25A803D7  PMPI_Init_f08         Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE182EAE630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE1830F1377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE1830F2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE190EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE190EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE190EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE191224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE191244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE1908232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE18BF3EEE4  mca_pml_ucx_progr     Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9A260E8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
libopen-pal.so.40  00002AE186A4C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE192AAAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE192A12846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE192AB04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE192A3D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE192A3DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE192AAEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE192AAB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE1929E2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE182B1186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE182B4A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE182B00C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE182A753D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE1830DD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEB1972C630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEB1996F377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEB19970A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEB236C28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEB236C6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEB236C70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEB23A26593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEB23A46D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEB22FF52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEB22FBCEE4  mca_pml_ucx_progr     Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA3B2C7E630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA3B2EC1377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA3B2EC2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA3C0C928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA3C0C96F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA3C0C970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA3C0FC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA3C0FE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA3BBD462EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA3BBD0DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEB1D2CA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D3DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D3E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D37055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D3E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEB2D3DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEB23F28B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEB1938F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEB193C86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEB1937EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEB192F33D7  PMPI_Init_f08         Unknown  Unknown
libopen-pal.so.40  00002BA3B681C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C2870E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C27D8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C28764A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C280355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C2803EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C2874C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3C2871015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA3C27A8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA3B28E186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA3B291A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA3B28D0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA3B28453D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEB1995B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA3B2EAD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B577706D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B57772B0377  gsignal               Unknown  Unknown
libc-2.17.so       00002B57772B1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B57850928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5785096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B57850970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B57853C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B57853E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B577FD352EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B577FCFCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B577AC0B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5786C60E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5786BC8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5786C664A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5786BF355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5786BF3EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5786C64C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5786C61015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5786B98B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5776CD086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5776D096AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5776CBFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5776C343D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B577729C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AD598020630  Unknown               Unknown  Unknown
libc-2.17.so       00002AD598263377  gsignal               Unknown  Unknown
libc-2.17.so       00002AD598264A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AD5A1FB68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AD5A1FBAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AD5A1FBB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AD5A231A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AD5A233AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AD5A18E92EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AD5A18B0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AD59BBBE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3C0CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3B74846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3C124A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3B9F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3B9FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3C10C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AD5A3C0D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AD5A3B44B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AD597C8386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AD597CBC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AD597C72C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AD597BE73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AD59824F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150724) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACBBCCB3630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACBBCEF6377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACBBCEF7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACBC6C498B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACBC6C4DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACBC6C4E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACBC6FAD593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACBC6FCDD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACBC657C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACBC6543EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACBC0851934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD08CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD0833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD08D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD085E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD085EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD08CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACBD08CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACBC7FC1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACBBC91686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACBBC94F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACBBC905C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACBBC87A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACBBCEE2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE01C539630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE01C77C377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE01C77DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE0264CF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE0264D3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE0264D40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE026833593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE026853D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE025E022EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE025DC9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE0200D7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE030127E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE03008F846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE03012D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0300BA55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0300BAEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE03012BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE030128015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE03005FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE01C19C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE01C1D56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE01C18BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE01C1003D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE01C768545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150622) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150653) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AACD7505630  Unknown               Unknown  Unknown
libc-2.17.so       00002AACD7748377  gsignal               Unknown  Unknown
libc-2.17.so       00002AACD7749A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AACE149B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AACE149FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AACE14A00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AACE17FF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AACE181FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AACE0DCE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AACE0D95EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AACDB0A3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AACE30F1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AACE3059846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AACE30F74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AACE308455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AACE3084EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AACE30F5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AACE30F2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AACE3029B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AACD716886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AACD71A16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AACD7157C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AACD70CC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AACD7734545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150693) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150662) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE2A9338630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE2A957B377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE2A957CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE2B32CE8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE2B32D2F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE2B32D30A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE2B3632593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE2B3652D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE2B2C012EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE2B2BC8EE4  mca_pml_ucx_progr     Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AECEA629630  Unknown               Unknown  Unknown
libc-2.17.so       00002AECEA86C377  gsignal               Unknown  Unknown
libc-2.17.so       00002AECEA86DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AECFC6EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AECFC6F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AECFC6F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AECFCA23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AECFCA43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AECFC0222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AECF3EB9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE2ACED6934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCF2EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCE96846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCF344A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCEC155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCEC1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCF32C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2BCF2F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE2BCE66B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE2A8F9B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE2A8FD46AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE2A8F8AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE2A8EFF3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE2A9567545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
libopen-pal.so.40  00002AECEE1C7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE25EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE1C6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE2644A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE1F155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE1F1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE262C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AECFE25F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AECF3FA2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AECEA28C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AECEA2C56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AECEA27BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AECEA1F03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AECEA858545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC3D5B62630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC3D5DA5377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC3D5DA6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC3DFAF88B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC3DFAFCF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC3DFAFD0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC3E801E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC3E803ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC3DF42B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC3DF3F2EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC3D9700934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E9859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E97C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E985F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E97EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E97ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E985DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3E985A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC3DFEE0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC3D57C586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC3D57FE6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC3D57B4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC3D57293D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC3D5D91545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B14FDE80630  Unknown               Unknown  Unknown
libc-2.17.so       00002B14FE0C3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B14FE0C4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B151004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1510053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B15100540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1510383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B15103A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B15077492EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1507710EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1501A1E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1511A6DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B15119D5846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1511A734A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1511A0055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1511A00EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1511A71C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1511A6E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B15119A5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B14FDAE386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B14FDB1C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B14FDAD2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B14FDA473D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B14FE0AF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B48B0FA4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B48B11E7377  gsignal               Unknown  Unknown
libc-2.17.so       00002B48B11E8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B48BAF3B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B48BAF3FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B48BAF400A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B48BB29F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B48BB2BFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B48BA86E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B48BA835EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B48B4B42934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4B91E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4AF9846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4B974A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4B2455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4B24EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4B95C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B48C4B92015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B48C4AC9B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B48B0C0786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B48B0C406AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B48B0BF6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B48B0B6B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B48B11D3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B984A8BC630  Unknown               Unknown  Unknown
libc-2.17.so       00002B984AAFF377  gsignal               Unknown  Unknown
libc-2.17.so       00002B984AB00A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B985C8928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B985C896F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B985C8970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B985CBC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B985CBE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9853D852EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9853D4CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B984E45A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B985E4ABE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B985E413846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B985E4B14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B985E43E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B985E43EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B985E4AFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B985E4AC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B985E3E3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B984A51F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B984A5586AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B984A50EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B984A4833D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B984AAEB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9DE18A4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9DE1AE7377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9DE1AE8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9DEB83A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9DEB83EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9DEB83F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9DEBB9E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9DEBBBED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9DEB16D2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9DEB134EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9DE5442934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF548EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF53F6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF54944A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF542155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF5421EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF5492C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9DF548F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9DF53C6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9DE150786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9DE15406AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9DE14F6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9DE146B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9DE1AD3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B577D00E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B577D251377  gsignal               Unknown  Unknown
libc-2.17.so       00002B577D252A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5786FA48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5786FA8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5786FA90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5787308593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5787328D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B57868D72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B578689EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5780BAC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5790CF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5790C5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5790CFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5790C8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5790C87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5790CF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5790CF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5787EF3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B577CC7186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B577CCAA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B577CC60C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B577CBD53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B577D23D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8AC3F46630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8AC4189377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8AC418AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8ACDEDC8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8ACDEE0F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8ACDEE10A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8ACE240593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8ACE260D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8ACD80F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8ACD7D6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8AC7AE4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFB32E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFA9A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFB384A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFAC555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFAC5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFB36C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8ACFB33015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8ACFA6AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8AC3BA986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8AC3BE26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8AC3B98C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8AC3B0D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8AC4175545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFE7E9C6630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE7EC09377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFE7EC0AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFE90AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFE90AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFE90AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFE90E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFE90E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFE904232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFE87E55EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFE82564934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFE9265FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFE925C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE926654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE925F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE925F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE92663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE92660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFE87F3EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFE7E62986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFE7E6626AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFE7E618C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFE7E58D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE7EBF5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150680) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150681) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACDF7F29630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACDF816C377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACDF816DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACE01EBF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACE01EC3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACE01EC40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACE02223593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACE02243D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACE017F22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACE017B9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACDFBAC7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03B15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03A7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03B1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03AA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03AA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03B19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE03B16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACE03A4DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACDF7B8C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACDF7BC56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACDF7B7BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACDF7AF03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACDF8158545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8D81C05630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8D81E48377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8D81E49A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8D8BB9B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8D8BB9FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8D8BBA00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8D9401E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8D9403ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8D8B4CE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8D8B495EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8D857A3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8D95859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8D957C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D9585F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D957EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D957ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D9585DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D9585A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8D8BF83B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8D8186886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8D818A16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8D81857C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8D817CC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8D81E34545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150684) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF87E04B630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF87E28E377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF87E28FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF89004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF890053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF8900540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF890383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF8903A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF8879142EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF8878DBEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF881BE9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF891C36E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF891B9E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF891C3C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF891BC955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF891BC9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF891C3AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF891C37015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF891B6EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF87DCAE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF87DCE76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF87DC9DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF87DC123D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF87E27A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150686) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
==== backtrace (tid: 150689) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150678) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2D1EA3C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2D1EC7F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2D1EC80A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2D30AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2D30AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2D30AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2D30E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2D30E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2D304232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2D27ECBEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2D225DA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2D3265FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2D325C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2D326654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2D325F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2D325F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2D32663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2D32660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2D27FB4B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2D1E69F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2D1E6D86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2D1E68EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2D1E6033D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2D1EC6B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150671) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B771DB6F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B771DDB2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B771DDB3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7727B058B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7727B09F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7727B0A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B773001E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B773003ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B77274382EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B77273FFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B772170D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7731859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B77317C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B773185F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B77317EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B77317ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B773185DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B773185A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7727EEDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B771D7D286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B771D80B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B771D7C1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B771D7363D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B771DD9E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACF521AA630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACF523ED377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACF523EEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACF642658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACF64269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACF6426A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACF64599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACF645B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACF5BA732EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACF5BA3AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACF55D48934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65DD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65D3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65DDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65D6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65D67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65DD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACF65DD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACF5BFADB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACF51E0D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACF51E466AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACF51DFCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACF51D713D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACF523D9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150663) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150675) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150668) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
==== backtrace (tid: 150669) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150670) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6DC3E96630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6DC40D9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6DC40DAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6DCDE2C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6DCDE30F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6DCDE310A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6DCE190593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6DCE1B0D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B6DCD75F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B6DCD726EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6DC7A34934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCFA82E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCF9EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCFA884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCFA1555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCFA15EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCFA86C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6DCFA83015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B6DCF9BAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6DC3AF986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6DC3B326AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6DC3AE8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6DC3A5D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6DC40C5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B75BE177630  Unknown               Unknown  Unknown
libc-2.17.so       00002B75BE3BA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B75BE3BBA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B75D02658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B75D0269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B75D026A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B75D0599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B75D05B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B75C7A402EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B75C7A07EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B75C1D15934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1DD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1D3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1DDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1D6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1D67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1DD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B75D1DD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B75C7F7AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B75BDDDA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B75BDE136AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B75BDDC9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B75BDD3E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B75BE3A6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150673) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADF0249E630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADF026E1377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADF026E2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ADF144918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ADF14495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ADF144960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ADF147C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ADF147E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ADF0BD672EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADF0BD2EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADF0603C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ADF16091E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ADF15FF9846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ADF160974A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ADF1602455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADF16024EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADF16095C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ADF16092015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADF15FC9B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADF0210186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADF0213A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADF020F0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADF020653D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADF026CD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B473E196630  Unknown               Unknown  Unknown
libc-2.17.so       00002B473E3D9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B473E3DAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B47502658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4750269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B475026A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4750599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B47505B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4747A5F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4747A26EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4741D34934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4751DD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4751D3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4751DDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4751D6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4751D67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4751DD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4751DD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4747F99B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B473DDF986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B473DE326AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B473DDE8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B473DD5D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B473E3C5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150618) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0A168C3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0A16B06377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0A16B07A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0A288928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0A28896F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0A288970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0A28BC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0A28BE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0A1FD8B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0A1FD52EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0A1A461934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A4AFE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A417846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A4B54A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A44255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A442EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A4B3C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0A2A4B0015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0A2A3E7B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0A1652686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0A1655F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0A16515C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0A1648A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0A16AF2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0852C24630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0852E67377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0852E68A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0860C928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0860C96F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0860C970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0860FC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0860FE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B085BCEC2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B085BCB3EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B08567C2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B086281AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0862782846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B08628204A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B08627AD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B08627ADEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B086281EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B086281B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0862752B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B085288786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B08528C06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0852876C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B08527EB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0852E53545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEA70B2A630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEA70D6D377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEA70D6EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEA7AAC08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEA7AAC4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEA7AAC50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEA7AE24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEA7AE44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEA7A3F32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEA7A3BAEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEA746C8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEA84725E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEA8468D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA8472B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA846B855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA846B8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA84729C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEA84726015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEA8465DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEA7078D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEA707C66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEA7077CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEA706F13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEA70D59545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B504531F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5045562377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5045563A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B504F2B58B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B504F2B9F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B504F2BA0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B504F619593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B504F639D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B504EBE82EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B504EBAFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5048EBD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5058F15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5058E7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5058F1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5058EA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5058EA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5058F19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5058F16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B504FFE3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5044F8286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5044FBB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5044F71C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5044EE63D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B504554E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150716) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9BF0CEC630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9BF0F2F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9BF0F30A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9BFAC838B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9BFAC87F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9BFAC880A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9BFAFE7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9BFB007D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9BFA5B62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9BFA57DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9BF488A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9C048E4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9C0484C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9C048EA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9C0487755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9C04877EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9C048E8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9C048E5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9C0481CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9BF094F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9BF09886AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9BF093EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9BF08B33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9BF0F1B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150650) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B07C494F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B07C4B92377  gsignal               Unknown  Unknown
libc-2.17.so       00002B07C4B93A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B07CE8E58B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B07CE8E9F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B07CE8EA0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B07CEC49593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B07CEC69D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B07CE2182EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B07CE1DFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B07C84ED934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B07D853CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B07D84A4846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B07D85424A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B07D84CF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B07D84CFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B07D8540C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B07D853D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B07D8474B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B07C45B286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B07C45EB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B07C45A1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B07C45163D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B07C4B7E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B583788A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5837ACD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5837ACEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B58418218B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5841825F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B58418260A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5841B85593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5841BA5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B58411542EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B584111BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B583B428934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5843477E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B58433DF846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B584347D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B584340A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B584340AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B584347BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5843478015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B58433AFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B58374ED86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B58375266AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B58374DCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B58374513D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5837AB9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE17DD9E630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE17DFE1377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE17DFE2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE19004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE190053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE1900540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE187D34593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE187D54D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE1876682EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE18762FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE18193C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE19198DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE1918F5846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE1919934A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE19192055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE191920EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE191991C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE19198E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE187FEAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE17DA0186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE17DA3A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE17D9F0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE17D9653D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE17DFCD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150652) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B81A197A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B81A1BBD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B81A1BBEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B81AB9108B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B81AB914F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B81AB9150A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B81ABC74593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B81ABC94D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B81AB2432EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B81AB20AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B81A5518934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B81B55F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B81B555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B81B55FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B81B558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B81B5589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B81B55FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B81B55F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B81ABF5BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B81A15DD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B81A16166AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B81A15CCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B81A15413D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B81A1BA9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B57A4FC0630  Unknown               Unknown  Unknown
libc-2.17.so       00002B57A5203377  gsignal               Unknown  Unknown
libc-2.17.so       00002B57A5204A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B57AEF568B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B57AEF5AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B57AEF5B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B57AF2BA593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B57AF2DAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B57AE8892EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B57AE850EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B57A8B5E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8CF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8C5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8CFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8C8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8C87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8CF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B57B8CF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B57AFEA5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B57A4C2386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B57A4C5C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B57A4C12C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B57A4B873D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B57A51EF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150656) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4991E60630  Unknown               Unknown  Unknown
libc-2.17.so       00002B49920A3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B49920A4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B49A404F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B49A4053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B49A40540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B49A4383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B49A43A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B499B7292EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B499B6F0EE4  mca_pml_ucx_progr     Unknown  Unknown
=================================
libopen-pal.so.40  00002B49959FE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B49A5A4FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B49A59B7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B49A5A554A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B49A59E255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B49A59E2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B49A5A53C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B49A5A50015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B49A5987B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4991AC386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4991AFC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4991AB2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4991A273D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B499208F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150646) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150647) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B51ED114630  Unknown               Unknown  Unknown
libc-2.17.so       00002B51ED357377  gsignal               Unknown  Unknown
libc-2.17.so       00002B51ED358A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B51F70AA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B51F70AEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B51F70AF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B51F740E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B51F742ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B51F69DD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B51F69A4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B51F0CB2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5200D0DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5200C75846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5200D134A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5200CA055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5200CA0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5200D11C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5200D0E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5200C45B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B51ECD7786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B51ECDB06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B51ECD66C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B51ECCDB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B51ED343545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4196142630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4196385377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4196386A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B41A82658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B41A8269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B41A826A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B41A8599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B41A85B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B419FA0B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B419F9D2EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4199CE0934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9DD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9D3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9DDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9D6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9D67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9DD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B41A9DD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B419FF45B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4195DA586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4195DDE6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4195D94C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4195D093D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4196371545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150648) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150720) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB4D06C0630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB4D0903377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB4D0904A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB4DA6578B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB4DA65BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB4DA65C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB4DA9BB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB4DA9DBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB4D9F8A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB4D9F51EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB4D425E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E42BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E4222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E42C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E424D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E424DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E42BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB4E42BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB4DBFE0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB4D032386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB4D035C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB4D0312C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB4D02873D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB4D08EF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150654) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150655) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150723) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150624) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150659) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150660) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150661) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150713) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6767A6D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6767CB0377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6767CB1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6771A038B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6771A07F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6771A080A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6771D67593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6771D87D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B67713362EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B67712FDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B676B60B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6773659E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B67735C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B677365F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B67735EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B67735ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B677365DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B677365A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B6773591B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B67676D086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B67677096AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B67676BFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B67676343D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6767C9C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B66D6EB3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B66D70F6377  gsignal               Unknown  Unknown
libc-2.17.so       00002B66D70F7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B66E4EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B66E4EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B66E4EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B66E5224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B66E5244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B66E48232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B66DFF42EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B66DAA51934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6AAAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6A12846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6AB04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6A3D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6A3DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6AAEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B66E6AAB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B66E69E2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B66D6B1686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B66D6B4F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B66D6B05C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B66D6A7A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B66D70E2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150722) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150721) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150617) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150717) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B868693D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8686B80377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8686B81A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B86988DA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B86988DEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B86988DF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8698C3E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8698C5ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B86984232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B868FDCDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B868A4DB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B869A52DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B869A495846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B869A5334A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B869A4C055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B869A4C0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B869A531C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B869A52E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B869A465B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B86865A086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B86865D96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B868658FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B86865043D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8686B6C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150714) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACFD8207630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACFD844A377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACFD844BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACFE219D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACFE21A1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACFE21A20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACFE2501593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACFE2521D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACFE1AD02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACFE1A97EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACFDBDA5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC0B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC01D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC0BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC04855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC0B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACFEC0B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACFE3D2BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACFD7E6A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACFD7EA36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACFD7E59C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACFD7DCE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACFD8436545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150719) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150715) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACE60D2E630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACE60F71377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACE60F72A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACE6ACC48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACE6ACC8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACE6ACC90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACE6B028593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACE6B048D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACE6A5F72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACE6A5BEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACE648CC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACE7492AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACE74892846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE749304A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE748BD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE748BDEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE7492EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE7492B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACE74862B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACE6099186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACE609CA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACE60980C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACE608F53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACE60F5D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B605C551630  Unknown               Unknown  Unknown
libc-2.17.so       00002B605C794377  gsignal               Unknown  Unknown
libc-2.17.so       00002B605C795A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B60664E78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B60664EBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B60664EC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B606684B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B606686BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B6065E1A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B6065DE1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B60600EF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6070146E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B60700AE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B607014C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B60700D955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B60700D9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B607014AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6070147015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B607007EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B605C1B486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B605C1ED6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B605C1A3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B605C1183D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B605C780545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B233B01E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B233B261377  gsignal               Unknown  Unknown
libc-2.17.so       00002B233B262A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B23490928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2349096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B23490970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B23493C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B23493E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2343CE62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2343CADEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B233EBBC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B234AC1AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B234AB82846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B234AC204A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B234ABAD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B234ABADEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B234AC1EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B234AC1B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B234AB52B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B233AC8186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B233ACBA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B233AC70C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B233ABE53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B233B24D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B27053C7630  Unknown               Unknown  Unknown
libc-2.17.so       00002B270560A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B270560BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B270F35D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B270F361F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B270F3620A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B270F6C1593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B270F6E1D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B270EC902EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B270EC57EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2708F65934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2718FB6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2718F1E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2718FBC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2718F4955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2718F49EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2718FBAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2718FB7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2718EEEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B270502A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B27050636AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2705019C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2704F8E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B27055F6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150627) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB99940D630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB999650377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB999651A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB9A33A38B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB9A33A7F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB9A33A80A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB9A3707593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB9A3727D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB9A2CD62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB9A2C9DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB99CFAB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB9AD002E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB9ACF6A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9AD0084A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9ACF9555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9ACF95EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9AD006C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9AD003015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB9ACF3AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB99907086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB9990A96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB99905FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB998FD43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB99963C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150712) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150711) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B12057E7630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1205A2A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1205A2BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B120F77D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B120F781F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B120F7820A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B120FAE1593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B120FB01D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B120F0B02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B120F077EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1209385934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B12193DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1219345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B12193E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B121937055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1219370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B12193E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B12193DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B120FFE3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B120544A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B12054836AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1205439C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B12053AE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1205A16545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B63710AF630  Unknown               Unknown  Unknown
libc-2.17.so       00002B63712F2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B63712F3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B637B0458B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B637B049F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B637B04A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B637B3A9593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B637B3C9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B637A9782EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B637A93FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6374C4D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6384CF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6384C5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6384CFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6384C8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6384C87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6384CF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6384CF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B637BF94B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6370D1286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6370D4B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6370D01C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6370C763D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B63712DE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B87D9701630  Unknown               Unknown  Unknown
libc-2.17.so       00002B87D9944377  gsignal               Unknown  Unknown
libc-2.17.so       00002B87D9945A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B87E36978B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B87E369BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B87E369C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B87E39FB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B87E3A1BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B87E2FCA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B87E2F91EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B87DD29F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED3DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED3E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED37055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED3E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B87ED3DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B87E3EFDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B87D936486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B87D939D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B87D9353C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B87D92C83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B87D9930545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B410A5F3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B410A836377  gsignal               Unknown  Unknown
libc-2.17.so       00002B410A837A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B411C6EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B411C6F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B411C6F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B411CA23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B411CA43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B411C0222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4113E83EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B410E191934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B411E25EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B411E1C6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B411E2644A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B411E1F155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B411E1F1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B411E262C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B411E25F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4113F6CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B410A25686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B410A28F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B410A245C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B410A1BA3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B410A822545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B67368FA630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6736B3D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6736B3EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B67488928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6748896F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B67488970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6748BF6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6748C16D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B673FDC22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B673FD89EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B673A498934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B674A4E5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B674A44D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B674A4EB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B674A47855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B674A478EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B674A4E9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B674A4E6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B674A41DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B673655D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B67365966AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B673654CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B67364C13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6736B29545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0DDB701630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0DDB944377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0DDB945A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0DE56978B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0DE569BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0DE569C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0DE59FB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0DE5A1BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0DE4FCA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0DE4F91EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0DDF29F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE72EDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE7255846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE72F34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE728055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE7280EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE72F1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0DE72EE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0DE7225B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0DDB36486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0DDB39D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0DDB353C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0DDB2C83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0DDB930545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AD92C360630  Unknown               Unknown  Unknown
libc-2.17.so       00002AD92C5A3377  gsignal               Unknown  Unknown
libc-2.17.so       00002AD92C5A4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AD9362F68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AD9362FAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AD9362FB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AD93665A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AD93667AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AD935C292EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AD935BF0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AD92FEFE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AD9400B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AD94001D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AD9400BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AD94004855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD940048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD9400B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AD9400B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AD937E84B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AD92BFC386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AD92BFFC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AD92BFB2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AD92BF273D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AD92C58F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB0B0214630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB0B0457377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB0B0458A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB0BA1AA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB0BA1AEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB0BA1AF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB0BA50E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB0BA52ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB0B9ADD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB0B9AA4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB0B3DB2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C40B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C401D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C40BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C404855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C4048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C40B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB0C40B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB0BBD38B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB0AFE7786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB0AFEB06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB0AFE66C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB0AFDDB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB0B0443545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1F912A1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1F914E4377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1F914E5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1F9B2378B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1F9B23BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1F9B23C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1F9B59B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1F9B5BBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1F9AB6A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1F9AB31EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1F94E3F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4F15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4E7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4F1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4EA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4EA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4F19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FA4F16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1F9BF65B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1F90F0486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1F90F3D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1F90EF3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1F90E683D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1F914D0545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B059DEDA630  Unknown               Unknown  Unknown
libc-2.17.so       00002B059E11D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B059E11EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B05B004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B05B0053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B05B00540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B05B0383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B05B03A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B05A77A32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B05A776AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B05A1A78934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B05B1BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B05A7EF3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B059DB3D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B059DB766AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B059DB2CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B059DAA13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B059E109545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1FB8151630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1FB8394377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1FB8395A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1FC20E78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1FC20EBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1FC20EC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1FC244B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1FC246BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1FC1A1A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1FC19E1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1FBBCEF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3D3DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3CA5846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3D434A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3CD055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3CD0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3D41C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1FC3D3E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1FC3C75B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1FB7DB486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1FB7DED6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1FB7DA3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1FB7D183D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1FB8380545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6FB03D7630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6FB061A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6FB061BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6FBA36D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6FBA371F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6FBA3720A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6FBA6D1593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6FBA6F1D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B6FB9CA02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B6FB9C67EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6FB3F75934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC40B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC401D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC40BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC404855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC4048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC40B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6FC40B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B6FBBEFBB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6FB003A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6FB00736AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6FB0029C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6FAFF9E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6FB0606545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150623) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4C12414630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4C12657377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4C12658A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4C244918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4C24495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4C244960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4C247C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4C247E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4C1BCDD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4C1BCA4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4C15FB2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4C26000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4C25F68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4C260064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4C25F9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4C25F93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4C26004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4C26001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4C1BFEBB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4C1207786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4C120B06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4C12066C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4C11FDB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4C12643545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB0253AA630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB0255ED377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB0255EEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB02F3418B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB02F345F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB02F3460A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB02F6A5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB02F6C5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB02EC742EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB02EC3BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB028F48934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB038FA6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB038F0E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB038FAC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB038F3955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB038F39EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB038FAAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB038FA7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB038EDEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB02500D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB0250466AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB024FFCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB024F713D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB0255D9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AAEE8513630  Unknown               Unknown  Unknown
libc-2.17.so       00002AAEE8756377  gsignal               Unknown  Unknown
libc-2.17.so       00002AAEE8757A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AAEF24A98B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AAEF24ADF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AAEF24AE0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AAEF280D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AAEF282DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AAEF1DDC2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AAEF1DA3EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AAEEC0B1934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC100E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC068846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC1064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC09355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC093EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC104C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AAEFC101015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AAEFC038B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AAEE817686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AAEE81AF6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AAEE8165C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AAEE80DA3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AAEE8742545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B17CE326630  Unknown               Unknown  Unknown
libc-2.17.so       00002B17CE569377  gsignal               Unknown  Unknown
libc-2.17.so       00002B17CE56AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B17E04918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B17E0495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B17E04960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B17E07C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B17E07E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B17D7BEF2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B17D7BB6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B17D1EC4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B17E2000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B17E1F68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B17E20064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B17E1F9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B17E1F93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B17E2004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B17E2001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B17D7EFDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B17CDF8986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B17CDFC26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B17CDF78C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B17CDEED3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B17CE555545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6D5BBC1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6D5BE04377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6D5BE05A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6D65B578B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6D65B5BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6D65B5C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6D65EBB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6D65EDBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B6D6548A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B6D65451EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6D5F75F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6D677ADE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6D67715846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6D677B34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6D6774055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6D67740EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6D677B1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6D677AE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B6D676E5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6D5B82486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6D5B85D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6D5B813C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6D5B7883D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6D5BDF0545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7F86A84630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7F86CC7377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7F86CC8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7F98AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7F98AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7F98AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7F98E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7F98E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7F984232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7F8FF13EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7F8A622934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A678E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A5E0846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A67E4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A60B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A60BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A67CC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9A679015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7F9A5B0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7F866E786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7F867206AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7F866D6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7F8664B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7F86CB3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B80FDD7C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B80FDFBF377  gsignal               Unknown  Unknown
libc-2.17.so       00002B80FDFC0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B811004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8110053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B81100540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8107D11593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8107D31D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B81076452EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B810760CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B810191A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8111974E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B81118DC846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B811197A4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B811190755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8111907EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8111978C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8111975015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B81118ACB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B80FD9DF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B80FDA186AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B80FD9CEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B80FD9433D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B80FDFAB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B27D7CCE630  Unknown               Unknown  Unknown
libc-2.17.so       00002B27D7F11377  gsignal               Unknown  Unknown
libc-2.17.so       00002B27D7F12A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B27E1C648B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B27E1C68F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B27E1C690A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B27E1FC8593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B27E1FE8D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B27E15972EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B27E155EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B27DB86C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B27E38BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B27E3822846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B27E38C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B27E384D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B27E384DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B27E38BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B27E38BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B27E37F2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B27D793186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B27D796A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B27D7920C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B27D78953D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B27D7EFD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE499209630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE49944C377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE49944DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE4A319F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE4A31A3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE4A31A40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE4A3503593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE4A3523D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE4A2AD22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE4A2A99EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE49CDA7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACF15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACE7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACF1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACEA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACEA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACF19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4ACF16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE4A3ECDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE498E6C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE498EA56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE498E5BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE498DD03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE499438545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150629) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B464D9A5630  Unknown               Unknown  Unknown
libc-2.17.so       00002B464DBE8377  gsignal               Unknown  Unknown
libc-2.17.so       00002B464DBE9A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B465793B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B465793FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B46579400A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4657C9F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4657CBFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B465726E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4657235EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4651543934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B46615F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B466155E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B46615FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B466158955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4661589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B46615FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B46615F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4657F86B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B464D60886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B464D6416AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B464D5F7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B464D56C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B464DBD4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150625) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179110) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179112) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
==== backtrace (tid: 150626) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3F984E3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3F98726377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3F98727A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3FA24798B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3FA247DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3FA247E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3FA27DD593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3FA27FDD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3FA1DAC2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3FA1D73EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3F9C081934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC0E2E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC04A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC0E84A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC07555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC075EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC0E6C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3FAC0E3015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3FAC01AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3F9814686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3F9817F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3F98135C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3F980AA3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3F98712545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8FDC662630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8FDC8A5377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8FDC8A6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8FE65F88B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8FE65FCF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8FE65FD0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8FE695C593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8FE697CD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8FE5F2B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8FE5EF2EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8FE0200934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF02BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF0222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF02C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF024D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF024DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF02BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8FF02BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8FE7F81B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8FDC2C586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8FDC2FE6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8FDC2B4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8FDC2293D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8FDC891545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B137A22F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B137A472377  gsignal               Unknown  Unknown
libc-2.17.so       00002B137A473A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B138C2658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B138C269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B138C26A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B138C599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B138C5B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1383AF82EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1383ABFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B137DDCD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B138DE1FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B138DD87846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B138DE254A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B138DDB255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B138DDB2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B138DE23C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B138DE20015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B138DD57B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1379E9286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1379ECB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1379E81C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1379DF63D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B137A45E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6094149630  Unknown               Unknown  Unknown
libc-2.17.so       00002B609438C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B609438DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B609E0DF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B609E0E3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B609E0E40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B609E443593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B609E463D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B609DA122EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B609D9D9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6097CE7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B609FD35E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B609FC9D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B609FD3B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B609FCC855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B609FCC8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B609FD39C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B609FD36015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B609FC6DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6093DAC86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6093DE56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6093D9BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6093D103D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6094378545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B355C46F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B355C6B2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B355C6B3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B35664058B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3566409F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B356640A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3566769593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3566789D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3565D382EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3565CFFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B356000D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B35700B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B357001D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B35700BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B357004855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3570048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B35700B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B35700B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3567F93B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B355C0D286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B355C10B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B355C0C1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B355C0363D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B355C69E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150616) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150621) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4F2C610630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4F2C853377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4F2C854A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4F365A68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4F365AAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4F365AB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4F3690A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4F3692AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4F35ED92EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4F35EA0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4F301AE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4F402BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4F40222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F402C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F4024D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F4024DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F402BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4F402BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4F37F2FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4F2C27386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4F2C2AC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4F2C262C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4F2C1D73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4F2C83F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150619) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE3118FA630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE311B3D377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE311B3EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE31B8908B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE31B894F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE31B8950A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE31BBF4593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE31BC14D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE31B1C32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE31B18AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE315498934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE3255F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE32555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE3255FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE32558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE325589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE3255FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE3255F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE31BEDBB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE31155D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE3115966AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE31154CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE3114C13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE311B29545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B59988E2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5998B25377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5998B26A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B59A28788B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B59A287CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B59A287D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B59A2BDC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B59A2BFCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B59A21AB2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B59A2172EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B599C480934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC4DAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC442846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC4E04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC46D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC46DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC4DEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B59AC4DB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B59AC412B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B599854586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B599857E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5998534C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B59984A93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5998B11545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 150620) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150614) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 150615) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7F87B19630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7F87D5C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7F87D5DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7F91AAF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7F91AB3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7F91AB40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7F91E13593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7F91E33D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7F913E22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7F913A9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7F8B6B7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7F93705E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9366D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9370B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F9369855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F93698EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F93709C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7F93706015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7F9363DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7F8777C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7F877B56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7F8776BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7F876E03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7F87D48545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7E8BD50630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7E8BF93377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7E8BF94A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7E95CE78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7E95CEBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7E95CEC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7E9604B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7E9606BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7E9561A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7E955E1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7E8F8EE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7E9793DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7E978A5846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7E979434A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7E978D055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7E978D0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7E97941C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7E9793E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7E97875B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7E8B9B386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7E8B9EC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7E8B9A2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7E8B9173D7  PMPI_Init_f08         Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACE78CF0630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACE78F33377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACE78F34A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACE82C878B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACE82C8BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACE82C8C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACE82FEB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACE8300BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACE825BA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACE82581EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACE7C88E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C8E4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C84C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C8EA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C87755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C877EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C8E8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE8C8E5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACE8C81CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACE7895386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACE7898C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACE78942C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACE788B73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7E8BF7F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACE78F1F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B483F33E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B483F581377  gsignal               Unknown  Unknown
libc-2.17.so       00002B483F582A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B484D2F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B484D2F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B484D2F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B484D647593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B484D667D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B484CC232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4847FCDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4842EDC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B484EF2CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B484EE94846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B484EF324A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B484EEBF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B484EEBFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B484EF30C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B484EF2D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B484EE64B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B483EFA186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B483EFDA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B483EF90C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B483EF053D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B483F56D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179118) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B08612F1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0861534377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0861535A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B086B2878B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B086B28BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B086B28C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B086B5EB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B086B60BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B086ABBA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B086AB81EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0864E8F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0874F15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0874E7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0874F1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0874EA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0874EA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0874F19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0874F16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B086BFB5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0860F5486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0860F8D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0860F43C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0860EB83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0861520545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0266B3F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0266D82377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0266D83A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0278AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0278AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0278AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0278E47593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0278E67D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B02784232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B026FFCEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B026A6DD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B027A72CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B027A694846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B027A7324A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B027A6BF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B027A6BFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B027A730C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B027A72D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B027A664B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B02667A286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B02667DB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0266791C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B02667063D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0266D6E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0CDBC49630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0CDBE8C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0CDBE8DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0CE5BDF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0CE5BE3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0CE5BE40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0CE5F43593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0CE5F63D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0CE55122EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0CE54D9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0CDF7E7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE7835E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE779D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE783B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE77C855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE77C8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE7839C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CE7836015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0CE776DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0CDB8AC86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0CDB8E56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0CDB89BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0CDB8103D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0CDBE78545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B40E6FAC630  Unknown               Unknown  Unknown
libc-2.17.so       00002B40E71EF377  gsignal               Unknown  Unknown
libc-2.17.so       00002B40E71F0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B40F4F458B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B40F4F49F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B40F4F4A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B40F52A9593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B40F52C9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B40F48782EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B40F483FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B40EAB4A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B98E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B00846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B9E4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B2B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B2BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B9CC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B40F6B99015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B40F6AD0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B40E6C0F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B40E6C486AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B40E6BFEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B40E6B733D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B40E71DB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179120) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9222AB6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9222CF9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9222CFAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9234AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9234AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9234AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9234E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9234E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B92344232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B922BF45EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9226654934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B92366AAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9236612846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B92366B04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B923663D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B923663DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B92366AEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B92366AB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B92365E2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B922271986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B92227526AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9222708C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B922267D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9222CE5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB9BB643630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB9BB886377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB9BB887A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB9C55D98B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB9C55DDF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB9C55DE0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB9C593D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB9C595DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB9C4F0C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB9C4ED3EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB9BF1E1934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C722FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C7197846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C72354A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C71C255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C71C2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C7233C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB9C7230015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB9C7167B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB9BB2A686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB9BB2DF6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB9BB295C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB9BB20A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB9BB872545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B89B1C55630  Unknown               Unknown  Unknown
libc-2.17.so       00002B89B1E98377  gsignal               Unknown  Unknown
libc-2.17.so       00002B89B1E99A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B89BBBEC8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B89BBBF0F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B89BBBF10A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B89C401E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B89C403ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B89BB51F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B89BB4E6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B89B57F3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B89C5859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B89C57C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B89C585F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B89C57EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B89C57ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B89C585DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B89C585A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B89BBFD5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B89B18B886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B89B18F16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B89B18A7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B89B181C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B89B1E84545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB8A1596630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB8A17D9377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB8A17DAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB8AB52C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB8AB530F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB8AB5310A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB8AB890593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB8AB8B0D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB8AAE5F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB8AAE26EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB8A5134934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B5182E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B50EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B51884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B511555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B5115EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B5186C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB8B5183015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB8ABFEDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB8A11F986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB8A12326AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB8A11E8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB8A115D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB8A17C5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B6CBBC67630  Unknown               Unknown  Unknown
libc-2.17.so       00002B6CBBEAA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B6CBBEABA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B6CC5BFD8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B6CC5C01F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B6CC5C020A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B6CC5F61593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B6CC5F81D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B6CC55302EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B6CC54F7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6CBF805934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC7853E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC77BB846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC78594A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC77E655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC77E6EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC7857C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B6CC7854015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B6CC778BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6CBB8CA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6CBB9036AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6CBB8B9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6CBB82E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B6CBBE96545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3C50AF4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3C50D37377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3C50D38A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3C5AA8A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3C5AA8EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3C5AA8F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3C5ADEE593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3C5AE0ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3C5A3BD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3C5A384EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3C54692934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3C646F3E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3C6465B846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3C646F94A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3C6468655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3C64686EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3C646F7C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3C646F4015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3C6462BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3C5075786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3C507906AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3C50746C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3C506BB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3C50D23545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179073) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179069) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF88A785630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF88A9C8377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF88A9C9A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF89C71F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF89C723F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF89C7240A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF89CA83593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF89CAA3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF89C0522EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF893FE5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF88E323934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E372E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E2DA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E3784A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E30555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E305EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E376C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF89E373015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF89E2AAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF88A3E886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF88A4216AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF88A3D7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF88A34C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF88A9B4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1C28A57630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1C28C9A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1C28C9BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1C329ED8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1C329F1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1C329F20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1C32D51593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1C32D71D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1C323202EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1C322E7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1C2C5F5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C6C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C62E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C6CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C65955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C6CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1C3C6C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1C33F6AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1C286BA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1C286F36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1C286A9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1C2861E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1C28C86545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE223011630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE223254377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE223255A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE2310928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE231096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE2310970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE2313C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE2313E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE22BCD92EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE22BCA0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE226BAF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE232C01E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE232B69846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE232C074A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE232B9455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE232B94EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE232C05C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE232C02015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE22BFE7B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE222C7486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE222CAD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE222C63C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE222BD83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE223240545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0CC3429630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0CC366C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0CC366DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0CCD3BF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0CCD3C3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0CCD3C40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0CCD723593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0CCD743D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0CCCCF22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0CCCCB9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0CC6FC7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCF015E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCEF7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCF01B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCEFA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCEFA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCF019C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0CCF016015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0CCEF4DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0CC308C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0CC30C56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0CC307BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0CC2FF03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0CC3658545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179116) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABB8331D630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABB83560377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABB83561A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABB912F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABB912F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABB912F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABB91624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABB91644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABB90C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABB8BFACEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABB86EBB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92F09E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92E71846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92F0F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92E9C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92E9CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92F0DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABB92F0A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABB92E41B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABB82F8086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABB82FB96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABB82F6FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABB82EE43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABB8354C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179136) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7B3B1A8630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B3B3EB377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7B3B3ECA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7B492F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7B492F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7B492F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7B49624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7B49644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7B48C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7B43E37EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7B3ED46934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4AE5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4ADC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4AE654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4ADF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4ADF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4AE63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B4AE60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7B43F20B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7B3AE0B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7B3AE446AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7B3ADFAC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7B3AD6F3D7  PMPI_Init_f08         Unknown  Unknown
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B3B3D7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9152B96630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9152DD9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9152DDAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9164B368B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9164B3AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9164B3B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9164E9A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9164EBAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B91644692EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9164436EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9156734934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9166782E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B91666EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B91667884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B916671555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9166715EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9166786C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9166783015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B91666BAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B91527F986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B91528326AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B91527E8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B915275D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9152DC5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B81DC93C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B81DCB7F377  gsignal               Unknown  Unknown
libc-2.17.so       00002B81DCB80A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B81E68D28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B81E68D6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B81E68D70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B81E6C36593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B81E6C56D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B81E62052EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B81E61CCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B81E04DA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B81F0530E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B81F0498846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B81F05364A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B81F04C355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B81F04C3EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B81F0534C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B81F0531015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B81F0468B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B81DC59F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B81DC5D86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B81DC58EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B81DC5033D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B81DCB6B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB218A39630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB218C7C377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB218C7DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB2229CF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB2229D3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB2229D40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB222D33593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB222D53D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB2223022EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB2222C9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB21C5D7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C6C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C62E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C6CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C65955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C6CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB22C6C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB223F4CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB21869C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB2186D56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB21868BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB2186003D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB218C68545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7D454B7630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7D456FA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7D456FBA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7D4F44E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7D4F452F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7D4F4530A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7D4F7B2593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7D4F7D2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7D4ED812EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7D4ED48EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7D49055934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7D59182E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7D590EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7D591884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7D5911555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7D59115EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7D59186C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7D59183015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7D4FF0FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7D4511A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7D451536AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7D45109C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7D4507E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7D456E6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179144) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFC16B60630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFC16DA3377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFC16DA4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFC28AF68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFC28AFAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFC28AFB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFC28E5A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFC28E7AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFC284292EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFC1FFEFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFC1A6FE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A74CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A6B4846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A7524A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A6DF55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A6DFEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A750C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFC2A74D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFC2A684B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFC167C386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFC167FC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFC167B2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFC167273D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFC16D8F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179124) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179109) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB823B98630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB823DDB377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB823DDCA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB82DB2E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB82DB32F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB82DB330A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB82DE92593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB82DEB2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB82D4612EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB82D428EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB827736934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F784E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F6EC846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F78A4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F71755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F717EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F788C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB82F785015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB82F6BCB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB8237FB86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB8238346AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB8237EAC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB82375F3D7  PMPI_Init_f08         Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACC97E31630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACC98074377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACC98075A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACCA1DC78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACCA1DCBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACCA1DCC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACCA212B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACCA214BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACCA16FA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACCA16C1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACC9B9CF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA3A1DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA3985846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA3A234A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA39B055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA39B0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA3A21C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACCA3A1E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACCA3955B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACC97A9486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACC97ACD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACC97A83C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACC979F83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACC98060545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB823DC7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179074) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1F87805630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1F87A48377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1F87A49A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1F9179B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1F9179FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1F917A00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1F91AFF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1F91B1FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1F910CE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1F91095EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1F8B3A3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1F933F1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1F93359846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F933F74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F9338455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F93384EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F933F5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1F933F2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1F93329B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1F8746886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1F874A16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1F87457C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1F873CC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1F87A34545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B22758DD630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2275B20377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2275B21A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B227F8738B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B227F877F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B227F8780A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B227FBD7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B227FBF7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B227F1A62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B227F16DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B227947B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B22895F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B228955E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B22895FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B228958955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2289589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B22895FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B22895F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B227FEBEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B227554086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B22755796AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B227552FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B22754A43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2275B0C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B01618BD630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0161B00377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0161B01A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B016B8538B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B016B857F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B016B8580A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B016BBB7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B016BBD7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B016B1862EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B016B14DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B016545B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B01755F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B017555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B01755FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B017558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0175589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B01755FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B01755F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B016BE9EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B016152086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B01615596AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B016150FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B01614843D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0161AEC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0E73DB9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0E73FFC377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0E73FFDA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0E7DD4F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0E7DD53F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0E7DD540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0E7E0B3593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0E7E0D3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0E7D6822EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0E7D649EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0E77957934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F9A5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F90D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F9AB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F93855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F938EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F9A9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E7F9A6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0E7F8DDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0E73A1C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0E73A556AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0E73A0BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0E739803D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0E73FE8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE0B0F3B630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE0B117E377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE0B117FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE0BAED18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE0BAED5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE0BAED60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE0BB235593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE0BB255D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE0BA8042EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE0BA7CBEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE0B4AD9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4B32E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4A9A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4B384A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4AC555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4AC5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4B36C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0C4B33015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE0C4A6AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE0B0B9E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE0B0BD76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE0B0B8DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE0B0B023D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE0B116A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B992419D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B99243E0377  gsignal               Unknown  Unknown
libc-2.17.so       00002B99243E1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B992E1338B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B992E137F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B992E1380A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B992E497593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B992E4B7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B992DA662EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B992DA2DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9927D3B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B99380B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B993801D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B99380BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B993804855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9938048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B99380B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B99380B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B992FCC1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9923E0086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9923E396AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9923DEFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9923D643D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B99243CC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4BC5F77630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4BC61BA377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4BC61BBA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4BD804F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4BD8053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4BD80540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4BD8383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4BD83A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4BCF8402EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4BCF807EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4BC9B15934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BD9BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4BCFF90B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4BC5BDA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4BC5C136AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4BC5BC9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4BC5B3E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4BC61A6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B57F64E4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B57F6727377  gsignal               Unknown  Unknown
libc-2.17.so       00002B57F6728A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B58084918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5808495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B58084960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B58087E8593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5808808D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B57FFDAE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B57FFD75EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B57FA082934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B580A0D1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B580A039846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B580A0D74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B580A06455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B580A064EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B580A0D5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B580A0D2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B580A009B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B57F614786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B57F61806AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B57F6136C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B57F60AB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B57F6713545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF4C8461630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF4C86A4377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF4C86A5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF4D23F78B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF4D23FBF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF4D23FC0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF4D275B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF4D277BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF4D1D2A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF4D1CF1EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF4CBFFF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC0B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC01D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC0BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC04855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC0B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF4DC0B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF4D3F85B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF4C80C486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF4C80FD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF4C80B3C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF4C80283D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF4C8690545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
srun: error: b1177: tasks 768-895: Aborted
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFF436BD630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFF43900377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFF43901A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFF4D6538B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFF4D657F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFF4D6580A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFF4D9B7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFF4D9D7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFF4CF862EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFF4CF4DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFF4725B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F2A9E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F211846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F2AF4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F23C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F23CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F2ADC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFF4F2AA015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFF4F1E1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFF4332086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFF433596AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFF4330FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFF432843D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFF438EC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179185) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179121) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179122) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179079) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179111) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179114) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179117) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179134) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179115) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179113) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AECDF808630  Unknown               Unknown  Unknown
libc-2.17.so       00002AECDFA4B377  gsignal               Unknown  Unknown
libc-2.17.so       00002AECDFA4CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AECE979E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AECE97A2F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AECE97A30A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AECE9B02593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AECE9B22D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AECE90D12EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AECE9098EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AECE33A6934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB3F4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB35C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB3FA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB38755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB387EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB3F8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AECEB3F5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AECEB32CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AECDF46B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AECDF4A46AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AECDF45AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AECDF3CF3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AECDFA37545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE0FA09F630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE0FA2E2377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE0FA2E3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE10C04F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE10C053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE10C0540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE10C3A6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE10C3C6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE1039682EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE10392FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE0FDC3D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DC8BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DBF3846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DC914A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DC1E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DC1EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DC8FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE10DC8C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE10DBC3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE0F9D0286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE0F9D3B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE0F9CF1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE0F9C663D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE0FA2CE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179140) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE0AB044630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE0AB287377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE0AB288A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE0B90928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE0B9096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE0B90970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE0B93C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE0B93E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE0B3D0C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE0B3CD3EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE0AEBE2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BAC2EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BAB96846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BAC344A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BABC155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BABC1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BAC32C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE0BAC2F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE0BAB66B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE0AACA786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE0AACE06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE0AAC96C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE0AAC0B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE0AB273545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC9B0652630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC9B0895377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC9B0896A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC9BA5E98B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC9BA5EDF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC9BA5EE0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC9BA94D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC9BA96DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC9B9F1C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC9B9EE3EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC9B41F0934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C42BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C4222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C42C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C424D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C424DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C42BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC9C42BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC9BBF72B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC9B02B586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC9B02EE6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC9B02A4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC9B02193D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC9B0881545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179138) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179068) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3A181A2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3A183E5377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3A183E6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3A221388B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3A2213CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3A2213D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3A2249C593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3A224BCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3A21A6B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3A21A32EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3A1BD40934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C0B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C01D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C0BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C04855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C0B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3A2C0B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3A23CC6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3A17E0586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3A17E3E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3A17DF4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3A17D693D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3A183D1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179119) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179123) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179066) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179067) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179091) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0E54066630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0E542A9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0E542AAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0E5DFFC8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0E5E000F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0E5E0010A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0E5E360593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0E5E380D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0E5D92F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0E5D8F6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0E57C04934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FC52E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FBBA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FC584A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FBE555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FBE5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FC56C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E5FC53015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0E5FB8AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0E53CC986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0E53D026AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0E53CB8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0E53C2D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0E54295545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABD2CADD630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABD2CD20377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABD2CD21A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABD36A738B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABD36A77F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABD36A780A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABD36DD7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABD36DF7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABD363A62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABD3636DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABD3067B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABD406DFE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABD40647846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABD406E54A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABD4067255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABD40672EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABD406E3C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABD406E0015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABD40617B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABD2C74086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABD2C7796AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABD2C72FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABD2C6A43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABD2CD0C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179151) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB4FE869630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB4FEAAC377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB4FEAADA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB5108928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB510896F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB5108970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB510BC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB510BE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB507D312EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB507CF8EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB502407934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB512460E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB5123C8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB5124664A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB5123F355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB5123F3EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB512464C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB512461015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB512398B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB4FE4CC86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB4FE5056AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB4FE4BBC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB4FE4303D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB4FEA98545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B11D7168630  Unknown               Unknown  Unknown
libc-2.17.so       00002B11D73AB377  gsignal               Unknown  Unknown
libc-2.17.so       00002B11D73ACA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B11E52F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B11E52F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B11E52F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B11E5624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B11E5644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B11E4C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B11DFDF7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B11DAD06934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6E5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6DC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6E654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6DF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6DF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6E63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B11E6E60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B11DFEE0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B11D6DCB86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B11D6E046AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B11D6DBAC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B11D6D2F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B11D7397545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B691408A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B69142CD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B69142CEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B691E0218B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B691E025F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B691E0260A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B691E385593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B691E3A5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B691D9542EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B691D91BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6917C28934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B691FC77E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B691FBDF846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B691FC7D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B691FC0A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B691FC0AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B691FC7BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B691FC78015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B691FBAFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B6913CED86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B6913D266AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B6913CDCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6913C513D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B69142B9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B736D8AC630  Unknown               Unknown  Unknown
libc-2.17.so       00002B736DAEF377  gsignal               Unknown  Unknown
libc-2.17.so       00002B736DAF0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B73778428B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7377846F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B73778470A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7377BA6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7377BC6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B73771752EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B737713CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B737144A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7381497E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B73813FF846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B738149D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B738142A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B738142AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B738149BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7381498015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B73813CFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B736D50F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B736D5486AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B736D4FEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B736D4733D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B736DADB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2A7A435630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2A7A678377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2A7A679A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2A8C4918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2A8C495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2A8C4960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2A8C7C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2A8C7E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2A83CFE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2A83CC5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2A7DFD3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8E02DE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8DF95846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8E0334A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8DFC055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8DFC0EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8E031C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2A8E02E015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2A8DF65B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2A7A09886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2A7A0D16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2A7A087C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2A79FFC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2A7A664545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7EE96AF630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7EE98F2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7EE98F3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7EF36458B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7EF3649F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7EF364A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7EF39A9593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7EF39C9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7EF2F782EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7EF2F3FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7EED24D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD3DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD3E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD37055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD3E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7EFD3DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7EF3EABB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7EE931286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7EE934B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7EE9301C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7EE92763D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7EE98DE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACE3A80A630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACE3AA4D377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACE3AA4EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACE4C8928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACE4C896F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACE4C8970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACE4CBC6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACE4CBE6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACE43CD22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACE43C99EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACE3E3A8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E401E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E369846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E4074A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E39455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E394EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E405C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACE4E402015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACE43FE0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACE3A46D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACE3A4A66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACE3A45CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACE3A3D13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACE3AA39545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9B2464D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9B24890377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9B24891A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9B2E5E38B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9B2E5E7F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9B2E5E80A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9B2E947593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9B2E967D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9B2DF162EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9B2DEDDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9B281EB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9B382BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9B38222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B382C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B3824D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B3824DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B382BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B382BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9B2FF6CB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9B242B086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9B242E96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9B2429FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9B242143D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9B2487C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179148) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179146) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3D386FA630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3D3893D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3D3893EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3D426908B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3D42694F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3D426950A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3D429F4593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3D42A14D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3D41FC32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3D41F8AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3D3C298934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C2E7E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C24F846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C2ED4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C27A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C27AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C2EBC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D4C2E8015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3D4C21FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3D3835D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3D383966AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3D3834CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3D382C13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3D38929545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF7B06C2630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF7B0905377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF7B0906A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF7BA6588B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF7BA65CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF7BA65D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF7BA9BC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF7BA9DCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF7B9F8B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF7B9F52EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF7B4260934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C42BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C4222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C42C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C424D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C424DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C42BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF7C42BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF7BBFE1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF7B032586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF7B035E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF7B0314C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF7B02893D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF7B08F1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEFAFBD5630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEFAFE18377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEFAFE19A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEFB9B6B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEFB9B6FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEFB9B700A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEFB9ECF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEFB9EEFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEFB949E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEFB9465EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEFB3773934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB7C1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB729846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB7C74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB75455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB754EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB7C5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFBB7C2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEFBB6F9B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEFAF83886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEFAF8716AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEFAF827C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEFAF79C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEFAFE04545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEDA0C4D630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEDA0E90377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEDA0E91A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEDAABE38B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEDAABE7F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEDAABE80A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEDAAF47593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEDAAF67D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEDAA5162EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEDAA4DDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEDA47EB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB48CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB4833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB48D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB485E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB485EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB48CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEDB48CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEDABF5BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEDA08B086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEDA08E96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEDA089FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEDA08143D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEDA0E7C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADD28D2C630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADD28F6F377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADD28F70A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ADD32CC28B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ADD32CC6F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ADD32CC70A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ADD33026593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ADD33046D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ADD325F52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADD325BCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADD2C8CA934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C92AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C892846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C9304A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C8BD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C8BDEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C92EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ADD3C92B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADD3C862B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADD2898F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADD289C86AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADD2897EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADD288F33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADD28F5B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3CEC41A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3CEC65D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3CEC65EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3CF63B08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3CF63B4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3CF63B50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3CF6714593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3CF6734D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3CF5CE32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3CF5CAAEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3CEFFB8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3D000B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3D0001D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D000BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D0004855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D00048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D000B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D000B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3CF7F3EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3CEC07D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3CEC0B66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3CEC06CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3CEBFE13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3CEC649545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179137) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179072) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B77ED470630  Unknown               Unknown  Unknown
libc-2.17.so       00002B77ED6B3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B77ED6B4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B77F74068B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B77F740AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B77F740B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B77F776A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B77F778AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B77F6D392EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B77F6D00EE4  mca_pml_ucx_progr     Unknown  Unknown
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
libopen-pal.so.40  00002B77F100E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7801182E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B78010EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B78011884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B780111555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7801115EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7801186C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7801183015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B77F7EC7B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B77ED0D386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B77ED10C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B77ED0C2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B77ED0373D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B77ED69F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179071) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACB7B12F630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACB7B372377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACB7B373A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACB890DA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACB890DEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACB890DF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACB89431593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACB89451D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACB88C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACB83DBFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACB7ECCD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8AD1BE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8AC83846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8AD214A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8ACAE55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8ACAEEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8AD1FC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACB8AD1C015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACB8AC53B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACB7AD9286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACB7ADCB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACB7AD81C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACB7ACF63D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACB7B35E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B28EEFAF630  Unknown               Unknown  Unknown
libc-2.17.so       00002B28EF1F2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B28EF1F3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B28FCF4B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B28FCF4FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B28FCF500A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B28FD2AF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B28FD2CFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B28FC87E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B28FC845EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B28F2B4D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEB9EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEB06846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEBA44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEB3155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEB31EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEBA2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B28FEB9F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B28FEAD6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B28EEC1286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B28EEC4B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B28EEC01C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B28EEB763D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B28EF1DE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179076) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179075) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B91421BD630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9142400377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9142401A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B91542658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9154269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B915426A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9154599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B91545B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B914BA862EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B914BA4DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9145D5B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9155DD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9155D3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9155DDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9155D6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9155D67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9155DD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9155DD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B914BFC0B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9141E2086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9141E596AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9141E0FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9141D843D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B91423EC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5CF04D6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5CF0719377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5CF071AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5CFA46D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5CFA471F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5CFA4720A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5CFA7D1593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5CFA7F1D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5CF9DA02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5CF9D67EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5CF4074934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5D040CEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5D04036846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5D040D44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5D0406155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5D04061EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5D040D2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5D040CF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5D04006B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5CF013986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5CF01726AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5CF0128C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5CF009D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5CF0705545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179088) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE299E4E630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE29A091377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE29A092A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE2AC04F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE2AC053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE2AC0540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE2AC383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE2AC3A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE2A37182EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE2A36DFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE29D9EC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE2ADA46E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE2AD9AE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2ADA4C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2AD9D955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2AD9D9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2ADA4AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE2ADA47015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE2AD97EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE299AB186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE299AEA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE299AA0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE299A153D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE29A07D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B35A44E1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B35A4724377  gsignal               Unknown  Unknown
libc-2.17.so       00002B35A4725A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B35AE4778B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B35AE47BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B35AE47C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B35AE7DB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B35AE7FBD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B35ADDAA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B35ADD71EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B35A807F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B35B80CEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B35B8036846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B35B80D44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B35B806155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B35B8061EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B35B80D2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B35B80CF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B35B8006B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B35A414486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B35A417D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B35A4133C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B35A40A83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B35A4710545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179070) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179065) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179152) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179064) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179061) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB8837F1630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB883A34377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB883A35A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB88D7878B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB88D78BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB88D78C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB88DAEB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB88DB0BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB88D0BA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB88D081EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB88738F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F3DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F3E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F37055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F3E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB88F3DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB88F315B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB88345486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB88348D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB883443C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB8833B83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB883A20545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179133) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179062) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179063) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179135) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACC788D5630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACC78B18377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACC78B19A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACC8286B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACC8286FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACC828700A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACC82BCF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACC82BEFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACC8219E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACC82165EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACC7C473934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C4C1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C429846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C4C74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C45455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C454EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C4C5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACC8C4C2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACC83FEDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACC7853886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACC785716AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACC78527C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACC7849C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACC78B04545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179083) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179139) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
==== backtrace (tid: 179080) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEA9D3F0630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEA9D633377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEA9D634A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEAA73868B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEAA738AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEAA738B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEAA76EA593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEAA770AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEAA6CB92EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEAA6C80EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEAA0F8E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0FDCE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0F44846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0FE24A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0F6F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0F6FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0FE0C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEAB0FDD015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEAB0F14B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEA9D05386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEA9D08C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEA9D042C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEA9CFB73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEA9D61F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179103) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179142) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179082) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179129) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179095) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEFC357E630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEFC37C1377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEFC37C2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEFCD5148B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEFCD518F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEFCD5190A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEFCD878593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEFCD898D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEFCCE472EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEFCCE0EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEFC711C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF16AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF0D2846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF1704A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF0FD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF0FDEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF16EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEFCF16B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEFCF0A2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEFC31E186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEFC321A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEFC31D0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEFC31453D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEFC37AD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179094) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179086) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179147) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179143) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0260B37630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0260D7A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0260D7BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B026AACD8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B026AAD1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B026AAD20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B026AE31593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B026AE51D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B026A4002EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B026A3C7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B02646D5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0274725E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B027468D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B027472B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B02746B855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B02746B8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0274729C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0274726015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B027465DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B026079A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B02607D36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0260789C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B02606FE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0260D66545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179141) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179145) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179101) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABE50A9C630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABE50CDF377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABE50CE0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABE5AA328B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABE5AA36F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABE5AA370A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABE5AD96593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABE5ADB6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABE5A3652EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABE5A32CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABE5463A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABE646C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABE6462E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE646CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE6465955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE64659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE646CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE646C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABE5BFAFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABE506FF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABE507386AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABE506EEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABE506633D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABE50CCB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4FD54A0630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4FD56E3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4FD56E4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4FDF4368B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4FDF43AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4FDF43B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4FDF79A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4FDF7BAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4FDED692EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4FDED30EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4FD903E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE9182E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE90EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE91884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE911555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE9115EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE9186C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4FE9183015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4FDFEF7B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4FD510386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4FD513C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4FD50F2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4FD50673D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4FD56CF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179130) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9D24C94630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9D24ED7377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9D24ED8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9D2EC2A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9D2EC2EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9D2EC2F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9D2EF8E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9D2EFAED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9D2E55D2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9D2E524EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9D28832934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9D388CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9D38833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D388D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D3885E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D3885EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D388CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9D388CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9D2FFA2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9D248F786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9D249306AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9D248E6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9D2485B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9D24EC3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B768FE01630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7690044377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7690045A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7699D978B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7699D9BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7699D9C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B769A0FB593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B769A11BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B76996CA2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7699691EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B769399F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B769B9EDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B769B955846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B769B9F34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B769B98055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B769B980EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B769B9F1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B769B9EE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B769B925B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B768FA6486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B768FA9D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B768FA53C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B768F9C83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7690030545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B095348D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B09536D0377  gsignal               Unknown  Unknown
libc-2.17.so       00002B09536D1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B095D4238B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B095D427F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B095D4280A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B095D787593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B095D7A7D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B095CD562EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B095CD1DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B095702B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B095F079E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B095EFE1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B095F07F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B095F00C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B095F00CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B095F07DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B095F07A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B095EFB1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B09530F086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B09531296AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B09530DFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B09530543D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B09536BC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B314F6A2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B314F8E5377  gsignal               Unknown  Unknown
libc-2.17.so       00002B314F8E6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B31596388B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B315963CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B315963D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B315999C593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B31599BCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3158F6B2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3158F32EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3153240934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B315B28EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B315B1F6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B315B2944A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B315B22155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B315B221EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B315B292C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B315B28F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B315B1C6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B314F30586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B314F33E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B314F2F4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B314F2693D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B314F8D1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC5CDF64630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC5CE1A7377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC5CE1A8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC5E004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC5E0053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC5E00540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC5E0383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC5E03A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC5D782D2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC5D77F4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC5D1B02934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC5E1BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC5D7F7DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC5CDBC786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC5CDC006AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC5CDBB6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC5CDB2B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC5CE193545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE491BFC630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE491E3F377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE491E40A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE49BB928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE49BB96F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE49BB970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE4A401E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE4A403ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE49B4C52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE49B48CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE49579A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A5859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A57C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A585F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A57EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A57ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A585DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4A585A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE49BF7AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE49185F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE4918986AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE49184EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE4917C33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE491E2B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179131) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179132) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B775D5D3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B775D816377  gsignal               Unknown  Unknown
libc-2.17.so       00002B775D817A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B77675698B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B776756DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B776756E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B77678CD593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B77678EDD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7766E9C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7766E63EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7761171934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B77711CDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7771135846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B77711D34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B777116055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7771160EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B77711D1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B77711CE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7771105B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B775D23686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B775D26F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B775D225C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B775D19A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B775D802545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABC55BB1630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABC55DF4377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABC55DF5A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABC5FB478B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABC5FB4BF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABC5FB4C0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABC6801E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABC6803ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABC5F47A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABC5F441EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABC5974F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABC69859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABC697C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC6985F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC697EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC697ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC6985DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABC6985A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABC5FF2FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABC5581486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABC5584D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABC55803C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABC557783D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABC55DE0545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179096) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179153) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2AF1D2F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2AF1F72377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2AF1F73A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2AFBCC58B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2AFBCC9F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2AFBCCA0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2B0402B593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2B0404BD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2AFB5F82EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2AFB5BFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2AF58CD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2B0591AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2B05882846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B059204A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B058AD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B058ADEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B0591EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2B0591B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2B05852B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2AF199286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2AF19CB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2AF1981C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2AF18F63D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2AF1F5E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B37BC007630  Unknown               Unknown  Unknown
libc-2.17.so       00002B37BC24A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B37BC24BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B37C5F9D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B37C5FA1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B37C5FA20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B37C6301593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B37C6321D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B37C58D02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B37C5897EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B37BFBA5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7BF3E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7B5B846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7BF94A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7B8655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7B86EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7BF7C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B37C7BF4015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B37C7B2BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B37BBC6A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B37BBCA36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B37BBC59C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B37BBBCE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B37BC236545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179149) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179150) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA336AE4630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA336D27377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA336D28A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA348AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA348AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA348AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA348E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA348E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA3484232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA33FF74EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA33A682934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A6D1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A639846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A6D74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A66455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A664EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A6D5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA34A6D2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA34A609B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA33674786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA3367806AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA336736C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA3366AB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA336D13545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B100E957630  Unknown               Unknown  Unknown
libc-2.17.so       00002B100EB9A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B100EB9BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1020AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1020AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1020AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1020E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1020E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B10204232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1017DE7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B10124F5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B102265FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B10225C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B10226654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B10225F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B10225F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1022663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1022660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1017ED1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B100E5BA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B100E5F36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B100E5A9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B100E51E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B100EB86545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179125) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3198A0A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3198C4D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3198C4EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B31A29A08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B31A29A4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B31A29A50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B31A2D04593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B31A2D24D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B31A22D32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B31A229AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B319C5A8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC6C6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC62E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC6CC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC65955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC659EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC6CAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B31AC6C7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B31A3F1DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B319866D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B31986A66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B319865CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B31985D13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3198C39545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179127) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179128) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179126) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9740D07630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9740F4A377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9740F4BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B974AC9D8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B974ACA1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B974ACA20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B974B001593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B974B021D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B974A5D02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B974A597EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B97448A5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B97548F8E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9754860846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B97548FE4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B975488B55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B975488BEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B97548FCC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B97548F9015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9754830B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B974096A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B97409A36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9740959C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B97408CE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9740F36545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE441AFC630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE441D3F377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE441D40A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE44BA928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE44BA96F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE44BA970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE45401E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE45403ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE44B3C52EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE44B38CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE44569A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE4556EAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE455652846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4556F04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE45567D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE45567DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4556EEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE4556EB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE455622B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE44175F86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE4417986AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE44174EC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE4416C33D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE441D2B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179090) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179078) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179077) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179154) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179156) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AE4790CF630  Unknown               Unknown  Unknown
libc-2.17.so       00002AE479312377  gsignal               Unknown  Unknown
libc-2.17.so       00002AE479313A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE4830668B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE48306AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE48306B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE4833CA593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE4833EAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE4829992EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AE482960EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AE47CC6D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CCF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CC5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CCFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CC8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CC87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CCF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE48CCF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AE483FB5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AE478D3286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AE478D6B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AE478D21C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AE478C963D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AE4792FE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179081) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179155) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179106) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179107) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
srun: error: b1155: tasks 0-127: Aborted
==== backtrace (tid: 179089) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179084) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2EBAD83630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2EBAFC6377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2EBAFC7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2EC8EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2EC8EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2EC8EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2EC9224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2EC9244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2EC88232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2EC3E13EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2EBE921934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECAA5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECA9C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECAA654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECA9F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECA9F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECAA63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2ECAA60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2EC3EFDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2EBA9E686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2EBAA1F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2EBA9D5C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2EBA94A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2EBAFB2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179093) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179174) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179085) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B57F8BA0630  Unknown               Unknown  Unknown
libc-2.17.so       00002B57F8DE3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B57F8DE4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5802B368B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5802B3AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5802B3B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5802E9A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5802EBAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B58024692EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5802430EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B57FC73E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B580C8CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B580C833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B580C8D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B580C85E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B580C85EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B580C8CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B580C8CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5803EAEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B57F880386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B57F883C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B57F87F2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B57F87673D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B57F8DCF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179087) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179092) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179104) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B995C0D4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B995C317377  gsignal               Unknown  Unknown
libc-2.17.so       00002B995C318A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B996606B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B996606FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B99660700A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B99663CF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B99663EFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B996599E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9965965EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B995FC72934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9967CC1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9967C29846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9967CC74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9967C5455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9967C54EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9967CC5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9967CC2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9967BF9B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B995BD3786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B995BD706AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B995BD26C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B995BC9B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B995C303545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC8E6F8D630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC8E71D0377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC8E71D1A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC8F4F268B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC8F4F2AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC8F4F2B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC8F528A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC8F52AAD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC8F48592EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC8EFFEDEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC8EAB2B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6B79E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6AE1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6B7F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6B0C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6B0CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6B7DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC8F6B7A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC8F6AB1B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC8E6BF086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC8E6C296AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC8E6BDFC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC8E6B543D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC8E71BC545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9AFD934630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9AFDB77377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9AFDB78A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9B078CA8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9B078CEF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9B078CF0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9B07C2E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9B07C4ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9B071FD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9B071C4EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9B014D2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9B115F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9B1155E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B115FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B1158955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B11589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B115FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9B115F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9B07F15B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9AFD59786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9AFD5D06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9AFD586C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9AFD4FB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9AFDB63545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179102) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179165) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
==== backtrace (tid: 179166) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179167) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179097) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179098) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
==== backtrace (tid: 179100) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179099) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179105) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179108) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7B0B5F3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B0B836377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7B0B837A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7B155898B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7B1558DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7B1558E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7B158ED593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7B1590DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7B14EBC2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7B14E83EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7B0F191934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7B171DFE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7B17147846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B171E54A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B1717255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B17172EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B171E3C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B171E0015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7B17117B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7B0B25686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7B0B28F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7B0B245C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7B0B1BA3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B0B822545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9F43404630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9F43647377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9F43648A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9F4D39A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9F4D39EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9F4D39F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9F4D6FE593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9F4D71ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9F4CCCD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9F4CC94EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9F46FA2934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EFF0E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EF58846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EFF64A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EF8355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EF83EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EFF4C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9F4EFF1015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9F4EF28B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9F4306786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9F430A06AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9F43056C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9F42FCB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9F43633545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179186) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
==== backtrace (tid: 179188) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179187) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179168) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179172) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179170) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
==== backtrace (tid: 179171) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179169) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179184) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
==== backtrace (tid: 179183) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179176) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179175) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179173) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179182) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC73C421630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC73C664377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC73C665A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC7463B88B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC7463BCF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC7463BD0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC74671C593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC74673CD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC745CEB2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC745CB2EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC73FFBF934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC7500B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC75001D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC7500BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC75004855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC750048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC7500B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC7500B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC747F46B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC73C08486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC73C0BD6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC73C073C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC73BFE83D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC73C650545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
==== backtrace (tid: 179181) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179160) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179161) ====
==== backtrace (tid: 179162) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179158) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179164) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179159) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179163) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179157) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179179) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
==== backtrace (tid: 179180) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179177) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
==== backtrace (tid: 179178) ====
 0 0x0000000000050ba5 ucs_debug_print_backtrace()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/debug/debug.c:625
 1 0x000000000001e593 uct_ib_mlx5_completion_with_err()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5_log.c:132
 2 0x000000000003ed5a uct_ib_mlx5_poll_cq()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/mlx5/ib_mlx5.inl:81
 3 0x000000000003ed5a uct_dc_mlx5_iface_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/ib/dc/dc_mlx5.c:238
 4 0x00000000000222ea ucs_callbackq_dispatch()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucs/datastruct/callbackq.h:211
 5 0x00000000000222ea uct_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/uct/api/uct.h:2221
 6 0x00000000000222ea ucp_worker_progress()  /build-result/src/hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1-redhat7.7-x86_64/ucx-v1.8.x/src/ucp/core/ucp_worker.c:1951
 7 0x0000000000005ee4 mca_pml_ucx_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/pml/ucx/pml_ucx.c:515
 8 0x000000000002f934 opal_progress()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/opal/runtime/opal_progress.c:231
 9 0x00000000000b5e29 wait_completion()  hcoll_collectives.c:0
10 0x000000000001d846 comm_allgather_hcolrte()  ???:0
11 0x00000000000bb4a7 hmca_coll_ml_hierarchy_discovery()  ???:0
12 0x000000000004855e hmca_coll_ml_comm_query_proceed()  ???:0
13 0x0000000000048eab hmca_coll_ml_comm_query()  ???:0
14 0x00000000000b9c01 hcoll_get_context_from_cache()  ???:0
15 0x00000000000b6015 hcoll_create_context()  ???:0
16 0x0000000000006b9c mca_coll_hcoll_comm_query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/hcoll/coll_hcoll_module.c:373
17 0x000000000008786d query_2_0_0()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:449
18 0x000000000008786d query()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:432
19 0x000000000008786d check_one_component()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:394
20 0x000000000008786d check_components()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:344
21 0x000000000008786d mca_coll_base_comm_select()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mca/coll/base/coll_base_comm_select.c:126
22 0x00000000000c06ad ompi_mpi_init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/runtime/ompi_mpi_init.c:958
23 0x0000000000076c3d PMPI_Init()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/c/profile/pinit.c:67
24 0x00000000000573d7 ompi_init_f()  /cluster/work/users/vegarde/build/OpenMPI/4.0.3/iccifort-2020.1.217/openmpi-4.0.3/ompi/mpi/fortran/mpif-h/profile/pinit_f.c:84
25 0x00000000004352a4 cime_comp_mod_mp_cime_pre_init1_()  /cluster/home/shofer/noresm/cime/src/drivers/mct/main/cime_comp_mod.F90:603
26 0x00000000004370c5 MAIN__()  ???:0
27 0x0000000000419512 main()  ???:0
28 0x0000000000022545 __libc_start_main()  ???:0
29 0x0000000000419429 _start()  ???:0
=================================
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5BD9F9A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5BDA1DD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5BDA1DEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5BEC04F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5BEC053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5BEC0540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5BEC383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5BEC3A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5BE38632EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5BE382AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5BDDB38934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDBBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDB26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDBC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDB5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDB51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDBC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5BEDBBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5BE3FB3B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5BD9BFD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5BD9C366AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5BD9BECC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5BD9B613D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5BDA1C9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B59C4F55630  Unknown               Unknown  Unknown
libc-2.17.so       00002B59C5198377  gsignal               Unknown  Unknown
libc-2.17.so       00002B59C5199A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B59CEEEB8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B59CEEEFF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B59CEEF00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B59CF24F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B59CF26FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B59CE81E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B59CE7E5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B59C8AF3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8B46E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8AAE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8B4C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8AD955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8AD9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8B4AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B59D8B47015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B59D8A7EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B59C4BB886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B59C4BF16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B59C4BA7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B59C4B1C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B59C5184545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8178D56630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8178F99377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8178F9AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8182CEC8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8182CF0F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8182CF10A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8183050593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8183070D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B818261F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B81825E6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B817C8F4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B818C943E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B818C8AB846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B818C9494A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B818C8D655E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B818C8D6EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B818C947C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B818C944015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B818C87BB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B81789B986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B81789F26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B81789A8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B817891D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8178F85545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B76D25B9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B76D27FC377  gsignal               Unknown  Unknown
libc-2.17.so       00002B76D27FDA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B76E46EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B76E46F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B76E46F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B76E4A23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B76E4A43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B76E40222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B76DBE49EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B76D6157934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B76E625EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B76E61C6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B76E62644A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B76E61F155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B76E61F1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B76E6262C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B76E625F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B76DBF32B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B76D221C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B76D22556AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B76D220BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B76D21803D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B76D27E8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5FC1E7B630  Unknown               Unknown  Unknown
libc-2.17.so       00002B5FC20BE377  gsignal               Unknown  Unknown
libc-2.17.so       00002B5FC20BFA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5FD404F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5FD4053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5FD40540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5FD4383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5FD43A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B5FCB7442EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5FCB70BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B5FC5A19934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD5A65E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD59CD846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD5A6B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD59F855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD59F8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD5A69C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5FD5A66015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5FD599DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5FC1ADE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5FC1B176AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5FC1ACDC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5FC1A423D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B5FC20AA545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B96952E6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9695529377  gsignal               Unknown  Unknown
libc-2.17.so       00002B969552AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B969F27C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B969F280F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B969F2810A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B969F5E0593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B969F600D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B969EBAF2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B969EB76EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9698E84934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8F15E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8E7D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8F1B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8EA855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8EA8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8F19C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B96A8F16015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B969FFAAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9694F4986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9694F826AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9694F38C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9694EAD3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9695515545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA398B27630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA398D6A377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA398D6BA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA3A2ABD8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA3A2AC1F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA3A2AC20A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA3A2E21593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA3A2E41D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA3A23F02EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA3A23B7EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA39C6C5934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC725E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC68D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC72B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC6B855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC6B8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC729C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA3AC726015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA3AC65DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA39878A86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA3987C36AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA398779C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA3986EE3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA398D56545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B98AFB7A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B98AFDBD377  gsignal               Unknown  Unknown
libc-2.17.so       00002B98AFDBEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B98B9B108B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B98B9B14F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B98B9B150A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B98B9E74593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B98B9E94D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B98B94432EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B98B940AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B98B3718934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB766E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB6CE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB76C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB6F955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB6F9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB76AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B98BB767015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B98BB69EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B98AF7DD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B98AF8166AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B98AF7CCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B98AF7413D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B98AFDA9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADFD3F55630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADFD4198377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADFD4199A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ADFDDEEB8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ADFDDEEFF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ADFDDEF00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ADFDE24F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ADFDE26FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ADFDD81E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADFDD7E5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADFD7AF3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFB41E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFAA9846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFB474A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFAD455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFAD4EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFB45C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ADFDFB42015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADFDFA79B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADFD3BB886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADFD3BF16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADFD3BA7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADFD3B1C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADFD4184545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC57FCFD630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC57FF40377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC57FF41A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC589C938B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC589C97F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC589C980A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC589FF7593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC58A017D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC5895C62EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC58958DEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC58389B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B8E9E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B851846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B8EF4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B87C55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B87CEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B8EDC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC58B8EA015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC58B821B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC57F96086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC57F9996AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC57F94FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC57F8C43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC57FF2C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B94D4D5B630  Unknown               Unknown  Unknown
libc-2.17.so       00002B94D4F9E377  gsignal               Unknown  Unknown
libc-2.17.so       00002B94D4F9FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B94DECF18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B94DECF5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B94DECF60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B94DF055593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B94DF075D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B94DE6242EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B94DE5EBEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B94D88F9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B94E8948E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B94E88B0846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B94E894E4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B94E88DB55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B94E88DBEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B94E894CC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B94E8949015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B94E8880B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B94D49BE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B94D49F76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B94D49ADC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B94D49223D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B94D4F8A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B425150E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4251751377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4251752A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B425B4A48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B425B4A8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B425B4A90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B425B808593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B425B828D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B425ADD72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B425AD9EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B42550AC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4265182E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B42650EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B42651884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B426511555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4265115EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4265186C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4265183015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B425BF65B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B425117186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B42511AA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4251160C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B42510D53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B425173D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B938995F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9389BA2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9389BA3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B93938F58B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B93938F9F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B93938FA0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9393C59593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9393C79D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B93932282EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B93931EFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B938D4FD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B939D5F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B939D55E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B939D5FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B939D58955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B939D589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B939D5FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B939D5F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9393F40B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B93895C286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B93895FB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B93895B1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B93895263D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9389B8E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFE5797A630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE57BBD377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFE57BBEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFE619108B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFE61914F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFE619150A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFE61C74593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFE61C94D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFE612432EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFE6120AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFE5B518934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFE63566E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFE634CE846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE6356C4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE634F955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE634F9EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE6356AC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE63567015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFE6349EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFE575DD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFE576166AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFE575CCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFE575413D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE57BA9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0468702630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0468945377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0468946A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B04726988B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B047269CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B047269D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B04729FC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0472A1CD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0471FCB2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0471F92EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B046C2A0934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B047C305E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B047C26D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B047C30B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B047C29855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B047C298EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B047C309C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B047C306015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B047C23DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B046836586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B046839E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0468354C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B04682C93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0468931545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B37474F2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3747735377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3747736A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B37514888B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B375148CF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B375148D0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B37517EC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B375180CD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3750DBB2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3750D82EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B374B090934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B37530DEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3753046846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B37530E44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B375307155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3753071EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B37530E2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B37530DF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3753016B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B374715586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B374718E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3747144C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B37470B93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3747721545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC3E93AA630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC3E95ED377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC3E95EEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC3F33408B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC3F3344F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC3F33450A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC3F36A4593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC3F36C4D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC3F2C732EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC3F2C3AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC3ECF48934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCFA6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCF0E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCFAC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCF3955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCF39EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCFAAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC3FCFA7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC3FCEDEB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC3E900D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC3E90466AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC3E8FFCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC3E8F713D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC3E95D9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B772E42A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B772E66D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B772E66EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B77404918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7740495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B77404960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B77407C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B77407E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7737CF32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7737CBAEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7731FC8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7742019E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7741F81846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B774201F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7741FAC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7741FACEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B774201DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B774201A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7741F51B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B772E08D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B772E0C66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B772E07CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B772DFF13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B772E659545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B83A891B630  Unknown               Unknown  Unknown
libc-2.17.so       00002B83A8B5E377  gsignal               Unknown  Unknown
libc-2.17.so       00002B83A8B5FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B83B28B18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B83B28B5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B83B28B60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B83B2C15593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B83B2C35D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B83B21E42EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B83B21ABEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B83AC4B9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC50CE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC474846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC5124A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC49F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC49FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC510C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B83BC50D015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B83BC444B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B83A857E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B83A85B76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B83A856DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B83A84E23D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B83A8B4A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2149073630  Unknown               Unknown  Unknown
libc-2.17.so       00002B21492B6377  gsignal               Unknown  Unknown
libc-2.17.so       00002B21492B7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B21530098B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B215300DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B215300E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B215336D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B215338DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B215293C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2152903EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B214CC11934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B215CCF4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B215CC5C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B215CCFA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B215CC8755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B215CC87EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B215CCF8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B215CCF5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2153F58B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2148CD686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2148D0F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2148CC5C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2148C3A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B21492A2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AFE6196C630  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE61BAF377  gsignal               Unknown  Unknown
libc-2.17.so       00002AFE61BB0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AFE6B9028B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AFE6B906F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AFE6B9070A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AFE6BC66593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AFE6BC86D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AFE6B2352EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AFE6B1FCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AFE6550A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AFE755F6E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AFE7555E846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE755FC4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE7558955E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE75589EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE755FAC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AFE755F7015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AFE6BF4DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AFE615CF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AFE616086AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AFE615BEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AFE615333D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AFE61B9B545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B74232F2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7423535377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7423536A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B74312F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B74312F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B74312F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7431624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7431644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7430C232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B742BF81EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7426E90934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7432EDCE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7432E44846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7432EE24A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7432E6F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7432E6FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7432EE0C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7432EDD015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7432E14B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7422F5586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7422F8E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7422F44C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7422EB93D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7423521545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B92DDDE4630  Unknown               Unknown  Unknown
libc-2.17.so       00002B92DE027377  gsignal               Unknown  Unknown
libc-2.17.so       00002B92DE028A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B92F004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B92F0053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B92F00540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B92E7D79593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B92E7D99D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B92E76AD2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B92E7674EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B92E1982934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B92F19CEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B92F1936846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B92F19D44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B92F196155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B92F1961EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B92F19D2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B92F19CF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B92F1906B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B92DDA4786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B92DDA806AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B92DDA36C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B92DD9AB3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B92DE013545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B52CDF2A630  Unknown               Unknown  Unknown
libc-2.17.so       00002B52CE16D377  gsignal               Unknown  Unknown
libc-2.17.so       00002B52CE16EA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B52E004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B52E0053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B52E00540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B52E0383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B52E03A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B52D77F32EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B52D77BAEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B52D1AC8934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B52E1BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B52D7F43B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B52CDB8D86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B52CDBC66AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B52CDB7CC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B52CDAF13D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B52CE159545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ACD12DB2630  Unknown               Unknown  Unknown
libc-2.17.so       00002ACD12FF5377  gsignal               Unknown  Unknown
libc-2.17.so       00002ACD12FF6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ACD20EF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ACD20EF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ACD20EF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ACD21224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ACD21244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ACD208232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ACD1BE41EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ACD16950934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ACD22A5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ACD229C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ACD22A654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ACD229F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACD229F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ACD22A63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ACD22A60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ACD1BF2AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ACD12A1586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ACD12A4E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ACD12A04C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ACD129793D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ACD12FE1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4FFAF75630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4FFB1B8377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4FFB1B9A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5008F0C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5008F10F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5008F110A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B5009270593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B5009290D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B500883F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5008806EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4FFEB13934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B500AB5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B500AAC7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B500AB654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B500AAF255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B500AAF2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B500AB63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B500AB60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B500AA97B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4FFABD886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4FFAC116AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4FFABC7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4FFAB3C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4FFB1A4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AF2A0CD8630  Unknown               Unknown  Unknown
libc-2.17.so       00002AF2A0F1B377  gsignal               Unknown  Unknown
libc-2.17.so       00002AF2A0F1CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AF2AAC6E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AF2AAC72F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AF2AAC730A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AF2AAFD2593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AF2AAFF2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AF2AA5A12EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AF2AA568EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AF2A4876934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B48CBE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B4833846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B48D14A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B485E55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B485EEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B48CFC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AF2B48CC015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AF2ABFE6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AF2A093B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AF2A09746AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AF2A092AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AF2A089F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AF2A0F07545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4B9FC95630  Unknown               Unknown  Unknown
libc-2.17.so       00002B4B9FED8377  gsignal               Unknown  Unknown
libc-2.17.so       00002B4B9FED9A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B4BA9C2B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4BA9C2FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B4BA9C300A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4BA9F8F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B4BA9FAFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B4BA955E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4BA9525EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4BA3833934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB881E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB7E9846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB8874A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB81455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB814EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB885C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4BAB882015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4BAB7B9B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4B9F8F886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4B9F9316AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4B9F8E7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4B9F85C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B4B9FEC4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ABE40B28630  Unknown               Unknown  Unknown
libc-2.17.so       00002ABE40D6B377  gsignal               Unknown  Unknown
libc-2.17.so       00002ABE40D6CA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002ABE4AABE8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002ABE4AAC2F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002ABE4AAC30A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002ABE4AE22593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002ABE4AE42D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002ABE4A3F12EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ABE4A3B8EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ABE446C6934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002ABE54725E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002ABE5468D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE5472B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE546B855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE546B8EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE54729C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002ABE54726015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ABE5465DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ABE4078B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ABE407C46AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ABE4077AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ABE406EF3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ABE40D57545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4652074630  Unknown               Unknown  Unknown
libc-2.17.so       00002B46522B7377  gsignal               Unknown  Unknown
libc-2.17.so       00002B46522B8A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B466404F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B4664053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B46640540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B4664383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B46643A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B465B93E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B465B905EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B4655C12934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B4665C62E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B4665BCA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B4665C684A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B4665BF555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4665BF5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B4665C66C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B4665C63015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4665B9AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B4651CD786D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B4651D106AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B4651CC6C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B4651C3B3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B46522A3545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA72667A630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA7268BD377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA7268BEA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA7386EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA7386F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA7386F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA738A23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA738A43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA7380222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA72FF0AEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA72A218934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A277E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A1DF846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A27D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A20A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A20AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A27BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA73A278015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA73A1AFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA7262DD86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA7263166AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA7262CCC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA7262413D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA7268A9545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA3F025B630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA3F049E377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA3F049FA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA3FA1F18B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA3FA1F5F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA3FA1F60A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA3FA555593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA3FA575D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA3F9B242EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA3F9AEBEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA3F3DF9934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA4040B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA40401D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA4040BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA40404855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA404048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA4040B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA4040B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA3FBD7FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA3EFEBE86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA3EFEF76AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA3EFEADC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA3EFE223D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA3F048A545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1AACF3F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1AAD182377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1AAD183A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1AB6ED68B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1AB6EDAF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1AB6EDB0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1AB723A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1AB725AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1AB68092EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1AB67D0EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1AB0ADD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0B32E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0A9A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0B384A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0AC555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0AC5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0B36C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1AC0B33015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1AC0A6AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1AACBA286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1AACBDB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1AACB91C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1AACB063D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1AAD16E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8DE77F0630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8DE7A33377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8DE7A34A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8DF17868B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8DF178AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8DF178B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8DF1AEA593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8DF1B0AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8DF10B92EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8DF1080EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8DEB38E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF33DCE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF3344846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF33E24A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF336F55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF336FEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF33E0C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8DF33DD015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8DF3314B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8DE745386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8DE748C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8DE7442C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8DE73B73D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8DE7A1F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3D5307C630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3D532BF377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3D532C0A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3D610928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3D61096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3D610970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3D613C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3D613E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3D5BD452EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3D5BD0CEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3D56C1A934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62C70E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62BD8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62C764A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62C0355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62C03EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62C74C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D62C71015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3D62BA8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3D52CDF86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3D52D186AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3D52CCEC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3D52C433D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3D532AB545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B090C6F3630  Unknown               Unknown  Unknown
libc-2.17.so       00002B090C936377  gsignal               Unknown  Unknown
libc-2.17.so       00002B090C937A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B09166898B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B091668DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B091668E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B09169ED593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0916A0DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0915FBC2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0915F83EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0910291934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B09202E7E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B092024F846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B09202ED4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B092027A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B092027AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B09202EBC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B09202E8015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B092021FB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B090C35686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B090C38F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B090C345C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B090C2BA3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B090C922545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1699BCF630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1699E12377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1699E13A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B16A3B658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B16A3B69F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B16A3B6A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B16AC01E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B16AC03ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B16A34982EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B16A345FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B169D76D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD859E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD7C1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD85F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD7EC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD7ECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD85DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B16AD85A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B16A3F4DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B169983286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B169986B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1699821C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B16997963D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1699DFE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B64D35F5630  Unknown               Unknown  Unknown
libc-2.17.so       00002B64D3838377  gsignal               Unknown  Unknown
libc-2.17.so       00002B64D3839A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B64DD58B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B64DD58FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B64DD5900A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B64DD8EF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B64DD90FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B64DCEBE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B64DCE85EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B64D7193934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF1E1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF149846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF1E74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF17455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF174EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF1E5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B64DF1E2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B64DF119B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B64D325886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B64D32916AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B64D3247C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B64D31BC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B64D3824545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3D122B8630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3D124FB377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3D124FCA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3D242658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3D24269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3D2426A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3D245BC593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3D245DCD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3D1BB822EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3D1BB49EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3D15E56934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25EA5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25E0D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25EAB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25E3855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25E38EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25EA9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D25EA6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3D25DDDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3D11F1B86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3D11F546AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3D11F0AC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3D11E7F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3D124E7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002BA1BA187630  Unknown               Unknown  Unknown
libc-2.17.so       00002BA1BA3CA377  gsignal               Unknown  Unknown
libc-2.17.so       00002BA1BA3CBA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002BA1CC2658B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002BA1CC269F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002BA1CC26A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002BA1CC599593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002BA1CC5B9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002BA1C3A502EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002BA1C3A17EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002BA1BDD25934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDDD4E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDD3C846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDDDA4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDD6755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDD67EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDDD8C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002BA1CDDD5015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002BA1C3F8AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002BA1B9DEA86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002BA1B9E236AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002BA1B9DD9C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002BA1B9D4E3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002BA1BA3B6545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0C22996630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0C22BD9377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0C22BDAA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0C34AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0C34AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0C34AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0C34E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0C34E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0C344232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0C2BE25EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0C26534934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0C3665FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0C365C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0C366654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0C365F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0C365F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0C36663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0C36660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0C2BF0EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0C225F986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0C226326AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0C225E8C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0C2255D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0C22BC5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B03A82C6630  Unknown               Unknown  Unknown
libc-2.17.so       00002B03A8509377  gsignal               Unknown  Unknown
libc-2.17.so       00002B03A850AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B03B225C8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B03B2260F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B03B22610A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B03B25C0593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B03B25E0D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B03B1B8F2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B03B1B56EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B03ABE64934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC0B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC01D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC0BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC04855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC0B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B03BC0B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B03B3DEAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B03A7F2986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B03A7F626AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B03A7F18C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B03A7E8D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B03A84F5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B71230B0630  Unknown               Unknown  Unknown
libc-2.17.so       00002B71232F3377  gsignal               Unknown  Unknown
libc-2.17.so       00002B71232F4A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B71310928B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7131096F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B71310970A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B71313C6593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B71313E6D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B712BD792EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B712BD40EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7126C4E934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7132CA2E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7132C0A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7132CA84A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7132C3555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7132C35EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7132CA6C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7132CA3015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7132BDAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7122D1386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7122D4C6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7122D02C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7122C773D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B71232DF545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3BCE549630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3BCE78C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3BCE78DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3BE06EF8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3BE06F3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3BE06F40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3BE0A23593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3BE0A43D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3BE00222EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3BD7DD9EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3BD20E7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE225EE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE21C6846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE22644A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE21F155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE21F1EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE2262C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3BE225F015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3BD7EC2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3BCE1AC86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3BCE1E56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3BCE19BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3BCE1103D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3BCE778545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB394483630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB3946C6377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB3946C7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB39E4198B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB39E41DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB39E41E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB39E77D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB39E79DD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB39DD4C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB39DD13EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB398021934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A80B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A801D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A80BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A804855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A8048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A80B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB3A80B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB39FFA7B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB3940E686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB39411F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB3940D5C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB39404A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB3946B2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B5716F6F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B57171B2377  gsignal               Unknown  Unknown
libc-2.17.so       00002B57171B3A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B5724F068B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B5724F0AF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B5724F0B0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B572526A593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B572528AD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B57248392EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B5724806EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B571AB0D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B5726B59E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B5726AC1846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B5726B5F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B5726AEC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5726AECEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B5726B5DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B5726B5A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B5726A91B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B5716BD286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B5716C0B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B5716BC1C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B5716B363D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B571719E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AC49F5BB630  Unknown               Unknown  Unknown
libc-2.17.so       00002AC49F7FE377  gsignal               Unknown  Unknown
libc-2.17.so       00002AC49F7FFA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AC4A95518B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AC4A9555F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AC4A95560A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AC4A98B5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AC4A98D5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AC4A8E842EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AC4A8E4BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AC4A3159934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB1A7E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB10F846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB1AD4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB13A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB13AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB1ABC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AC4AB1A8015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AC4AB0DFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AC49F21E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AC49F2576AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AC49F20DC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AC49F1823D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AC49F7EA545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B7B1D709630  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B1D94C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B7B1D94DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B7B2769F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B7B276A3F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B7B276A40A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B7B27A03593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B7B27A23D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B7B26FD22EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B7B26F99EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B7B212A7934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B7B313DDE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B7B31345846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B313E34A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B3137055E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B31370EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B313E1C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B7B313DE015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B7B27F05B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B7B1D36C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B7B1D3A56AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B7B1D35BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B7B1D2D03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B7B1D938545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B2DEF2D9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B2DEF51C377  gsignal               Unknown  Unknown
libc-2.17.so       00002B2DEF51DA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B2DFD2F08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B2DFD2F4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B2DFD2F50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B2DFD624593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B2DFD644D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B2DFCC232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B2DF7F68EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B2DF2E77934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEECEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEE36846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEED44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEE6155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEE61EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEED2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B2DFEECF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B2DFEE06B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B2DEEF3C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B2DEEF756AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B2DEEF2BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B2DEEEA03D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B2DEF508545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB678785630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB6789C8377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB6789C9A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB68271B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB68271FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB6827200A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB682A7F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB682A9FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB68204E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB682015EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB67C323934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C4C1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C429846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C4C74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C45455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C454EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C4C5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB68C4C2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB683E9DB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB6783E886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB6784216AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB6783D7C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB67834C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB6789B4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8C93A26630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8C93C69377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8C93C6AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8C9D9BC8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8C9D9C0F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8C9D9C10A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8C9DD20593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8C9DD40D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8C9D2EF2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8C9D2B6EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8C975C4934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F612E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F57A846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F6184A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F5A555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F5A5EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F616C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8C9F613015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8C9F54AB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8C9368986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8C936C26AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8C93678C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8C935ED3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8C93C55545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B30B7DC5630  Unknown               Unknown  Unknown
libc-2.17.so       00002B30B8008377  gsignal               Unknown  Unknown
libc-2.17.so       00002B30B8009A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B30C1D5B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B30C1D5FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B30C1D600A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B30C20BF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B30C20DFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B30C168E2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B30C1655EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B30BB963934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B30C39B1E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B30C3919846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B30C39B74A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B30C394455E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B30C3944EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B30C39B5C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B30C39B2015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B30C38E9B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B30B7A2886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B30B7A616AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B30B7A17C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B30B798C3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B30B7FF4545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B38FC725630  Unknown               Unknown  Unknown
libc-2.17.so       00002B38FC968377  gsignal               Unknown  Unknown
libc-2.17.so       00002B38FC969A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B39066BB8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B39066BFF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B39066C00A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3906A1F593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3906A3FD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3905FEE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3905FB5EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B39002C3934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3910319E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3910281846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B391031F4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B39102AC55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B39102ACEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B391031DC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B391031A015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3910251B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B38FC38886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B38FC3C16AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B38FC377C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B38FC2EC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B38FC954545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AB9636AB630  Unknown               Unknown  Unknown
libc-2.17.so       00002AB9638EE377  gsignal               Unknown  Unknown
libc-2.17.so       00002AB9638EFA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AB96D6418B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AB96D645F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AB96D6460A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AB96D9A5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AB96D9C5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AB96CF742EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AB96CF3BEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AB967249934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F297E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F1FF846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F29D4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F22A55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F22AEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F29BC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AB96F298015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AB96F1CFB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AB96330E86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AB9633476AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AB9632FDC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AB9632723D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AB9638DA545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B1EE918E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B1EE93D1377  gsignal               Unknown  Unknown
libc-2.17.so       00002B1EE93D2A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B1EF31248B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B1EF3128F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B1EF31290A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B1EF3488593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B1EF34A8D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B1EF2A572EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B1EF2A1EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B1EECD2C934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCD85E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCCED846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCD8B4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCD1855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCD18EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCD89C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B1EFCD86015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B1EFCCBDB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B1EE8DF186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B1EE8E2A6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B1EE8DE0C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B1EE8D553D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B1EE93BD545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B922C6A2630  Unknown               Unknown  Unknown
libc-2.17.so       00002B922C8E5377  gsignal               Unknown  Unknown
libc-2.17.so       00002B922C8E6A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B92366398B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B923663DF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B923663E0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B923699D593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B92369BDD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9235F6C2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9235F33EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9230240934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B92402BAE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9240222846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B92402C04A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B924024D55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B924024DEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B92402BEC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B92402BB015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9237FC2B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B922C30586D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B922C33E6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B922C2F4C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B922C2693D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B922C8D1545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8CEDFC1630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8CEE204377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8CEE205A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B8D0004F8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B8D00053F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B8D000540A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B8D00383593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B8D003A3D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B8CF788A2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B8CF7851EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B8CF1B5F934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01BBEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01B26846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01BC44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01B5155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01B51EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01BC2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B8D01BBF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B8CF7FDAB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8CEDC2486D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8CEDC5D6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8CEDC13C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8CEDB883D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8CEE1F0545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B9BB2950630  Unknown               Unknown  Unknown
libc-2.17.so       00002B9BB2B93377  gsignal               Unknown  Unknown
libc-2.17.so       00002B9BB2B94A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B9BC4AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B9BC4AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B9BC4AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9BC4E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9BC4E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B9BC44232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B9BBBDDFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B9BB64EE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC665FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC65C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC66654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC65F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC65F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC6663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9BC6660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9BBBEC8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B9BB25B386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B9BB25EC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B9BB25A2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B9BB25173D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B9BB2B7F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0E0F673630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0E0F8B6377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0E0F8B7A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0E1960A8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0E1960EF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0E1960F0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0E1996E593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0E1998ED5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0E18F3D2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0E18F04EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0E13211934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B260E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B1C8846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B2664A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B1F355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B1F3EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B264C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0E1B261015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0E1B198B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0E0F2D686D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0E0F30F6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0E0F2C5C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0E0F23A3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0E0F8A2545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B67923B9630  Unknown               Unknown  Unknown
libc-2.17.so       00002B67925FC377  gsignal               Unknown  Unknown
libc-2.17.so       00002B67925FDA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B67A44918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B67A4495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B67A44960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B67A47C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B67A47E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B679BC822EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B679BC49EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B6795F57934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B67A6000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B67A5F68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B67A60064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B67A5F9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B67A5F93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B67A6004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B67A6001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B679BF90B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B679201C86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B67920556AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B679200BC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B6791F803D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B67925E8545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AEF76AC6630  Unknown               Unknown  Unknown
libc-2.17.so       00002AEF76D09377  gsignal               Unknown  Unknown
libc-2.17.so       00002AEF76D0AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AEF88AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AEF88AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AEF88AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AEF88E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AEF88E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AEF884232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AEF7FF55EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AEF7A664934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A6BEE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A626846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A6C44A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A65155E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A651EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A6C2C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AEF8A6BF015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AEF8A5F6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AEF7672986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AEF767626AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AEF76718C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AEF7668D3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AEF76CF5545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3779A1E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3779C61377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3779C62A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B37839B48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B37839B8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B37839B90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3783D18593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3783D38D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B37832E72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B37832AEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B377D5BC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B378D60FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B378D577846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B378D6154A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B378D5A255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B378D5A2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B378D613C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B378D610015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B378D547B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B377968186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B37796BA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3779670C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B37795E53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3779C4D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3CFB598630  Unknown               Unknown  Unknown
libc-2.17.so       00002B3CFB7DB377  gsignal               Unknown  Unknown
libc-2.17.so       00002B3CFB7DCA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B3D0552E8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B3D05532F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B3D055330A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B3D05892593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B3D058B2D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B3D04E612EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B3D04E28EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B3CFF136934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B3D07184E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B3D070EC846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D0718A4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D0711755E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D07117EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D07188C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B3D07185015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B3D070BCB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B3CFB1FB86D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B3CFB2346AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B3CFB1EAC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B3CFB15F3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B3CFB7C7545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002ADFF69F6630  Unknown               Unknown  Unknown
libc-2.17.so       00002ADFF6C39377  gsignal               Unknown  Unknown
libc-2.17.so       00002ADFF6C3AA68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AE008AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AE008AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AE008AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AE008E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AE008E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AE0084232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002ADFFFE85EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002ADFFA594934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A65FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A5C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A6654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A5F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A5F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AE00A660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002ADFFFF6EB9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002ADFF665986D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002ADFF66926AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002ADFF6648C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002ADFF65BD3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002ADFF6C25545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B08A01E5630  Unknown               Unknown  Unknown
libc-2.17.so       00002B08A0428377  gsignal               Unknown  Unknown
libc-2.17.so       00002B08A0429A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B08AA17B8B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B08AA17FF75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B08AA1800A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B08AA4DF593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B08AA4FFD5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B08A9AAE2EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B08A9A75EE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B08A3D83934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B08B40B5E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B08B401D846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B08B40BB4A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B08B404855E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B08B4048EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B08B40B9C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B08B40B6015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B08ABD09B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B089FE4886D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B089FE816AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B089FE37C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B089FDAC3D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B08A0414545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B8FFC93F630  Unknown               Unknown  Unknown
libc-2.17.so       00002B8FFCB82377  gsignal               Unknown  Unknown
libc-2.17.so       00002B8FFCB83A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B90068D58B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B90068D9F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B90068DA0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B9006C39593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B9006C59D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B90062082EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B90061CFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B90004DD934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B9010530E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B9010498846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B90105364A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B90104C355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B90104C3EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B9010534C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B9010531015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B9010468B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B8FFC5A286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B8FFC5DB6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B8FFC591C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B8FFC5063D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B8FFCB6E545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B87AFF2E630  Unknown               Unknown  Unknown
libc-2.17.so       00002B87B0171377  gsignal               Unknown  Unknown
libc-2.17.so       00002B87B0172A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B87B9EC48B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B87B9EC8F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B87B9EC90A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B87BA228593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B87BA248D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B87B97F72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B87B97BEEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B87B3ACC934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBB1AE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBA82846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBB204A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBAAD55E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBAADEAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBB1EC01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B87BBB1B015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B87BBA52B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B87AFB9186D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B87AFBCA6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B87AFB80C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B87AFAF53D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B87B015D545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B450EE4D630  Unknown               Unknown  Unknown
libc-2.17.so       00002B450F090377  gsignal               Unknown  Unknown
libc-2.17.so       00002B450F091A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B451CEF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B451CEF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B451CEF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B451D224593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B451D244D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B451C8232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B4517EDCEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B45129EB934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B451EA5FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B451E9C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B451EA654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B451E9F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B451E9F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B451EA63C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B451EA60015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B4517FC5B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B450EAB086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B450EAE96AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B450EA9FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B450EA143D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B450F07C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AD09EA50630  Unknown               Unknown  Unknown
libc-2.17.so       00002AD09EC93377  gsignal               Unknown  Unknown
libc-2.17.so       00002AD09EC94A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AD0B0AF08B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AD0B0AF4F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AD0B0AF50A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AD0B0E24593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AD0B0E44D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AD0B04232EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AD0A7EDFEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AD0A25EE934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B265FE29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B25C7846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B26654A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B25F255E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B25F2EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B2663C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AD0B2660015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AD0A7FC8B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AD09E6B386D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AD09E6EC6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AD09E6A2C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AD09E6173D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AD09EC7F545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002AD1F54BF630  Unknown               Unknown  Unknown
libc-2.17.so       00002AD1F5702377  gsignal               Unknown  Unknown
libc-2.17.so       00002AD1F5703A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002AD1FF4558B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002AD1FF459F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002AD1FF45A0A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002AD1FF7B9593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002AD1FF7D9D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002AD1FED882EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002AD1FED4FEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002AD1F905D934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002AD209182E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002AD2090EA846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002AD2091884A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002AD20911555E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD209115EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002AD209186C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002AD209183015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002AD1FFF16B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002AD1F512286D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002AD1F515B6AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002AD1F5111C3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002AD1F50863D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002AD1F56EE545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
cesm.exe           000000000294589B  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B0FBA2FD630  Unknown               Unknown  Unknown
libc-2.17.so       00002B0FBA540377  gsignal               Unknown  Unknown
libc-2.17.so       00002B0FBA541A68  abort                 Unknown  Unknown
libucs.so.0.0.0    00002B0FCC4918B5  ucs_fatal_error_m     Unknown  Unknown
libucs.so.0.0.0    00002B0FCC495F75  Unknown               Unknown  Unknown
libucs.so.0.0.0    00002B0FCC4960A4  ucs_log_dispatch      Unknown  Unknown
libuct_ib.so.0.0.  00002B0FCC7C5593  uct_ib_mlx5_compl     Unknown  Unknown
libuct_ib.so.0.0.  00002B0FCC7E5D5A  Unknown               Unknown  Unknown
libucp.so.0.0.0    00002B0FC3BC72EA  ucp_worker_progre     Unknown  Unknown
mca_pml_ucx.so     00002B0FC3B8EEE4  mca_pml_ucx_progr     Unknown  Unknown
libopen-pal.so.40  00002B0FBDE9B934  opal_progress         Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCE000E29  Unknown               Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCDF68846  comm_allgather_hc     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCE0064A7  hmca_coll_ml_hier     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCDF9355E  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCDF93EAB  hmca_coll_ml_comm     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCE004C01  hcoll_get_context     Unknown  Unknown
libhcoll.so.1.0.8  00002B0FCE001015  hcoll_create_cont     Unknown  Unknown
mca_coll_hcoll.so  00002B0FC3ED6B9C  mca_coll_hcoll_co     Unknown  Unknown
libmpi.so.40.20.3  00002B0FB9F6086D  mca_coll_base_com     Unknown  Unknown
libmpi.so.40.20.3  00002B0FB9F996AD  ompi_mpi_init         Unknown  Unknown
libmpi.so.40.20.3  00002B0FB9F4FC3D  MPI_Init              Unknown  Unknown
libmpi_mpifh.so.4  00002B0FB9EC43D7  PMPI_Init_f08         Unknown  Unknown
cesm.exe           00000000004352A4  cime_comp_mod_mp_         603  cime_comp_mod.F90
cesm.exe           00000000004370C5  MAIN__                     58  cime_driver.F90
cesm.exe           0000000000419512  Unknown               Unknown  Unknown
libc-2.17.so       00002B0FBA52C545  __libc_start_main     Unknown  Unknown
cesm.exe           0000000000419429  Unknown               Unknown  Unknown
srun: error: b1170: tasks 640-767: Aborted
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: b1167: tasks 256-383: Killed
srun: error: b1184: tasks 1024-1151: Killed
srun: error: b1178: tasks 896-1023: Killed
srun: error: b1158: tasks 128-255: Killed
srun: error: b1168: tasks 384-511: Killed
srun: error: b1169: tasks 512-579,581-639: Killed
slurmstepd: error: b1169 [4] pmixp_client_v2.c:210 [_errhandler] mpi/pmix: ERROR: Error handler invoked: status = -25: Success (0)
srun: error: Timed out waiting for job step to complete