Software And Hardware Codesign Of Smartnic-Based Heterogeneous Hpc Clusters With Machine Learning Case Studies