AI开发平台MODELARTS-NPU Snt9B集合通信算子单机多卡性能测试指导:操作步骤

时间:2024-07-08 09:25:27

操作步骤

  1. 安装cann-toolkit。下载Ascend-cann-toolkit_6.3.T205_linux-aarch64.run,并执行:

    ./Ascend-cann-toolkit_6.3.T205_linux-aarch64.run --quiet --full

  2. 安装mpich-3.2.1.tar.gz。下载mpich-3.2.1.tar.gz,并执行:

    mkdir -p /home/mpich
    mv /root/mpich-3.2.1.tar.gz /home/
    cd /home/;tar -zxvf mpich-3.2.1.tar.gz
    cd /home/mpich-3.2.1
    ./configure --prefix=/home/mpich --disable-fortran
    make && make install

    设置环境变量和编译hccl算子。

    export PATH=/home/hjh/mpich/bin:$PATH
    cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test
    export LD_LIBRARY_PATH=/home/hjh/mpich/lib/:/usr/local/Ascend/ascend-toolkit/latest/lib64:$LD_LIBRARY_PATH
    make MPI_HOME=/home/hjh/mpich ASCEND_DIR=/usr/local/Ascend/ascend-toolkit/latest

    算子编译完成后显示内容:

    图1 算子编译完成

  3. 测试通信算子。

    执行命令:

    export HCCL_BUFFSIZE=2048
    • 单机all_reduce_test
      cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test
      • 单机单卡:
        mpirun -n 1 ./bin/all_reduce_test -b 8 -e 1024M -f 2 -p 8 
      • 单机多卡:
        mpirun -n 8 ./bin/all_reduce_test -b 8 -e 2048M -f 2 -p 8
        图2 单机all_reduce_test
    • 单机reduce_scatter_test
      cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test

      单机多卡:

      mpirun -n 8 ./bin/reduce_scatter_test -b 8K -e 2048M -f 2 -d fp32 -o sum -p 8
      图3 单机reduce_scatter_test
    • 单机all_gather_test
      cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test
      单机多卡:
      mpirun -n 8 ./bin/all_gather_test -b 8K -e 2048M -f 2 -d fp32 -p 8
      图4 单机all_gather_test
    • 单机alltoallv_test
      cd /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test

      单机多卡:

      mpirun -n 8 ./bin/alltoallv_test -b 8K -e 2048M -f 2 -d fp32 -p 8
      图5 单机alltoallv_test

    以下为昇腾相关的常用命令:

    • 查看网卡状态UP和DOWN
      for i in {0..7};do hccn_tool -i ${i} -link -g;done
    • 查看单节点内网卡IP连通性
      for i in $(seq 0 7);do hccn_tool -i $i -net_health -g;done
    • 当某网卡已经开始RoCE带宽测试时,再次启动任务会报错如下,解决这个问题需要关闭roce_test任务。
      hccn_tool -i 7 -roce_test reset
      图6 RoCE带宽测试

support.huaweicloud.com/usermanual-modelarts-lite/usermanual-modelarts-lite-0219.html