云服务器内容精选

  • ma-cli ma-job训练作业支持的命令 使用ma-cli ma-job命令可以提交训练作业,查询训练作业日志、事件、使用的AI引擎、资源规格及停止训练作业等。 $ ma-cli ma-job -h Usage: ma-cli ma-job [OPTIONS] COMMAND [ARGS]... ModelArts job submission and query jod details. Options: -h, -H, --help Show this message and exit. Commands: delete Delete training job by job id. get-engine Get job engines. get-event Get job running event. get-flavor Get job flavors. get-job Get job details. get-log Get job log details. get-pool Get job engines. stop Stop training job by job id. submit Submit training job. 表1 训练作业支持的命令 命令 命令详情 get-job 查询ModelArts训练作业列表及详情。 get-log 查询ModelArts训练作业运行日志。 get-engine 查询ModelArts训练AI引擎。 get-event 查询ModelArts训练作业事件。 get-flavor 查询ModelArts训练资源规格。 get-pool 查询ModelArts训练专属池。 stop 停止ModelArts训练作业。 submit 提交ModelArts训练作业。 delete 删除指定作业id的训练作业。 父主题: 使用ma-cli ma-job命令提交ModelArts训练作业
  • 基于ModelArts预置镜像提交训练作业 指定命令行options参数提交训练作业 ma-cli ma-job submit --code-dir obs://your-bucket/mnist/code/ \ --boot-file main.py \ --framework-type PyTorch \ --working-dir /home/ma-user/modelarts/user-job-dir/code \ --framework-version pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64 \ --data-url obs://your-bucket/mnist/dataset/MNIST/ \ --log-url obs://your-bucket/mnist/logs/ \ --train-instance-type modelarts.vm.cpu.8u \ --train-instance-count 1 \ -q 使用预置镜像的train.yaml样例:
  • 基于 自定义镜像 创建训练作业 指定命令行options参数提交训练作业 ma-cli ma-job submit --image-url atelier/pytorch_1_8:pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64-20220926104358-041ba2e \ --code-dir obs://your-bucket/mnist/code/ \ --user-command "export LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH && cd /home/ma-user/modelarts/user-job-dir/code && /home/ma-user/anaconda3/envs/PyTorch-1.8/bin/python main.py" \ --data-url obs://your-bucket/mnist/dataset/MNIST/ \ --log-url obs://your-bucket/mnist/logs/ \ --train-instance-type modelarts.vm.cpu.8u \ --train-instance-count 1 \ -q 使用自定义镜像的train.yaml样例: # .ma/train.yaml样例(自定义镜像) image-url: atelier/pytorch_1_8:pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64-20220926104358-041ba2e user-command: export LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH && cd /home/ma-user/modelarts/user-job-dir/code && /home/ma-user/anaconda3/envs/PyTorch-1.8/bin/python main.py train-instance-type: modelarts.vm.cpu.8u train-instance-count: 1 data-url: obs://your-bucket/mnist/dataset/MNIST/ code-dir: obs://your-bucket/mnist/code/ log-url: obs://your-bucket/mnist/logs/ ##[Optional] Uncomment to set uid when use custom image mode uid: 1000 ##[Optional] Uncomment to upload output file/dir to OBS from training platform output: - name: output_dir obs_path: obs://your-bucket/mnist/output1/ ##[Optional] Uncomment to download input file/dir from OBS to training platform input: - name: data_url obs_path: obs://your-bucket/mnist/dataset/MNIST/ ##[Optional] Uncomment pass hyperparameters parameters: - epoch: 10 - learning_rate: 0.01 - pretrained: ##[Optional] Uncomment to use dedicated pool pool_id: pool_xxxx ##[Optional] Uncomment to use volumes attached to the training job volumes: - efs: local_path: /xx/yy/zz read_only: false nfs_server_path: xxx.xxx.xxx.xxx:/
  • 示例 基于yaml文件提交训练作业 ma-cli ma-job submit ./train-job.yaml 基于命令行和预置镜像pytorch1.8-cuda10.2-cudnn7-ubuntu18.04提交训练作业。 ma-cli ma-job submit --code-dir obs://automation-use-only/Original/TrainJob/TrainJob-v2/pytorch1.8.0_cuda10.2/code/ \ --boot-file test-pytorch.py \ --framework-type PyTorch \ --working-dir /home/ma-user/modelarts/user-job-dir/code \ --framework-version pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64 \ --data-url obs://automation-use-only/Original/TrainJob/TrainJob-v2/pytorch1.8.0_cuda10.2/data/ \ --log-url obs://automation-use-only/Original/TrainJob/TrainJob-v2/pytorch1.8.0_cuda10.2/data/logs/ \ --train-instance-type modelarts.vm.cpu.8u \ --train-instance-count 1 \