AI开发平台ModelArts-启动推理服务:离线推理

时间：2025-02-12 15:14:12

AI开发平台ModelArts 推理服务部署

离线推理

编辑一个python脚本，脚本内容如下，运行该脚本使用ascend-vllm进行模型离线推理。

from vllm import LLM, SamplingParamsdef main():    prompts = [        "Hello, my name is",        "The president of the United States is",        "The capital of France is",        "The future of AI is",    ]    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)    model_path = "/path/to/model"    llm = LLM(model=model_path, tensor_parallel_size=1, max_model_len=8192)    outputs = llm.generate(prompts, sampling_params)    # Print the outputs.    for output in outputs:        prompt = output.prompt        generated_text = output.outputs[0].text        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")if __name__=="__main__":    main()

上一篇：AI开发平台ModelArts-guided-decoding:什么是guided-decoding

下一篇：AI开发平台ModelArts-guided-decoding:什么是guided-decoding