云服务器内容精选

推理服务部署

AI开发平台ModelArts-启动推理服务:离线推理

离线推理编辑一个python脚本，脚本内容如下，运行该脚本使用ascend-vllm进行模型离线推理。 from vllm import LLM, SamplingParamsdef main(): prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) model_path = "/path/to/model" llm = LLM(model=model_path, tensor_parallel_size=1, max_model_len=8192) outputs = llm.generate(prompts, sampling_params) # Print the outputs. for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")if __name__=="__main__": main()

AI开发平台ModelArts 推理服务部署