数据湖探索 DLI-Spark作业相关:提交批处理作业

时间：2024-07-05 17:23:48

数据湖探索 DLI Python SDK

提交批处理作业

DLI 提供执行批处理作业的接口。您可以使用该接口执行批处理作业。示例代码如下：

  
       def submit_spark_batch_job(dli_client, batch_queue_name, batch_job_info):
    try:
        batch_job = dli_client.submit_spark_batch_job(batch_queue_name, batch_job_info)
    except DliException as e:
        print(e)
        return

    print(batch_job.job_id)
    while True:
        time.sleep(3)
        job_status = batch_job.get_job_status()
        print('Job status: {0}'.format(job_status))
        if job_status == 'dead' or job_status == 'success':
            break

    logs = batch_job.get_driver_log(500)
    for log_line in logs:
        print(log_line)