MAPREDUCE服务 MRS-输入文件数超出设置限制导致任务执行失败:问题背景与现象

时间：2024-08-27 10:23:54

MAPREDUCE服务 MRS

问题背景与现象

Hive执行查询操作时报Job Submission failed with exception 'java.lang.RuntimeException(input file number exceeded the limits in the conf;input file num is: 2380435,max heap memory is: 16892035072,the limit conf is: 500000/4)'，此报错中具体数值根据实际情况会发生变化，具体报错信息如下：

ERROR : Job Submission failed with exception 'java.lang.RuntimeException(input file numbers exceeded the limits in the conf;
 input file num is: 2380435 ,
 max heap memory is: 16892035072 ,
 the limit conf is: 500000/4)'
java.lang.RuntimeException: input file numbers exceeded the limits in the conf;
 input file num is: 2380435 ,
 max heap memory is: 16892035072 ,
 the limit conf is: 500000/4
	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.checkFileNum(ExecDriver.java:545)
	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:430)
	at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
	at org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
	at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
	at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

上一篇：MAPREDUCE服务 MRS-Hive任务执行中报栈内存溢出导致任务执行失败:问题背景与现象

下一篇：MAPREDUCE服务 MRS-beeline客户端报错“OutOfMemoryError”:解决办法