MAPREDUCE服务 MRS-Spark REST API接口介绍:REST API增强
REST API增强
- SQL相关的命令:获取所有SQL语句和执行时间最长的SQL语句
- SparkUI命令:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/SQL"
其中192.168.195.232为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1476947670799_0053为在YARN中的应用ID。
可以在命令后的url路径增加相应的参数设置,搜索对应的SQL语句。
例如,查看100条sql语句:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/SQL?limit=100"
查看正在运行的参数:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/SQL?completed=false"
- JobHistory命令:
curl -k -i --negotiate -u: "https://192.168.227.16:4040/api/v1/applications/application_1478570725074_0004/SQL"
其中192.168.227.16为JobHistory节点的业务IP,4040为JobHistory的端口号,application_1478570725074_0004为应用ID。
- 结果:
SparkUI命令和JobHistory命令的查询结果均为:
{ "longestDurationOfCompletedSQL" : [ { "id" : 0, "status" : "COMPLETED", "description" : "getCallSite at SQLExecution.scala:48", "submissionTime" : "2016/11/08 15:39:00", "duration" : "2 s", "runningJobs" : [ ], "successedJobs" : [ 0 ], "failedJobs" : [ ] } ], "sqls" : [ { "id" : 0, "status" : "COMPLETED", "description" : "getCallSite at SQLExecution.scala:48", "submissionTime" : "2016/11/08 15:39:00", "duration" : "2 s", "runningJobs" : [ ], "successedJobs" : [ 0 ], "failedJobs" : [ ] }] }
- 结果分析:
通过这个命令,可以查询当前应用的所有SQL语句的信息(即结果中“sqls”的部分),执行时间最长的SQL语句的信息(即结果中“longestDurationOfCompletedSQL”的部分)。每个SQL语句的信息如下表3。
- SparkUI命令:
- JDBC Server相关的命令:获取连接数,正在执行的SQL数,所有session信息,所有SQL的信息
- 命令:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1476947670799_0053/api/v1/applications/application_1476947670799_0053/sqlserver"
其中192.168.195.232为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1476947670799_0053为在YARN中的应用ID。
- 结果:
{ "sessionNum" : 1, "runningSqlNum" : 0, "sessions" : [ { "user" : "spark", "ip" : "192.168.169.84", "sessionId" : "9dfec575-48b4-4187-876a-71711d3d7a97", "startTime" : "2016/10/29 15:21:10", "finishTime" : "", "duration" : "1 minute 50 seconds", "totalExecute" : 1 } ], "sqls" : [ { "user" : "spark", "jobId" : [ ], "groupId" : "e49ff81a-230f-4892-a209-a48abea2d969", "startTime" : "2016/10/29 15:21:13", "finishTime" : "2016/10/29 15:21:14", "duration" : "555 ms", "statement" : "show tables", "state" : "FINISHED", "detail" : "== Parsed Logical Plan ==\nShowTablesCommand None\n\n== Analyzed Logical Plan ==\ntableName: string, isTemporary: boolean\nShowTablesCommand None\n\n== Cached Logical Plan ==\nShowTablesCommand None\n\n== Optimized Logical Plan ==\nShowTablesCommand None\n\n== Physical Plan ==\nExecutedCommand ShowTablesCommand None\n\nCode Generation: true" } ] }
- 结果分析:
- 命令:
- JDBC api增强通过beeline里面获取的executionID 取消当前正在执行的SQL
- Streaming相关的命令:获取平均输入频率,平均调度时延,平均执行时长,总时延平均值
- 命令:
curl -k -i --negotiate -u: "https://192.168.195.232:8090/proxy/application_1477722033672_0008/api/v1/applications/application_1477722033672_0008/streaming/statistics"
其中192.168.195.232为ResourceManager主节点的业务IP,8090为ResourceManager的端口号,application_1477722033672_0008为在YARN中的应用ID。
- 结果:
{ "startTime" : "2018-12-25T08:58:10.836GMT", "batchDuration" : 1000, "numReceivers" : 1, "numActiveReceivers" : 1, "numInactiveReceivers" : 0, "numTotalCompletedBatches" : 373, "numRetainedCompletedBatches" : 373, "numActiveBatches" : 0, "numProcessedRecords" : 1, "numReceivedRecords" : 1, "avgInputRate" : 0.002680965147453083, "avgSchedulingDelay" : 14, "avgProcessingTime" : 47, "avgTotalDelay" : 62 }
- 结果分析:
通过这个命令,可以查询当前Streaming应用的平均输入频率(events/sec),平均调度时延(ms),平均执行时长(ms),总时延平均值(ms)。
- 命令:
- 华为云CodeArts API_接口迁移_接口导入
- MapReduce服务_什么是Hue_如何使用Hue
- 调用GaussDB(DWS) API接口_数据仓库服务调用API_如何调用API_在DWS中调用API
- 华为云CodeArts API_API测试_接口测试_关键字
- 华为云CodeArts API_如何利用CodeArts API设计一个接口?
- 快递查询_快递物流查询_快递查询API接口
- 华为云CodeArts API_API质量_API规范_API审核
- 华为云CodeArts API_APIFirst_API设计_API设计规范_API全生命周期
- 华为云CodeArts API_什么是API_APIFirst_API设计
- API Explorer 产品介绍与快速入门