应用平台 APPSTAGE-算子清洗功能介绍:算子组合使用样例

时间：2024-11-08 10:18:31

算子组合使用样例

日志样例

2022-11-15 19:51:43.735 [20221115T115143]|AI||system|system|inner_ip|ADD|LogRecord|AIOperationLog(id=12d6ffbd-f371-4222-aa69-167bcd7ba3ee, serviceId=null, identityId=1111111111111, userId=test, timestamp=2022-11-15 19:51:43.67, operationType=WELINK_TRIGGER)||Y - serviceId:null

算子功能

对于非json格式的如何转化成json直接解析拿到key、value。例如id=12d6ffbd-f371-4222-aa69-167bcd7ba3ee, serviceId=null, identityId=1111111111111, userId=test, timestamp=2022-11-15 19:51:43.67, operationType=WELINK_TRIGGER。

算子脚本

- dissect:
    tokenizer: '%{time}|%{thread}|%{logVersion}|%{uid}|%{sign}|%{traceId}|%{cc}|%{operationLog}|%{bb}|%{aa}'
    field: "message"
    target_prefix: ""
    trim_values: "none"
    ignore_failure: true

- replace:
    fail_on_error: false
    ignore_missing: true
    fields:          
     - {field: "bb",pattern: 'AIOperationLog\(',replacement: "{\""}
     - {field: "bb",pattern: '\)',replacement: "\"}"}
     - {field: "bb",pattern: ' ,',replacement:"\""}
- if:
    contains:
        "operationLog": "LogRecord"
  then:
     - replace:
         fail_on_error: false
         ignore_missing: true
         fields:          
          - {field: "bb",pattern: '=',replacement: "\":\""}
          - {field: "bb",pattern: ' ',replacement:""}
          - {field: "bb",pattern: '\,',replacement: "\",\""}
     - decode_json:
        ignore_missing: true
        ignore_failure: true
        fields:
          - field: "bb"
            filters:
             - {target_field: "id",type: "string",filter_keys: ["id"]}
             - {target_field: "serviceId",type: "string",filter_keys: ["serviceId"]}
             - {target_field: "identityId",type: "string",filter_keys: ["identityId"]}
             - {target_field: "userId",type: "string",filter_keys: ["userId"]}
             - {target_field: "timestamp",type: "string",filter_keys: ["timestamp"]}
             - {target_field: "operationType",type: "string",filter_keys: ["operationType"]}  
     - drop_fields:
        fields: ["time","thread","logVersion","uid","sign","traceId","cc","aa","bb"]
        ignore_missing: true          
- drop_event:
     when:
       not:
         equals:
            "operationLog": "LogRecord"

上一篇：应用平台 APPSTAGE-算子清洗功能介绍:功能说明

下一篇：应用平台 APPSTAGE-算子清洗功能介绍:功能说明