数据湖探索 DLI-怎样将OBS表映射为DLI的分区表?:操作步骤

时间:2024-11-06 21:53:30

操作步骤

该示例将car_info数据,以day字段为分区字段,parquet为编码格式,转储数据到OBS。更多内容请参考《 数据湖探索 Flink SQL语法参考》。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
create sink stream car_infos (
  carId string,
  carOwner string,
  average_speed double,
  day string
  ) partitioned by (day)
  with (
    type = "filesystem",
    file.path = "obs://obs-sink/car_infos",
    encode = "parquet",
    ak = "{{myAk}}",
    sk = "{{mySk}}"
);

数据最终在OBS中的存储目录结构为:obs://obs-sink/car_infos/day=xx/part-x-x。

数据生成后,可通过如下SQL语句建立OBS分区表,用于后续批处理:

  1. 创建OBS分区表。
    1
    2
    3
    4
    5
    6
    7
    8
    create table car_infos (
      carId string,
      carOwner string,
      average_speed double
    )
      partitioned by (day string)
      stored as parquet
      location 'obs://obs-sink/car-infos';
    
  2. 从关联OBS路径中恢复分区信息。
    1
    alter table car_infos recover partitions;
    
support.huaweicloud.com/dli_faq/dli_03_0089.html