云服务器内容精选

  • 示例 创建示例所需视图: Create schema test1; Use test1; Create table t1(id int, name string); Create view v1 as select * from t1; Create view v2 as select * from t1; Create view t1view as select * from t1; Create view t2view as select * from t1; Show views; Table -------- t1view t2view v1 v2 (4 rows) Show views like 'v1'; Table ------- v1 (1 row) Show views 'v_'; Table ------- v1 v2 (2 rows) show views like 't*'; Table -------- t1view t2view Show views in test1; Table -------- t1view t2view v1 v2 (4 rows)
  • 示例 创建一个新表orders,使用子句with指定创建表的存储格式、存储位置、以及是否为外表。 通过“auto.purge”参数可以指定涉及到数据移除操作(如DROP、DELETE、INSERT OVERWRITE、TRUNCATE TABLE)时是否清除相关数据: "auto.purge"='true'时,清除元数据和数据文件。 "auto.purge"='false'时,仅清除元数据,数据文件会移入HDFS回收站。默认值为“false”,且不建议用户修改此属性,避免数据删除后无法恢复。 CREATE TABLE orders ( orderkey bigint, orderstatus varchar, totalprice double, orderdate date ) WITH (format = 'ORC', location='/user',orc_compress='ZLIB',external=true, "auto.purge"=false); -- 通过DESC FORMATTED 语句,可以查看建表的详细信息 desc formatted orders ; Describe Formatted Table ------------------------------------------------------------------------------ # col_name data_type comment orderkey bigint orderstatus varchar totalprice double orderdate date # Detailed Table Information Database: default Owner: admintest LastAccessTime: 0 Location: hdfs://hacluster/user Table Type: EXTERNAL_TABLE # Table Parameters: EXTERNAL TRUE auto.purge false orc.compress.size 262144 orc.compression.codec ZLIB orc.row.index.stride 10000 orc.stripe.size 67108864 presto_query_id 20220812_084110_00050_srknk@default@HetuEngine presto_version 1.2.0-h0.cbu.mrs.320.r1-SNAPSHOT transient_lastDdlTime 1660293670 # Storage Information SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: serialization.format 1 (1 row) 创建一个新表,指定Row format: --建表时,指定表的字段分隔符为‘,’号(如果创建外表,要求数据文件中的每条记录的字段是以逗号进行分隔) CREATE TABLE student( id string,birthday string, grade int, memo string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; --建表时,指定字段分隔符为'\t',换行符为'\n' CREATE TABLE test( id int, name string , tel string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE; 如果表orders不存在,则创建表orders,并且增加表注释和列注释: CREATE TABLE IF NOT EXISTS orders ( orderkey bigint, orderstatus varchar, totalprice double COMMENT 'Price in cents.', orderdate date ) COMMENT 'A table to keep track of orders.'; insert into orders values (202011181113,'online',9527,date '2020-11-11'), (202011181114,'online',666,date '2020-11-11'), (202011181115,'online',443,date '2020-11-11'), (202011181115,'offline',2896,date '2020-11-11'); 使用表orders的列定义创建表bigger_orders: CREATE TABLE bigger_orders ( another_orderkey bigint, LIKE orders, another_orderdate date ); SHOW CREATE TABLE bigger_orders ; Create Table --------------------------------------------------------------------- CREATE TABLE hive.default.bigger_orders ( another_orderkey bigint, orderkey bigint, orderstatus varchar, totalprice double, ordersdate date, another_orderdate date ) WITH ( external = false, format = 'ORC', location = 'hdfs://hacluster/user/hive/warehouse/bigger_orders', orc_compress = 'GZIP', orc_compress_size = 262144, orc_row_index_stride = 10000, orc_stripe_size = 67108864 ) (1 row) 标号① 建表示例: CREATE EXTERNAL TABLE hetu_test (orderkey bigint, orderstatus varchar, totalprice double, orderdate date) PARTITIONED BY(ds int) SORT BY (orderkey, orderstatus) COMMENT 'test' STORED AS ORC LOCATION '/user' TBLPROPERTIES (orc_compress = 'SNAPPY', orc_compress_size = 6710422, orc_bloom_filter_columns = 'orderstatus,totalprice'); 标号② 建表示例: CREATE EXTERNAL TABLE hetu_test1 (orderkey bigint, orderstatus varchar, totalprice double, orderdate date) COMMENT 'test' PARTITIONED BY(ds int) CLUSTERED BY (orderkey, orderstatus) SORTED BY (orderkey, orderstatus) INTO 16 BUCKETS STORED AS ORC LOCATION '/user' TBLPROPERTIES (orc_compress = 'SNAPPY', orc_compress_size = 6710422, orc_bloom_filter_columns = 'orderstatus,totalprice'); 标号③ 建表示例: CREATE TABLE hetu_test2 (orderkey bigint, orderstatus varchar, totalprice double, orderdate date, ds int) COMMENT 'This table is in Hetu syntax' WITH (partitioned_by = ARRAY['ds'], bucketed_by = ARRAY['orderkey', 'orderstatus'], sorted_by = ARRAY['orderkey', 'orderstatus'], bucket_count = 16, orc_compress = 'SNAPPY', orc_compress_size = 6710422, orc_bloom_filter_columns = ARRAY['orderstatus', 'totalprice'], external = true, format = 'orc', location = '/user');
  • 限制 session属性可以设置bucket_count,默认值为-1,表示未设置。创建分区表时,如果bucket_count为-1且建表语句中未设置buckets,则使用默认值16。 默认外部表存储位置/user/hive/warehouse/{schema_name}/{table_name},其中{schema_name}为建表时使用的schema,{table_name}为表名。 指定属性“transactional=true”可以让表支持“原子性、一致性、隔离性、持久性”写入的事务能力,但是将表定义为事务表后,无法通过设置“transactional=false”将其退化为非事务表。 transactional='true'或 '0'在执行过程中不会进行类型转换,所以这种写法会抛出异常: Cannot convert ['true'] to boolean Cannot convert ['0'] to boolean 默认不允许向托管表(表属性external = true)插入数据,如需使用该功能,可参考注意事项,添加hive自定义属性:hive.non-managed-table-writes-enabled=true。 Mppdb有一个限制,数据库的标识符的最大长度为63,如果把标识符命名超过了最大长度,那么会被自动截取掉超出的部分,只留下最大长度的标识符。 跨域场景不支持建表。
  • 语法 ① CREATE TABLE [ IF NOT EXISTS ] [catalog_name.][db_name.]table_name ( { column_name data_type [ NOT NULL ] [ COMMENT col_comment] [ WITH ( property_name = expression [, ...] ) ] | LIKE existing_table_name [ { INCLUDING | EXCLUDING } PROPERTIES ] } [, ...] ) [ COMMENT table_comment ] [ WITH ( property_name = expression [, ...] ) ] ② CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [catalog_name.][db_name.]table_name ( { column_name data_type [ NOT NULL ] [ COMMENT comment ] [ WITH ( property_name = expression [, ...] ) ] | LIKE existing_table_name [ { INCLUDING | EXCLUDING } PROPERTIES ] } [, ...] ) [COMMENT 'table_comment'] [PARTITIONED BY(col_name data_type, ....)] [CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name, col_name, ...)] INTO num_buckets BUCKETS] ] [ROW FORMAT row_format] [STORED AS file_format] [LOCATION 'hdfs_path'] [TBLPROPERTIES (orc_table_property = value [, ...] ) ] ③ CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [catalog_name.][db_name.]table_name ( { column_name data_type [ NOT NULL ] [ COMMENT comment ] [ WITH ( property_name = expression [, ...] ) ] | LIKE existing_table_name [ { INCLUDING | EXCLUDING } PROPERTIES ] } [, ...] ) [PARTITIONED BY(col_name data_type, ....)] [SORT BY ([column [, column ...]])] [COMMENT 'table_comment'] [ROW FORMAT row_format] [STORED AS file_format] [LOCATION 'hdfs_path'] [TBLPROPERTIES (orc_table_property = value [, ...] ) ]
  • 描述 使用CREATE TABLE创建一个具有指定列的、新的空表。使用CREATE TABLE AS创建带数据的表。 使用可选参数IF NOT EXISTS,如果表已经存在则不会报错。 WITH子句可用于在新创建的表或单列上设置属性,如表的存储位置(location)、是不是外表(external)等。 LIKE子句用于在新表中包含来自现有表的所有列定义。可以指定多个LIKE子句,从而允许从多个表中复制列。如果指定了INCLUDING PROPERTIES,则将所有表属性复制到新表中。如果WITH子句指定的属性名称与复制的属性名称相同,则将使用WITH子句中的值。默认是EXCLUDING PROPERTIES属性,而且最多只能为一个表指定INCLUDING PROPERTIES属性。 PARTITIONED BY能够用于指定分区的列;CLUSTERED BY能够被用于指定分桶的列;SORT BY和 SORTED BY能够用于给指定的分桶列进行排序;BUCKETS能够被用于指定分桶数;EXTERNAL可用于指定创建外部表;STORED AS能被用于指定文件存储的格式;LOCATION能被用于指定在HDFS上存储的路径。 想要查看支持哪些column属性,可以运行以下命令,会显示当前对接的catalog分别支持哪些列属性。 SELECT * FROM system.metadata.column_properties; 想要查看支持哪些table属性,可以运行以下命令: SELECT * FROM system.metadata.table_properties; 下表为catalog为hive时的查询结果。 SELECT * FROM system.metadata.table_properties where catalog_name = 'hive'; catalog_name property_name default_value type description hive auto_purge false boolean Skip trash when table or partition is deleted hive avro_schema_url - varchar URI pointing to Avro schema for the table hive bucket_count 0 integer Number of buckets hive bucketed_by [] array(varchar) Bucketing columns hive bucketing_version - integer Bucketing version hive csv_escape - varchar CS V escape character hive csv_quote - varchar CSV quote character hive csv_separator - varchar CSV separator character hive external_location - varchar File system location URI for external table hive format ORC varchar Hive storage format for the table. Possible values: [ORC, PARQUET, AVRO, RCBINARY, RCTEXT, SEQUENCEFILE, JSON, TEXTFILE, TEXTFILE_MULTIDELIM, CSV] hive orc_compress GZIP varchar Compression codec used. Possible values: [NONE, SNAPPY, LZ4, ZSTD, GZIP, ZLIB] hive orc_compress_size 262144 bigint orc compression size hive orc_row_index_stride 10000 integer no. of row index strides hive orc_stripe_size 67108864 bigint orc stripe size hive orc_bloom_filter_columns [] array(varchar) ORC Bloom filter index columns hive orc_bloom_filter_fpp 0.05 double ORC Bloom filter false positive probability hive partitioned_by [] array(varchar) Partition columns hive sorted_by [] array(varchar) Bucket sorting columns hive textfile_skip_footer_line_count - integer Number of footer lines hive textfile_skip_header_line_count - integer Number of header lines hive transactional false boolean Is transactional property enabled
  • 示例 显示能够创建orders表的SQL 语句: CREATE TABLE orders ( orderkey bigint, orderstatus varchar, totalprice double, orderdate date ) WITH (format = 'ORC', location='/user',orc_compress='ZLIB',external=true, "auto.purge"=false); show create table orders; Create Table ------------------------------------------------- CREATE TABLE hive.default.orders ( orderkey bigint, orderstatus varchar, totalprice double, orderdate date ) WITH ( external_location = 'hdfs://hacluster/user', format = 'ORC', orc_compress = 'ZLIB', orc_compress_size = 262144, orc_row_index_stride = 10000, orc_stripe_size = 67108864 ) (1 row)
  • 示例 列出当前catalog所有的schemas: SHOW SCHEMAS; 列出指定catalog下的schema_name前缀为"t"的所有schemas: SHOW SCHEMAS FROM hive LIKE 't%'; --等价写法: SHOW SCHEMAS IN hive LIKE 't%'; 如果匹配字符串中有字符与通配符冲突,可以指定转义字符来标识,示例为查询hive这个catalog下,schema_name前缀为“pm_”的所有schema,转义字符为“/”: SHOW SCHEMAS IN hive LIKE 'pm/_%' ESCAPE '/';
  • 示例 用指定列的查询结果创建新表orders_column_aliased: CREATE TABLE orders_column_aliased (order_date, total_price) AS SELECT orderdate, totalprice FROM orders; 用表orders的汇总结果新建一个表orders_by_data: CREATE TABLE orders_by_date COMMENT 'Summary of orders by date' WITH (format = 'ORC') AS SELECT orderdate, sum(totalprice) AS price FROM orders GROUP BY orderdate; 如果表orders_by_date不存在,则创建表orders_by_date: CREATE TABLE IF NOT EXISTS orders_by_date AS SELECT orderdate, sum(totalprice) AS price FROM orders GROUP BY orderdate; 用和表orders具有相同schema创建新表empty_orders table,但是没数据: CREATE TABLE empty_orders AS SELECT * FROM orders WITH NO DATA; 使用VALUES 创建表,参考 VALUES。 分区表示例: CREATE EXTERNAL TABLE hetu_copy(corderkey, corderstatus, ctotalprice, corderdate, cds) PARTITIONED BY(cds) SORT BY (corderkey, corderstatus) COMMENT 'test' STORED AS orc LOCATION '/user/hetuserver/tmp' TBLPROPERTIES (orc_bloom_filter_fpp = 0.3, orc_compress = 'SNAPPY', orc_compress_size = 6710422, orc_bloom_filter_columns = 'corderstatus,ctotalprice') as select * from hetu_test; CREATE TABLE hetu_copy1(corderkey, corderstatus, ctotalprice, corderdate, cds) WITH (partitioned_by = ARRAY['cds'], bucketed_by = ARRAY['corderkey', 'corderstatus'], sorted_by = ARRAY['corderkey', 'corderstatus'], bucket_count = 16, orc_compress = 'SNAPPY', orc_compress_size = 6710422, orc_bloom_filter_columns = ARRAY['corderstatus', 'ctotalprice'], external = true, format = 'orc', location = '/user/hetuserver/tmp ') as select * from hetu_test;
  • 语法 CREATE [EXTERNAL]① TABLE [IF NOT EXISTS] [catalog_name.][db_name.]table_name [ ( column_alias, ... ) ] [[PARTITIONED BY ①(col_name, ....)] [SORT BY① ([column [, column ...]])] ]① [COMMENT 'table_comment'] [ WITH ( property_name = expression [, ...] ) ]② [[STORED AS file_format]① [LOCATION 'hdfs_path']① [TBLPROPERTIES (orc_table_property = value [, ...] ) ] ]① AS query [ WITH [ NO ] DATA ]②
  • 描述 “ALTER VIEW view_name AS select_statement;”用于改变已存在的视图的定义,语法效果与CREATE OR REPLACE VIEW类似。 “ALTER VIEW view_name SET TBLPROPERTIES table_properties;”中table_properties格式为 (property_name = property_value, property_name = property_value, ...)。 视图可以包含Limit和ORDER BY子句,如果关联视图的查询语句也包含了这类子句,则最后执行结果将根据视图的子句运算后得到。例如视图V指定了返回5条数据,而关联查询为select * from V limit 10,则最终只有5条数据返回。
  • 示例 CREATE OR REPLACE VIEW tv_view as SELECT id,name from (values (1, 'HetuEngine')) as x(id,name); SELECT * FROM tv_view; id | name ----|------ 1 | HetuEngine (1 row) ALTER VIEW tv_view as SELECT id, brand FROM (VALUES (1, 'brand_1', 100), (2, 'brand_2', 300) ) AS x (id, brand, price); SELECT * FROM tv_view; id | brand ----|--------- 1 | brand_1 2 | brand_2 (2 rows) ALTER VIEW tv_view SET TBLPROPERTIES ('comment' = 'This is a new comment'); show tblproperties tv_view; SHOW TBLPROPERTIES -------------------------------------------------------------------- comment 'This is a new comment' presto_query_id '20210325_034712_00040_f63xj@default@HetuEngine' presto_version presto_view 'true' transient_lastDdlTime '1616644032' (1 row)
  • 描述 列出catalogName为mv中的所有物化视图以及对应的数据表。如果希望只查看某个schema中的物化视图,可以使用子句[IN/FROM schema_name] 通过可选子句“LIKE”,筛选视图名满足规则运算表达式的视图,如果不使用这个子句,会列举所有视图。匹配的视图会按字母顺序排列。 目前规则运算表达式支持“*”或“%”用于匹配任何字符,下划线“_”用于匹配一个字符,或“|”用于条件连接两个或多个条件。