MAPREDUCE服务 MRS-连接postgresql或者gaussdb时报错:回答

时间:2024-11-25 09:23:33

回答

  • 场景一:(import场景)使用sqoop import命令抽取开源postgre到 MRS hdfs或hive等。
    • 问题现象:

      使用sqoop命令查询postgre表可以,但是执行sqoop import命令倒数时报错:

      • The authentication type 12 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it
      • The authentication type 5 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it
    • 问题根因:
      • 报错中type为5时:在执行sqoop import命令时,会启动MapReduce任务,由于MRS Hadoop安装目录(/opt/Bigdata/ FusionInsight _HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib)下自带了postgre驱动包gsjdbc4-*.jar,与开源postgre服务不兼容导致报错。
      • 报错中type为12时:数据库的pg_hba.conf文件配置有误。
    • 解决方案:
      • 报错中type为5时:在每台MRS NodeManager实例所在节点上移动驱动包gsjdbc4-*.jar到tmp目录下。

        mv /opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib/gsjdbc4-*.jar /tmp

      • 报错中type为12时:调整数据库的pg_hba.conf文件,将address改成sqoop所在节点的ip。

  • 场景二:(export场景)使用sqoop export命令抽取开源postgre到MRS hdfs或hive等。
    • 问题现象:

      使用sqoop命令查询postgre表可以,但是执行sqoop export命令倒数时报错:The authentication type 5 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it

    • 问题根因:

      在执行sqoop export命令时,会启动MapReduce任务,由于MRS Hadoop安装目录(/opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib)下自带了postgre驱动包gsjdbc4-*.jar,与开源postgre服务不兼容导致报错。

    • 解决方案:
      1. 在每台MRS NodeManager实例所在节点上移动驱动包gsjdbc4-*.jar到tmp目录下。

        mv /opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib/gsjdbc4-*.jar /tmp

      1. 将/opt/Bigdata/client/Hive/Beeline/lib/gsjdbc4-*.jar删除 。
support.huaweicloud.com/cmpntguide-mrs/mrs_01_24460.html