idea远程执行yarn的job可能出现的问题_20201227201501-qe9rbjd.sy

668 阅读1分钟

1使用debuger调试提交时出现:

链接

这个是因为idea会自动处理toString,需要设置一下,stackOver里面我有回答。

2摘录部分日志:

java.io.EOFException: End of File Exception between local host is: "node01/10.211.55.4"; destination host is: "node01":9000; : java.io.EOFException; For more details see:  http://wiki.apache.org/hadoop/EOFException
警告: The following warnings have been detected with resource and/or provider classes:
  WARNING: A sub-resource method, public javax.ws.rs.core.Response org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.putRoot(org.apache.hadoop.security.UserGroupInformation,org.apache.hadoop.hdfs.web.resources.DelegationParam,org.apache.hadoop.hdfs.web.resources.UserParam,org.apache.hadoop.hdfs.web.resources.DoAsParam,org.apache.hadoop.hdfs.web.resources.PutOpParam,org.apache.hadoop.hdfs.web.resources.DestinationParam,org.apache.hadoop.hdfs.web.resources.OwnerParam,org.apache.hadoop.hdfs.web.resources.GroupParam,org.apache.hadoop.hdfs.web.resources.PermissionParam,org.apache.hadoop.hdfs.web.resources.OverwriteParam,org.apache.hadoop.hdfs.web.resources.BufferSizeParam,org.apache.hadoop.hdfs.web.resources.ReplicationParam,org.apache.hadoop.hdfs.web.resources.BlockSizeParam,org.apache.hadoop.hdfs.web.resources.ModificationTimeParam,org.apache.hadoop.hdfs.web.resources.AccessTimeParam,org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam,org.apache.hadoop.hdfs.web.resources.CreateParentParam,org.apache.hadoop.hdfs.web.resources.TokenArgumentParam,org.apache.hadoop.hdfs.web.resources.AclPermissionParam,org.apache.hadoop.hdfs.web.resources.XAttrNameParam,org.apache.hadoop.hdfs.web.resources.XAttrValueParam,org.apache.hadoop.hdfs.web.resources.XAttrSetFlagParam,org.apache.hadoop.hdfs.web.resources.SnapshotNameParam,org.apache.hadoop.hdfs.web.resources.OldSnapshotNameParam,org.apache.hadoop.hdfs.web.resources.ExcludeDatanodesParam) throws java.io.IOException,java.lang.InterruptedException, with URI template, "/", is treated as a resource method

第一个异常是因为idea中配置文件因为远程调用设置的hdfs的节点名用的是ip,没用实际设置的机器名,这个在hadoop中是有限制的

第二个原因是idea中resource中的mapred-site.xml配置文件的

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>local</value>
    </property>
</configuration>

这个配置有问题,如果是单机单节点,就要配置成local,如果环境搭建的配置的是yarn,但是实际仍是单节点运行yarn的话,仍要配置成local,不然也会出现类似问题。

集群环境下,yarn任务会有resouncemanager来分配资源,但是单机单节点就不会走resourcemanger