1使用debuger调试提交时出现:
这个是因为idea会自动处理toString,需要设置一下,stackOver里面我有回答。
2摘录部分日志:
java.io.EOFException: End of File Exception between local host is: "node01/10.211.55.4"; destination host is: "node01":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException
警告: The following warnings have been detected with resource and/or provider classes:
WARNING: A sub-resource method, public javax.ws.rs.core.Response org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.putRoot(org.apache.hadoop.security.UserGroupInformation,org.apache.hadoop.hdfs.web.resources.DelegationParam,org.apache.hadoop.hdfs.web.resources.UserParam,org.apache.hadoop.hdfs.web.resources.DoAsParam,org.apache.hadoop.hdfs.web.resources.PutOpParam,org.apache.hadoop.hdfs.web.resources.DestinationParam,org.apache.hadoop.hdfs.web.resources.OwnerParam,org.apache.hadoop.hdfs.web.resources.GroupParam,org.apache.hadoop.hdfs.web.resources.PermissionParam,org.apache.hadoop.hdfs.web.resources.OverwriteParam,org.apache.hadoop.hdfs.web.resources.BufferSizeParam,org.apache.hadoop.hdfs.web.resources.ReplicationParam,org.apache.hadoop.hdfs.web.resources.BlockSizeParam,org.apache.hadoop.hdfs.web.resources.ModificationTimeParam,org.apache.hadoop.hdfs.web.resources.AccessTimeParam,org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam,org.apache.hadoop.hdfs.web.resources.CreateParentParam,org.apache.hadoop.hdfs.web.resources.TokenArgumentParam,org.apache.hadoop.hdfs.web.resources.AclPermissionParam,org.apache.hadoop.hdfs.web.resources.XAttrNameParam,org.apache.hadoop.hdfs.web.resources.XAttrValueParam,org.apache.hadoop.hdfs.web.resources.XAttrSetFlagParam,org.apache.hadoop.hdfs.web.resources.SnapshotNameParam,org.apache.hadoop.hdfs.web.resources.OldSnapshotNameParam,org.apache.hadoop.hdfs.web.resources.ExcludeDatanodesParam) throws java.io.IOException,java.lang.InterruptedException, with URI template, "/", is treated as a resource method
第一个异常是因为idea中配置文件因为远程调用设置的hdfs的节点名用的是ip,没用实际设置的机器名,这个在hadoop中是有限制的
第二个原因是idea中resource中的mapred-site.xml配置文件的
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>local</value>
</property>
</configuration>
这个配置有问题,如果是单机单节点,就要配置成local,如果环境搭建的配置的是yarn,但是实际仍是单节点运行yarn的话,仍要配置成local,不然也会出现类似问题。
集群环境下,yarn任务会有resouncemanager来分配资源,但是单机单节点就不会走resourcemanger