当前位置: 首页 > hadoop, 分布式系统 > 正文

cygwin hadoop windows7异常(一)

1 星2 星3 星4 星5 星 (1 次投票, 评分: 5.00, 总分: 5)
Loading ... Loading ...
baidu_share
文章目录

在windows7下利用cygwin搭建hadoop 1.2.1集群,出现各种问题。

一.不能load native-hadoop库在cygwin下

在测试wordcount例子是出现以下警告:

13/09/28 12:11:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/28 12:11:46 WARN snappy.LoadSnappy: Snappy native library not loaded

意思是在cygwin下,不能调用native-hadoop库。网上说要编译ant compile-native。
可是又出现以下异常:

D:\hq\sdk\hadoop-1.2.1\build.xml:627: The following error occurred while executing this line:
D:\hq\sdk\hadoop-1.2.1\build.xml:704: Execute failed: java.io.IOException: Cannot run program "make" (in directory "D:\hq\sdk\hadoop-1.2.1\build\native\Windows_7-x86-32"): CreateProcess error=2, ?????????
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
        at java.lang.Runtime.exec(Runtime.java:617)
        at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:862)
        at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:481)
        at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:495)
        at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:631)
        at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
        at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:392)
        at org.apache.tools.ant.Target.performTasks(Target.java:413)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
        at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
        at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
        at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:392)
        at org.apache.tools.ant.Target.performTasks(Target.java:413)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
        at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
        at org.apache.tools.ant.Main.runBuild(Main.java:811)
        at org.apache.tools.ant.Main.startAnt(Main.java:217)
        at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
        at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Caused by: java.io.IOException: CreateProcess error=2, ?????????
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:376)
        at java.lang.ProcessImpl.start(ProcessImpl.java:136)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
        ... 36 more

build.xml配置文件已经移除匹配的6个executable=”autoreconf”的exec配置。

经过各种折腾,例如装 autoconf, automake, libtool,gcc类库,异常依旧。最后看hadoop官方文档:Native Libraries Guide支持的平台有:
1.RHEL4/Fedora
2.Ubuntu
3.Gentoo
不支持在cygwin下进行native compile。

二.hadoop安全访问控制异常

2013-09-27 17:12:29,590 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9000, call getFileInfo(/hadoop/tmp/mapred/system/job_201309271701_0003/jobToken) from 127.0.0.1:59938: error: org.apache.hadoop.security.AccessControlException: Permission denied: user=cyg_server, access=EXECUTE, inode="job_201309271701_0003":Administrator:supergroup:rwx------
org.apache.hadoop.security.AccessControlException: Permission denied: user=cyg_server, access=EXECUTE, inode="job_201309271701_0003":Administrator:supergroup:rwx------
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:217)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:170)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5758)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:5736)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:2458)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getFileInfo(NameNode.java:942)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

在hdfs-site.xml文件加以下配置:

 <property>  
  <name>dfs.permissions</name>  
  <value>false</value>  
</property>

上述异常消失。

本文固定链接: http://www.chepoo.com/cygwin-hadoop-windows7-error-1.html | IT技术精华网

cygwin hadoop windows7异常(一):等您坐沙发呢!

发表评论