Apache Hadoop Development Tools (HDT) is still in development phase. So, no official distribution of Hadoop 2.2.0 Eclipse Plugin is available now. But we can build the same using winghc/hadoop2x-eclipse-plugin. In this post, we'll build, install and configure the plugin with the Eclipse or any Eclipse based IDE (say, Spring Tool Suite) to ease the development activities using Hadoop framework.
The Hadoop Development Tools (HDT) is a set of plugins for the Eclipse IDE for developing against the Hadoop platform.
Currently we are in the process of porting the existing MapReduce tools from the Apache Hadoop project to allow working with multiple versions of Hadoop from within one IDE.
Come get involved as we move towards our first release!
This project is currently a member of the Apache Incubator, so check back for updates, or come join us [email protected].
Tools and Technologies used in this article
1. Download
Download winghc/hadoop2x-eclipse-plugin zip.
2. Extract
Extract to a local directory (say, 'C:\hadoop2x-eclipse-plugin').
3. Build
- Open '
\src\contrib\eclipse-plugin' in the Command prompt.C:\>cd C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin - Run ANT build
eclipse.home: Installation directory of Eclipse IDE.C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin>ant jar -Dversion=2.2.0 -Declipse.home=C:/IDE/sts-3.5.0 -Dhadoop.home=c:/hadoop Buildfile: C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin\build.xml check-contrib: init: [echo] contrib: eclipse-plugin [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\classes [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\test [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\system [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\system\classes [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\examples [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\test\logs init-contrib: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: C:\hadoop2x-eclipse-plugin\ivy\ivy-2.1.0.jar ivy-probe-antlib: ivy-init-antlib: ivy-init: [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = C:\hadoop2x-eclipse-plugin\ivy\ivysettings.xml ivy-resolve-common: ivy-retrieve-common: [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:cachepath] :: loading settings :: file = C:\hadoop2x-eclipse-plugin\ivy\ivysettings.xml compile: [echo] contrib: eclipse-plugin [javac] C:\hadoop2x-eclipse-plugin\src\contrib\eclipse-plugin\build.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 45 source files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\classes [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found [javac] c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.2.0.jar(org/apache/hadoop/hdfs/DistributedFileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 7 warnings jar: [mkdir] Created dir: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying 9 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-app-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-app-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-common-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-core-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-core-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-hs-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-plugins-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-hs-plugins-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.2.0-tests.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-jobclient-2.2.0-tests.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-jobclient-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-shuffle-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-client-shuffle-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-mapreduce-examples-2.2.0.jar [copy] Copying 3 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\hadoop-common-2.2.0-tests.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-common-2.2.0-tests.jar [copy] Copying c:\hadoop\share\hadoop\common\hadoop-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-common-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\common\hadoop-nfs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-nfs-2.2.0.jar [copy] Copying 3 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.2.0-tests.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-hdfs-2.2.0-tests.jar [copy] Copying c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-hdfs-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\hdfs\hadoop-hdfs-nfs-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-hdfs-nfs-2.2.0.jar [copy] Copying 11 files to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-api-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-api-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-distributedshell-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-applications-distributedshell-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-client-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-client-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-common-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-common-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-common-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-nodemanager-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-nodemanager-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-resourcemanager-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-resourcemanager-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-tests-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-tests-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-server-web-proxy-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-server-web-proxy-2.2.0.jar [copy] Copying c:\hadoop\share\hadoop\yarn\hadoop-yarn-site-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-yarn-site-2.2.0.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\protobuf-java-2.5.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\protobuf-java-2.5.0.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\log4j-1.2.17.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\log4j-1.2.17.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\commons-cli-1.2.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\commons-cli-1.2.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\commons-configuration-1.6.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\commons-configuration-1.6.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\commons-lang-2.5.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\commons-lang-2.5.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\jackson-core-asl-1.8.8.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\jackson-core-asl-1.8.8.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\jackson-mapper-asl-1.8.8.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\jackson-mapper-asl-1.8.8.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\slf4j-log4j12-1.7.5.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\slf4j-log4j12-1.7.5.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\slf4j-api-1.7.5.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\slf4j-api-1.7.5.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\guava-11.0.2.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\guava-11.0.2.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\hadoop-auth-2.2.0.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\hadoop-auth-2.2.0.jar [copy] Copying 1 file to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib [copy] Copying c:\hadoop\share\hadoop\common\lib\netty-3.6.2.Final.jar to C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\lib\netty-3.6.2.Final.jar [jar] Building jar: C:\hadoop2x-eclipse-plugin\build\contrib\eclipse-plugin\hadoop-eclipse-plugin-2.2.0.jar BUILD SUCCESSFUL Total time: 42 seconds
hadoop.home: Hadoop installation directory.
Note: Internet connection is required as 'ivy-2.1.0.jar' will be downloaded.
4. Install
On successful build, 'hadoop-eclipse-plugin-2.2.0.jar' will be generated inside '
5. Configure
Restart the Eclipse IDE if already started. Otherwise start it.
- Goto Window --> Open Perspective --> Other and select 'Map/Reduce' perspective.
- Click 'New Hadoop location...' (Blue Elephant icon) and define Hadoop location to run MapReduce applications. Click 'Finish' button.
Map/Reduce(V2) Master: Address of the Map/Reduce master node (The Job Tracker).
DFS Master: Address of the Distributed FileSystem Master node (The Name Node).
To know the 'Port' numbers, start Hadoop and open http://localhost:8088/cluster in a browser. Click Tools --> Configuration and search for the following properties.
DFS Master
Map/Reduce(V2) Master<property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> <source>core-site.xml</source> </property><property> <name>mapreduce.jobtracker.http.address</name> <value>0.0.0.0:50030</value> <source>mapred-default.xml</source> </property>
- Now we can browse the Hadoop file system and perform different files / folder operations using the GUI only.
Also, we can easily create Map/Reduce Project, Mapper, Reducer and MapReduce Driver using the wizard (File --> New --> Other... --> Map/Reduce) and jump into Hadoop programming.
Comments