NAME
Apache::Hadoop::Config - Perl extension for Hadoop node configuration
SYNOPSIS
use Apache::Hadoop::Config;
Hadoop configuration setupConfiguration of Apache Hadoop is easy to build a cluster with default settings. But those settings are not suitable for a wide variety of hardware configuration. This perl package proposes optimal properties for some of the configuration parameters based on hardware configuration and user requirement.
It is primarily designed to extract hardware configuration from /proc file system to understand cpu cores, system memory and disk layout information. But these parameters can be manually fed as arguments to generate recommended settings.
This perl package can create namenode and datanode repositories, set appropriate permissions and generate configuration XML files with recommended settings.
DESCRIPTION
Perl extension Apache::Hadoop::Config is designed to address Hadoop deployment and configuration practices, enabling rapid provisioning of Hadoop cluster with customization. It has two distinct capabilities (1) to generate configuration files, (2) create namenode and datanode repositories.
This package need to be installed ideally on at least one of the nodes in the cluster, assuming that all nodes have identical hardware configuration. However, this package can be installed on any other node and required hardware information can be supplied using arguments and configuration files can be generated and copied to actual cluster nodes.
This package is capable of creating repositories for namenode and datanodes, for which it should be installed on ALL hadoop cluster nodes.
Create a new Apache::Hadoop::Config object, either using system configuration or by supplying from command line arguments.
my $h = Apache::Hadoop::Config->new; Basic configuration and memory settings are available using two functions. Calling basic configuration function is required while memory configuration is recommended.
$h->basic_config;
$h->memory_config;The package can print or create XML configuration files independently, using print and write functions, for configuration. It is necessary to provide conf directory, writable, to write configuration XML files.
$h->print_config;
$h->write_config (confdir=>'etc/hadoop');Additional configuration parameters can be supplied at the time of creating the object.
my $h = Apache::Hadoop::Config->new (
config=> {
'mapred-site.xml' => {
'mapreduce.task.io.sort.mb' => 256,
},
'core-site.xml' => {
'hadoop.tmp.dir' => '/tmp/hadoop',
},
},
);These parameters will override any automatically generated parameters, built into this package.
The package creates namenode and datanode volumes along with setting permission of hadoop.tmp.dir and log directories. The disk information can be supplied at object construction time.
my $h = Apache::Hadoop::Config->new (
hdfs_name_disks => [ '/hdfs/namedisk1', '/hdfs/namedisk2' ],
hdfs_data_disks => [ '/hdfs/datadisk1', '/hdfs/datadisk2' ],
hdfs_tmp => '/hdfs/tmp',
hdfs_logdir => [ '/logs', '/logs/userlog' ],
);Note that name disks and data disks accept reference to array type of data. The package creates all the namenode and datanode volumes and creates log and tmp directories.
$h->create_hdfs_name_disks;
$h->create_hdfs_data_disks;
$h->create_hdfs_tmpdir;
$h->create_hadoop_logdir;The permission will be set as appropriate. It is strongly recommended that this package and associated script is executed by Hadoop Admin user (hduser).
Some of the basic configuration can be customized externally using object arguments. Namenode, secondary namenode, proxy node informations can be customized. Default is localhost for each of them.
my $h = Apache::Hadoop::Config->new (
namenode => 'nn.myorg.com',
secondary=> 'nn2.myorg.com',
proxynode=> 'pr.myorg.com',
proxyport=> '8888', # default, optional
);These are optional and required only when secondary namenode and proxy node are different than primary namenode.
EXAMPLES
Below are a few examples of different uses. The first example is to create recommended configurations for the localhost or command-line provided data:
#!/usr/bin/perl -w
use strict;
use warnings;
use Apache::Hadoop::Config;
use Getopt::Long;
my %opts;
GetOptions (\%opts, 'disks=s','memory=s','cores=s');
my $h = Apache::Hadoop::Config->new (
meminfo=>$opts{'memory'} || undef,
cpuinfo=>$opts{'cores'} || undef,
diskinfo=>$opts{'disks'} || undef,
);
# setup configs
$h->basic_config;
$h->memory_config;
# print and save
$h->print_config;
$h->write_config (confdir=>'.');
exit(0);The above gives an output like below, if no argument is supplied:
min cont size (mb) : 256
num of containers : 7
mem per container (mb): 368
disk : 4
cpu : 4
mem : 3.52075958251953
---------------
hdfs-site.xml
dfs.namenode.secondary.http-address: 0.0.0.0:50090
dfs.replication: 1
dfs.datanode.data.dir: file:///hdfs/data1,file:///hdfs/data2,file:///hdfs/data3,file:///hdfs/data4
dfs.namenode.secondary.https-address: 0.0.0.0:50091
dfs.namenode.name.dir: file:///hdfs/name1,file:///hdfs/name2
yarn-site.xml
yarn.web-proxy.address: localhost:8888
yarn.nodemanager.aux-services: mapreduce_shuffle
yarn.scheduler.minimum-allocation-mb: 368
yarn.scheduler.maximum-allocation-mb: 2576
yarn.nodemanager.aux-services.mapreduce.shuffle.class: org.apache.hadoop.mapred.ShuffleHandler
yarn.nodemanager.resource.memory-mb: 2576
core-site.xml
hadoop.tmp.dir: /hdfs/tmp
fs.defaultFS: http://localhost:9000
mapred-site.xml
mapreduce.reduce.java.opts: -Xmx588m
mapreduce.map.memory.mb: 368
mapreduce.map.java.opts: -Xmx294m
mapreduce.framework.name: yarn
mapreduce.reduce.memory.mb: 736
---------------
-> writing to ./hdfs-site.xml ...
-> writing to ./yarn-site.xml ...
-> writing to ./core-site.xml ...
-> writing to ./mapred-site.xml ...If supplied with some arguments, basically for a different clusters, the configuration files can still be generated:
$ perl hadoop_config.pl --cores 16 --memory 64 --disks 6
min cont size (mb) : 2048
num of containers : 10
mem per container (mb): 5734
disk : 6
cpu : 16
mem : 64
---------------
hdfs-site.xml
dfs.namenode.secondary.http-address: 0.0.0.0:50090
dfs.replication: 1
dfs.datanode.data.dir: file:///hdfs/data1,file:///hdfs/data2,file:///hdfs/data3,file:///hdfs/data4
dfs.namenode.secondary.https-address: 0.0.0.0:50091
dfs.namenode.name.dir: file:///hdfs/name1,file:///hdfs/name2
yarn-site.xml
yarn.web-proxy.address: localhost:8888
yarn.nodemanager.aux-services: mapreduce_shuffle
yarn.scheduler.minimum-allocation-mb: 5734
yarn.scheduler.maximum-allocation-mb: 57340
yarn.nodemanager.aux-services.mapreduce.shuffle.class: org.apache.hadoop.mapred.ShuffleHandler
yarn.nodemanager.resource.memory-mb: 57340
core-site.xml
hadoop.tmp.dir: /hdfs/tmp
fs.defaultFS: http://localhost:9000
mapred-site.xml
mapreduce.reduce.java.opts: -Xmx9174m
mapreduce.map.memory.mb: 5734
mapreduce.map.java.opts: -Xmx4587m
mapreduce.framework.name: yarn
mapreduce.reduce.memory.mb: 11468
---------------
-> writing to ./hdfs-site.xml ...
-> writing to ./yarn-site.xml ...
-> writing to ./core-site.xml ...
-> writing to ./mapred-site.xml ...Different customization can be done, using object's constructor arguments.
SEE ALSO
hadoop.apache.org - The Hadoop documentation and authoritative source for Apache Hadoop and its components.
AUTHOR
Snehasis Sinha, <snehasis@cpan.org>
COPYRIGHT AND LICENSE
Copyright (C) 2015 by Snehasis Sinha
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.10.1 or, at your option, any later version of Perl 5 you may have available.