Hadoop >

info.png過去の設定情報の履歴は、Hadoopデフォルト設定の履歴をご参照ください。

目次

User limits

  • Hadoopの各デーモンは、ライブラリ参照のため多くのファイルをオープンしたり、多くのプロセスを起動させます。そのため、システムによるそれらのリソース制限をあらかじめ緩和しておく必要があります。
  • 通常、limitsファイルにて緩和設定を行いますが、この設定は PAM(Pluggable Authentication Modules)認証を経た場合にのみ有効となりますので、各デーモンの起動時にそれらが反映されるかどうかをあらかじめ確認しておくとよいでしょう。Hadoopの起動スクリプトでは、su コマンドを用いていますので、以下の要領で反映状況を確認できます。
    $ sudo su -s /bin/bash alice -c 'ulimit -a'
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 9858
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1024
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
    
    $ sudo su -s /bin/bash hdfs -c 'ulimit -a'
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 9858
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 32768
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 65536
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
  • 参考までに、各ディストリビューションの設定内容を以下の通りです。

HDP1.3、HDP1.2

$ cat /etc/security/limits.d/hdfs.conf
...
hdfs   - nofile 32768
hdfs   - nproc  65536
$ cat /etc/security/limits.d/mapred.conf
...
mapred - nofile 32768
mapred - nproc  65536

CDH5、CDH4

$ cat /etc/security/limits.d/hdfs.conf
...
hdfs - nofile 32768
hdfs - nproc  65536
$ cat /etc/security/limits.d/yarn.conf
...
yarn   - nofile 32768
yarn   - nproc  65536
$ cat /etc/security/limits.d/mapreduce.conf
...
mapred    - nofile 32768
mapred    - nproc  65536

HDP2.0

$ cat /etc/security/limits.d/hdfs.conf 
...
hdfs   - nofile 32768
hdfs   - nproc  65536
$ cat /etc/security/limits.d/yarn.conf 
...
yarn   - nofile 32768
yarn   - nproc  65536
$ cat /etc/security/limits.d/mapreduce.conf 
...
mapred    - nofile 32768
mapred    - nproc  65536

Hadoop本体

1.x系

  1. 各Hadoopディストリビューションのドキュメントには、デフォルト設定を示す *-default.xml が同梱されていますが、内容が必ずしも正しくない場合があります。以下では動作する最低限の設定(fs.default.name、mapred.job.trackerのみ設定の疑似分散)で構築したクラスタ上でサンプルジョブを実行し、そのジョブ設定一覧をスナップショットして各ディストリビューションのデフォルト設定を調査しています。クラスタ構築の要領は以下の通りです。
    $ cd $HADOOP_PREFIX
    $ sudo mkdir logs
    $ sudo chown hadoop:hadoop logs/
    $ sudo chmod 775 logs/
    $ sudo -u hdfs ./bin/hadoop namenode -format
    $ sudo -u hdfs ./bin/hadoop-daemon.sh start namenode
    $ sudo -u hdfs ./bin/hadoop-daemon.sh start datanode
    $ sudo -u hdfs ./bin/hadoop fs -mkdir /tmp
    $ sudo -u hdfs ./bin/hadoop fs -chmod 777 /tmp
    $ sudo -u mapred ./bin/hadoop-daemon.sh start jobtracker
    $ sudo -u mapred ./bin/hadoop-daemon.sh start tasktracker
    $ sudo -u hdfs ./bin/hadoop fs -mkdir /user/alice
    $ sudo -u hdfs ./bin/hadoop fs -chown alice:alice /user/alice
    $ sudo -u alice ./bin/hadoop jar hadoop-examples-*.jar pi 5 10
  2. 結果の設定一覧の各TSVファイル(default.tsv)は以下の通りです。
    1. Apache Hadoop 1.1.x
    2. Apache Hadoop 1.0.x
    3. CDH3
    4. HDP1.2

Apache Hadoop 1.0 と 1.1 の差異

  • 1.0.4 と 1.1.2 の差異
    1. $ diff -U 0 localhost-1.0/default.tsv localhost-1.1/default.tsv
    2. --- localhost-1.0/default.tsv 2013-05-17 19:07:13.324619781 +0900
    3. +++ localhost-1.1/default.tsv 2013-05-17 19:17:46.687661551 +0900
    4. @@ -10,0 +11 @@
    5. +dfs.client.use.datanode.hostname false
    6. @@ -21,0 +23,2 @@
    7. +dfs.datanode.max.xcievers 4096
    8. +dfs.datanode.use.datanode.hostname false
    9. @@ -33,0 +37 @@
    10. +dfs.namenode.check.stale.datanode false
    11. @@ -39,0 +44,2 @@
    12. +dfs.namenode.invalidate.work.pct.per.iteration 0.32f
    13. +dfs.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal}
    14. @@ -40,0 +47,3 @@
    15. +dfs.namenode.replication.work.multiplier.per.iteration 2
    16. +dfs.namenode.safemode.min.datanodes 0
    17. +dfs.namenode.stale.datanode.interval 30000
    18. @@ -51 +60 @@
    19. -dfs.support.append false
    20. +dfs.secondary.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal}
    21. @@ -74,0 +84,6 @@
    22. +hadoop.http.authentication.kerberos.keytab ${user.home}/hadoop.keytab
    23. +hadoop.http.authentication.kerberos.principal HTTP/localhost@LOCALHOST
    24. +hadoop.http.authentication.signature.secret.file ${user.home}/hadoop-http-auth-signature-secret
    25. +hadoop.http.authentication.simple.anonymous.allowed true
    26. +hadoop.http.authentication.token.validity 36000
    27. +hadoop.http.authentication.type simple
    28. @@ -77,0 +93 @@
    29. +hadoop.relaxed.worker.version.check false
    30. @@ -83,0 +100 @@
    31. +hadoop.security.use-weak-http-crypto false
    32. @@ -122,0 +140 @@
    33. +mapred.disk.healthChecker.interval 60000
    34. @@ -210,0 +229,2 @@
    35. +mapreduce.ifile.readahead true
    36. +mapreduce.ifile.readahead.bytes 4194304
    37. @@ -215 +235,4 @@
    38. -mapreduce.job.counters.limit 120
    39. +mapreduce.job.counters.counter.name.max 64
    40. +mapreduce.job.counters.group.name.max 128
    41. +mapreduce.job.counters.groups.max 50
    42. +mapreduce.job.counters.max 120

Apache Hadoop と CDH3 の差異

  1. $ diff -U 0 hadoop_default_conf-apache1.0.4.tsv hadoop_default_conf-cdh3.tsv
  2. --- hadoop_default_conf-apache1.0.4.tsv 2013-03-26 22:15:20.774527826 +0900
  3. +++ hadoop_default_conf-cdh3.tsv 2013-03-26 19:53:18.120266266 +0900
  4. @@ -10,0 +11 @@
  5. +dfs.client.use.datanode.hostname false
  6. @@ -13 +14,2 @@
  7. -dfs.datanode.data.dir.perm 755
  8. +dfs.datanode.data.dir.perm 700
  9. +dfs.datanode.directoryscan.threads 1
  10. @@ -15,0 +18,2 @@
  11. +dfs.datanode.drop.cache.behind.reads false
  12. +dfs.datanode.drop.cache.behind.writes false
  13. @@ -21,0 +26,3 @@
  14. +dfs.datanode.readahead.bytes 4193404
  15. +dfs.datanode.sync.behind.writes false
  16. +dfs.datanode.use.datanode.hostname false
  17. @@ -30,0 +38 @@
  18. +dfs.image.transfer.bandwidthPerSec 0
  19. @@ -39,0 +48,2 @@
  20. +dfs.namenode.invalidate.work.pct.per.iteration 0.32
  21. +dfs.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal}
  22. @@ -40,0 +51,2 @@
  23. +dfs.namenode.name.dir.restore false
  24. +dfs.namenode.replication.work.multiplier.per.iteration 2
  25. @@ -48,0 +61 @@
  26. +dfs.safemode.min.datanodes 0
  27. @@ -51 +64 @@
  28. -dfs.support.append false
  29. +dfs.secondary.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal}
  30. @@ -52,0 +66,2 @@
  31. +dfs.webhdfs.enabled false
  32. +fs.automatic.close true
  33. @@ -71,0 +87 @@
  34. +fs.s3n.block.size 67108864
  35. @@ -74,0 +91,10 @@
  36. +group.name alice
  37. +hadoop.fuse.connection.timeout 300
  38. +hadoop.fuse.timer.period 5
  39. +hadoop.http.authentication.kerberos.keytab ${user.home}/hadoop.keytab
  40. +hadoop.http.authentication.kerberos.principal HTTP/_HOST@LOCALHOST
  41. +hadoop.http.authentication.signature.secret.file ${user.home}/hadoop-http-auth-signature-secret
  42. +hadoop.http.authentication.simple.anonymous.allowed true
  43. +hadoop.http.authentication.token.validity 36000
  44. +hadoop.http.authentication.type simple
  45. +hadoop.kerberos.kinit.command kinit
  46. @@ -77,0 +104 @@
  47. +hadoop.relaxed.worker.version.check true
  48. @@ -81,0 +109 @@
  49. +hadoop.security.instrumentation.requires.admin false
  50. @@ -83,0 +112 @@
  51. +hadoop.security.use-weak-http-crypto true
  52. @@ -85,0 +115 @@
  53. +hadoop.workaround.non.threadsafe.getpwuid false
  54. @@ -87 +117 @@
  55. -io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,
  56. org.apache.hadoop.io.compress.SnappyCodec
  57. +io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,
  58. org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec
  59. @@ -109,0 +140 @@
  60. +jobclient.completion.poll.interval 5000
  61. @@ -110,0 +142 @@
  62. +jobclient.progress.monitor.poll.interval 1000
  63. @@ -121 +152,0 @@
  64. -mapred.combine.recordsBeforeProgress 10000
  65. @@ -122,0 +154 @@
  66. +mapred.disk.healthChecker.interval 60000
  67. @@ -146,2 +177,0 @@
  68. -mapred.jobtracker.blacklist.fault-bucket-width 15
  69. -mapred.jobtracker.blacklist.fault-timeout-window 180
  70. @@ -148,0 +179 @@
  71. +mapred.jobtracker.instrumentation org.apache.hadoop.mapred.JobTrackerMetricsInst
  72. @@ -156,0 +188 @@
  73. +mapred.map.child.log.level INFO
  74. @@ -175,0 +208 @@
  75. +mapred.reduce.child.log.level INFO
  76. @@ -200,0 +234 @@
  77. +mapred.tasktracker.instrumentation org.apache.hadoop.mapred.TaskTrackerMetricsInst
  78. @@ -215,3 +249,5 @@
  79. -mapreduce.job.counters.limit 120
  80. -mapreduce.job.split.metainfo.maxsize 10000000
  81. +mapreduce.job.counters.counter.name.max 64
  82. +mapreduce.job.counters.group.name.max 128
  83. +mapreduce.job.counters.groups.max 50
  84. +mapreduce.job.counters.max 120
  85. @@ -219,0 +256 @@
  86. +mapreduce.jobtracker.split.metainfo.maxsize 10000000
  87. @@ -224,0 +262 @@
  88. +mapreduce.tasktracker.cache.local.numberdirectories 10000
  89. @@ -226 +263,0 @@
  90. -mapreduce.tasktracker.outofband.heartbeat.damper 1000000

Apache Hadoop と HDP1.2 の差異

  1. $ diff -U 0 hadoop_default_conf-apache1.0.4.tsv hadoop_default_conf-hdp1.2.tsv
  2. --- hadoop_default_conf-apache1.0.4.tsv 2013-03-26 22:15:20.774527826 +0900
  3. +++ hadoop_default_conf-hdp1.2.tsv 2013-03-26 19:53:51.764433103 +0900
  4. @@ -10,0 +11 @@
  5. +dfs.client.use.datanode.hostname false
  6. @@ -21,0 +23,2 @@
  7. +dfs.datanode.max.xcievers 4096
  8. +dfs.datanode.use.datanode.hostname false
  9. @@ -33,0 +37 @@
  10. +dfs.namenode.check.stale.datanode false
  11. @@ -39,0 +44,2 @@
  12. +dfs.namenode.invalidate.work.pct.per.iteration 0.32f
  13. +dfs.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal}
  14. @@ -40,0 +47,3 @@
  15. +dfs.namenode.replication.work.multiplier.per.iteration 2
  16. +dfs.namenode.safemode.min.datanodes 0
  17. +dfs.namenode.stale.datanode.interval 30000
  18. @@ -51 +60 @@
  19. -dfs.support.append false
  20. +dfs.secondary.namenode.kerberos.internal.spnego.principal ${dfs.web.authentication.kerberos.principal}
  21. @@ -74,0 +84,6 @@
  22. +hadoop.http.authentication.kerberos.keytab ${user.home}/hadoop.keytab
  23. +hadoop.http.authentication.kerberos.principal HTTP/localhost@LOCALHOST
  24. +hadoop.http.authentication.signature.secret.file ${user.home}/hadoop-http-auth-signature-secret
  25. +hadoop.http.authentication.simple.anonymous.allowed true
  26. +hadoop.http.authentication.token.validity 36000
  27. +hadoop.http.authentication.type simple
  28. @@ -77,0 +93 @@
  29. +hadoop.relaxed.worker.version.check false
  30. @@ -81,0 +98 @@
  31. +hadoop.security.instrumentation.requires.admin false
  32. @@ -83,0 +101 @@
  33. +hadoop.security.use-weak-http-crypto false
  34. @@ -122,0 +141 @@
  35. +mapred.disk.healthChecker.interval 60000
  36. @@ -210,0 +230,2 @@
  37. +mapreduce.ifile.readahead true
  38. +mapreduce.ifile.readahead.bytes 4194304
  39. @@ -215,2 +236,5 @@
  40. -mapreduce.job.counters.limit 120
  41. +mapreduce.job.counters.counter.name.max 64
  42. +mapreduce.job.counters.group.name.max 128
  43. +mapreduce.job.counters.groups.max 50
  44. +mapreduce.job.counters.max 120

2.x系

  1. Hadoop2.x系では設定ファイルの管理方法が整理されたため、以下の動作に必要な最小限の設定を行った上でWeb UI(http://localhost:8088/conf)経由でデフォルト設定を調査しています。
    • プロパティ
      • fs.defaultFS: hdfs://localhost:9000
      • mapreduce.framework.name: yarn
    • capacity-scheduler.xml の追加(ディストリビューションに含まれない場合)
    • クラスタ構築の要領は以下のとおり。
      $ cd $HADOOP_PREFIX
      $ sudo mkdir logs
      $ sudo chown hadoop:hadoop logs/
      $ sudo chmod 775 logs/
      $ sudo -u hdfs ./bin/hadoop namenode -format
      $ sudo -u hdfs ./sbin/hadoop-daemon.sh start namenode
      $ sudo -u hdfs ./sbin/hadoop-daemon.sh start datanode
      $ sudo -u hdfs ./bin/hadoop fs -mkdir /tmp
      $ sudo -u hdfs ./bin/hadoop fs -chmod 1777 /tmp
      $ sudo -u yarn ./sbin/yarn-daemon.sh start resourcemanager
      $ sudo -u yarn ./sbin/yarn-daemon.sh start nodemanager
      $ sudo -u mapred ./sbin/mr-jobhistory-daemon.sh start historyserver
      $ sudo -u hdfs ./bin/hadoop fs -mkdir -p /user/alice
      $ sudo -u hdfs ./bin/hadoop fs -chown alice:alice /user/alice
  2. 結果の設定一覧の各XMLファイル(default.xml)とプロパティでソート済みのTSVファイル(default.tsv)は以下の通りです。
    1. Apache Hadoop 2.2.x
    2. Apache Hadoop 2.0.x
    3. HDP2.0
    4. CDH5

Apache Hadoop 2.0.4a と 2.2.0GA の差異

  1. $ diff -U 0 localhost-2.0/default.tsv localhost-2.2/default.tsv
  2. --- localhost-2.0/default.tsv   2013-05-22 18:21:16.000000000 +0900
  3. +++ localhost-2.2/default.tsv   2013-11-13 19:56:44.648820443 +0900
  4. @@ -11,0 +12 @@
  5. +fs.client.resolve.remote.symlinks  true
  6. @@ -55,0 +57 @@
  7. +hadoop.security.group.mapping.ldap.directory.search.timeout    10000
  8. @@ -90,0 +93 @@
  9. +ipc.client.fallback-to-simple-auth-allowed false
  10. @@ -96,5 +98,0 @@
  11. -kfs.blocksize  67108864
  12. -kfs.bytes-per-checksum 512
  13. -kfs.client-write-packet-size   65536
  14. -kfs.replication    3
  15. -kfs.stream-buffer-size 4096
  16. @@ -104 +102 @@
  17. -mapreduce.application.classpath    $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
  18. +mapreduce.am.max-attempts  2
  19. @@ -139,0 +138 @@
  20. +mapreduce.job.token.tracking.ids.enabled   false
  21. @@ -144,0 +144,8 @@
  22. +mapreduce.jobhistory.cleaner.enable    true
  23. +mapreduce.jobhistory.cleaner.interval-ms   86400000
  24. +mapreduce.jobhistory.client.thread-count   10
  25. +mapreduce.jobhistory.datestring.cache.size 200000
  26. +mapreduce.jobhistory.done-dir  ${yarn.app.mapreduce.am.staging-dir}/history/done
  27. +mapreduce.jobhistory.http.policy   HTTP_ONLY
  28. +mapreduce.jobhistory.intermediate-done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate
  29. +mapreduce.jobhistory.joblist.cache.size    20000
  30. @@ -145,0 +153,5 @@
  31. +mapreduce.jobhistory.loadedjobs.cache.size 5
  32. +mapreduce.jobhistory.max-age-ms    604800000
  33. +mapreduce.jobhistory.minicluster.fixed.ports   false
  34. +mapreduce.jobhistory.move.interval-ms  180000
  35. +mapreduce.jobhistory.move.thread-count 3
  36. @@ -198 +210 @@
  37. -mapreduce.shuffle.port 8080
  38. +mapreduce.shuffle.port 13562
  39. @@ -228,0 +241 @@
  40. +net.topology.impl  org.apache.hadoop.net.NetworkTopology
  41. @@ -230,0 +244,6 @@
  42. +nfs3.mountd.port   4242
  43. +nfs3.server.port   2049
  44. +rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB    org.apache.hadoop.ipc.ProtobufRpcEngine
  45. +rpc.engine.org.apache.hadoop.yarn.api.ApplicationClientProtocolPB  org.apache.hadoop.ipc.ProtobufRpcEngine
  46. +rpc.engine.org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB   org.apache.hadoop.ipc.ProtobufRpcEngine
  47. +rpc.engine.org.apache.hadoop.yarn.server.api.ResourceTrackerPB org.apache.hadoop.ipc.ProtobufRpcEngine
  48. @@ -255 +274 @@
  49. -yarn.app.mapreduce.client-am.ipc.max-retries   1
  50. +yarn.app.mapreduce.client-am.ipc.max-retries   3
  51. @@ -257,0 +277,5 @@
  52. +yarn.client.app-submission.poll-interval   1000
  53. +yarn.client.max-nodemanagers-proxies   500
  54. +yarn.client.nodemanager-client-async.thread-pool-max-size  500
  55. +yarn.dispatcher.exit-on-error  true
  56. +yarn.http.policy   HTTP_ONLY
  57. @@ -264 +288 @@
  58. -yarn.nodemanager.address   0.0.0.0:0
  59. +yarn.nodemanager.address   ${yarn.nodemanager.hostname}:0
  60. @@ -266 +290 @@
  61. -yarn.nodemanager.aux-services.mapreduce.shuffle.class  org.apache.hadoop.mapred.ShuffleHandler
  62. +yarn.nodemanager.aux-services.mapreduce_shuffle.class  org.apache.hadoop.mapred.ShuffleHandler
  63. @@ -277 +301 @@
  64. -yarn.nodemanager.heartbeat.interval-ms 1000
  65. +yarn.nodemanager.hostname  0.0.0.0
  66. @@ -281,0 +306 @@
  67. +yarn.nodemanager.local-cache.max-files-per-directory   8192
  68. @@ -283 +308 @@
  69. -yarn.nodemanager.localizer.address 0.0.0.0:8040
  70. +yarn.nodemanager.localizer.address ${yarn.nodemanager.hostname}:8040
  71. @@ -295 +320 @@
  72. -yarn.nodemanager.resource.cpu-cores    8
  73. +yarn.nodemanager.resource.cpu-vcores   8
  74. @@ -296,0 +322,2 @@
  75. +yarn.nodemanager.resourcemanager.connect.retry_interval.secs   30
  76. +yarn.nodemanager.resourcemanager.connect.wait.secs 900
  77. @@ -298 +324,0 @@
  78. -yarn.nodemanager.vcores-pcores-ratio   2
  79. @@ -301,3 +327,3 @@
  80. -yarn.nodemanager.webapp.address    0.0.0.0:8042
  81. -yarn.resourcemanager.address   0.0.0.0:8032
  82. -yarn.resourcemanager.admin.address 0.0.0.0:8033
  83. +yarn.nodemanager.webapp.address    ${yarn.nodemanager.hostname}:8042
  84. +yarn.resourcemanager.address   localhost:8032
  85. +yarn.resourcemanager.admin.address localhost:8033
  86. @@ -305 +331 @@
  87. -yarn.resourcemanager.am.max-retries    1
  88. +yarn.resourcemanager.am.max-attempts   2
  89. @@ -308,0 +335,2 @@
  90. +yarn.resourcemanager.connect.max-wait.ms   900000
  91. +yarn.resourcemanager.connect.retry-interval.ms 30000
  92. @@ -312 +340,2 @@
  93. -yarn.resourcemanager.fs.rm-state-store.uri ${hadoop.tmp.dir}/yarn/system/rmstore
  94. +yarn.resourcemanager.fs.state-store.uri    ${hadoop.tmp.dir}/yarn/system/rmstore
  95. +yarn.resourcemanager.hostname  0.0.0.0
  96. @@ -315,0 +345 @@
  97. +yarn.resourcemanager.nodemanagers.heartbeat-interval-ms    1000
  98. @@ -317 +347 @@
  99. -yarn.resourcemanager.resource-tracker.address  0.0.0.0:8031
  100. +yarn.resourcemanager.resource-tracker.address  localhost:8031
  101. @@ -319 +349 @@
  102. -yarn.resourcemanager.scheduler.address 0.0.0.0:8030
  103. +yarn.resourcemanager.scheduler.address localhost:8030
  104. @@ -321,0 +352,2 @@
  105. +yarn.resourcemanager.scheduler.monitor.enable  false
  106. +yarn.resourcemanager.scheduler.monitor.policies    org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
  107. @@ -323 +355,2 @@
  108. -yarn.resourcemanager.webapp.address    0.0.0.0:8088
  109. +yarn.resourcemanager.webapp.address    ${yarn.resourcemanager.hostname}:8088
  110. +yarn.resourcemanager.webapp.https.address  ${yarn.resourcemanager.hostname}:8090

Apache Hadoop 2.2.0GA と HDP2.0.6.0GA の差異

  • デフォルト設定の差異はありません。

Apache Hadoop 2.2.0GA と CDH5.0.0 の差異

  1. $ diff -U 0 localhost-2.2/default.tsv localhost-cdh5/default.tsv
  2. --- localhost-2.2/default.tsv   2013-11-13 19:56:44.648820443 +0900
  3. +++ localhost-cdh5/default.tsv  2013-11-13 20:56:39.702406792 +0900
  4. @@ -276 +276 @@
  5. -yarn.application.classpath $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
  6. $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
  7. +yarn.application.classpath $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
  8. $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
  9. @@ -284 +284 @@
  10. -yarn.log-aggregation-enable    false
  11. +yarn.log-aggregation-enable    true
  12. @@ -289,0 +290 @@
  13. +yarn.nodemanager.aux-services  mapreduce_shuffle
  14. @@ -304,0 +306,2 @@
  15. +yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user    nobody
  16. +yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern  ^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$
  17. @@ -307 +310 @@
  18. -yarn.nodemanager.local-dirs    ${hadoop.tmp.dir}/nm-local-dir
  19. +yarn.nodemanager.local-dirs    /var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir
  20. @@ -314 +317 @@
  21. -yarn.nodemanager.log-dirs  ${yarn.log.dir}/userlogs
  22. +yarn.nodemanager.log-dirs  /var/log/hadoop-yarn/containers
  23. @@ -318 +321 @@
  24. -yarn.nodemanager.remote-app-log-dir    /tmp/logs
  25. +yarn.nodemanager.remote-app-log-dir    /var/log/hadoop-yarn/apps
  26. @@ -334,0 +338 @@
  27. +yarn.resourcemanager.cluster.id    yarn-rm-cluster
  28. @@ -340,0 +345,7 @@
  29. +yarn.resourcemanager.ha.admin.address  ${yarn.resourcemanager.hostname}:8034
  30. +yarn.resourcemanager.ha.admin.client.thread-count  1
  31. +yarn.resourcemanager.ha.automatic-failover.controller.class    org.apache.hadoop.yarn.server.resourcemanager.RMFailoverControllerZKImpl
  32. +yarn.resourcemanager.ha.automatic-failover.enabled false
  33. +yarn.resourcemanager.ha.automatic-failover.port    8035
  34. +yarn.resourcemanager.ha.enabled    false
  35. +yarn.resourcemanager.ha.fencer org.apache.hadoop.yarn.server.resourcemanager.ZKStoreNodeFencer
  36. @@ -350 +361 @@
  37. -yarn.resourcemanager.scheduler.class   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
  38. +yarn.resourcemanager.scheduler.class   org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
  39. @@ -356,0 +368,4 @@
  40. +yarn.resourcemanager.zk.state-store.acl    world:anyone:rwcda
  41. +yarn.resourcemanager.zk.state-store.num-retries    3
  42. +yarn.resourcemanager.zk.state-store.parent-path    /rmstore
  43. +yarn.resourcemanager.zk.state-store.timeout.ms 60000

トップ   編集 凍結 差分 バックアップ 添付 複製 名前変更 リロード   新規 一覧 単語検索 最終更新   ヘルプ   最終更新のRSS
Last-modified: 2013-11-19 (火) 18:25:34 (2932d)