hadoop jar 部分通用参数

1. mapreduce.job.queuename=root.monitor

MapReduce执行任务队列名称，不指定时使用默认队列

2. mapreduce.map.memory.mb 和mapreduce.reduce.memory.mb

Scope

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>-1</value>
  <description>The amount of memory to request from the scheduler for each
    map task. If this is not specified or is non-positive, it is inferred from
    mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
    If java-opts are also not specified, we set it to 1024.
  </description>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>-1</value>
  <description>The amount of memory to request from the scheduler for each
    reduce task. If this is not specified or is non-positive, it is inferred
    from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
    If java-opts are also not specified, we set it to 1024.
  </description>
</property>

解释：
每隔map任务从调度器中请求的内存量，如果没有指定该参数，或者改参数值为非正数，则会根据mapreduce.map.memory和mapreduce.job.heap.memory-mb.ratio进行推断。如果java.opts参数未被设置，则该参数会被设置为1024。
Yarn的ResourceManger（简称RM）通过逻辑上的队列分配内存，CPU等资源给application，默认情况下RM允许最大AM申请Container资源为8192MB(“yarn.scheduler.maximum-allocation-mb“)，默认情况下的最小分配资源为1024M(“yarn.scheduler.minimum-allocation-mb“)，AM只能以增量（”yarn.scheduler.minimum-allocation-mb“）和不会超过(“yarn.scheduler.maximum-allocation-mb“)的值去向RM申请资源，AM负责将(“mapreduce.map.memory.mb“)和(“mapreduce.reduce.memory.mb“)的值规整到能被(“yarn.scheduler.minimum-allocation-mb“)整除，RM会拒绝申请内存超过8192MB和不能被1024MB整除的资源请求。
yarn对其默认计算方式max(MIN_CONTAINER_SIZE, (Total Available RAM) / containers))

3. mapreduce.map.java.opts 和 mapreduce.reduce.java.opts

示例 ：

 -Dmapreduce.map.java.opts=-Xmx1000m -Dmapreduce.reduce.java.opts="-Xmx1000m -XX:+UseG1GC"·

mapreduce.map.java.opts的Scope描述

   <name>mapreduce.map.java.opts</name>
  <value></value>
  <description>Java opts only for the child processes that are maps. If set,
  this will be used instead of mapred.child.java.opts. If -Xmx is not set,
  it is inferred from mapreduce.map.memory.mb and
  mapreduce.job.heap.memory-mb.ratio.
  </description>

mapreduce.reduce.java.opts的Scope描述

<name>mapreduce.reduce.java.opts</name>
  <value></value>
  <description>Java opts only for the child processes that are reduces. If set,
  this will be used instead of mapred.child.java.opts. If -Xmx is not set,
  it is inferred from mapreduce.reduce.memory.mb and
  mapreduce.job.heap.memory-mb.ratio.
  </description>

以mapreduce.map.java.opts解释：
Map的子进程Java参数，如果设置了该参数，将会覆盖mapred.child.java.opts参数。如果java.opts参数中不包含"-Xmx"字段，则会根据mapreduce.map.memory和mapreduce.job.heap.memory-mb.ratio进行推断，计算出对应的值。


mapreduce.map.java.opts 和 mapreduce.reduce.java.opts默认值都为分别与mapreduce.map.memory.mb参数和mapreduce.reduce.memory.mb相同，当mapreduce.map.memory.mb参数和mapreduce.reduce.memory.mb不存在是取值为1024m。

4. mapred.child.java.opts

示例

-Dmapred.child.java.opts="-Xmx8192m"

Scope描述

<name>mapred.child.java.opts</name>
  <value></value>
  <description>Java opts for the task processes.
  The following symbol, if present, will be interpolated: @taskid@ is replaced
  by current TaskID. Any other occurrences of '@' will go unchanged.
  For example, to enable verbose gc logging to a file named for the taskid in
  /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
        -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc

  Usage of -Djava.library.path can cause programs to no longer function if
  hadoop native libraries are used. These values should instead be set as part
  of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
  mapreduce.reduce.env config settings.

  If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and
  mapreduce.job.heap.memory-mb.ratio.
  </description>

解释：
task进程参数。参数如果出现@taskid@，会以当前TaskID对@taskid@进行内插替换。其他情况下的"@“符号将不会改变。
使用-Djava.library.path参数指定hadoop本地库可能会会导致程序无法正常工作。这些对应的参数应该做为LD_LIBRARY_PATH的一部分在map或reduce的JVM环境变量（mapredice.{map|reduce}.env）中进行设置。
如果java.opts参数中不包含”-Xmx"字段，则会根据mapreduce.map.memory和mapreduce.job.heap.memory-mb.ratio进行推断，计算出对应的值。

5. yarn.scheduler.minimum-allocation-mb

示例：

yarn.scheduler.minimum-allocation-mb=2048

Scope描述：

 <property>
    <description>The minimum allocation for every container request at the RM
    in MBs. Memory requests lower than this will be set to the value of this
    property. Additionally, a node manager that is configured to have less memory
    than this value will be shut down by the resource manager.</description>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>1024</value>
  </property>

解释：
在RM上每个容器请求的最小分配(以mb为单位)。低于此值的内存请求将被设置为该属性的值。此外，配置内存小于此值的节点管理器将被资源管理器关闭。
在不指定该参数的情况下，默认值为1024