List of YARN Enhancements in Version 5.2.0

MapR 5.2 runs the 2.7.0 version of Hadoop. In the MapR 5.2.0 release, a number of bug fixes from various Hadoop versions were backported into 2.7.0.

Bug fixes backported in 5.2.0

The following bug fixes were backported from Hadoop 2.7.3 to Hadoop 2.7.0 for 5.2.0:
  • YARN-4452: NPE when submit Unmanaged application

  • YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page

  • YARN-3849: Too much of preemption activity causing continuos killing of containers across queues

  • YARN-4794: Deadlock in NMClientImpl

  • YARN-4546: ResourceManager crash due to scheduling opportunity overflow

  • MAPREDUCE-6622: Add capability to set JHS job cache to a task-based limit

  • MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException

  • MAPREDUCE-6454: MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.

  • MAPREDUCE-6376: Add avro binary support for jhist files

  • MAPREDUCE-6554: MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory

  • MAPREDUCE-6302: Preempt reducers after a configurable timeout irrespective of headroom

  • MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers

  • MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests

  • MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API

  • MAPREDUCE-6555: TestMRAppMaster fails on trunk

  • MAPREDUCE-6595: Fix findbugs warnings in OutputCommitter and FileOutputCommitter

  • MAPREDUCE-6478: Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob().

The following bug fixes were backported from Hadoop 2.7.2 to Hadoop 2.7.0 for MapR 5.2.0:
  • YARN-4424: Fix deadlock in RMAppImpl

  • YARN-4354: Public resource localization fails with NPE

  • YARN-2902: Killing a container that is localizing can orphan resources in the DOWNLOADING state

  • YARN-4180: AMLauncher does not retry on failures when talking to NM

  • YARN-3697: FairScheduler: ContinuousSchedulingThread can fail to shutdown

  • YARN-3535: Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED

  • YARN-3857: Memory leak in ResourceManager with SIMPLE mode

  • YARN-3878: AsyncDispatcher can hang while stopping if it is configured for draining events on stop

  • YARN-3990: AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected

  • YARN-3793: Several NPEs when deleting local files on NM recovery

  • YARN-4047: ClientRMService getApplications has high scheduler lock contention

  • HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.

  • MAPREDUCE-5948: org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well

  • MAPREDUCE-6481: LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes.

  • MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask

  • MAPREDUCE-5982: Task attempts that fail from the ASSIGNED state can disappear

  • MAPREDUCE-5649: Reduce cannot use more than 2G memory for the final merge

  • MAPREDUCE-6549: Multibyte delimiters with LineRecordReader cause duplicate records

The following bug fixes were backported from other Hadoop versions to Hadoop 2.7.0:
  • YARN-2336: Fair scheduler REST api returns a missing '[' bracket JSON for deep queue tree

  • YARN-1050: Document the Fair Scheduler REST API

  • YARN-3464: Race condition in LocalizerRunner kills localizer before localizing all resources

  • YARN-4041: Slow delegation token renewal can severely prolong RM recovery

  • YARN-4347: Resource manager fails with Null pointer exception

  • MAPREDUCE-6284: Add Task Attempt State API to MapReduce Application Master REST API

  • MAPREDUCE-6316: Task Attempt List entries should link to the task overview

  • MAPREDUCE-6364: Add a "Kill" link to Task Attempts pageAdd a "Kill" link to Task Attempts page

  • MAPREDUCE-6405: NullPointerException in App Attempts page

  • MAPREDUCE-6419: JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed 9999

  • MAPREDUCE-5649: Reduce cannot use more than 2G memory for the final merge