Dynamically Change the HDInsight Yarn Capacity Allocation

Dynamically Change the HDInsight Yarn Capacity Allocation

Requirement:

The fundamental unit of YARN is a queue. The user can submit a job to a specific queue. Each queue has a capacity defined by cluster admin and accordingly share of resources are allocated to the queue.

Requirement here is to re-define the YARN memory allocation using scheduled job.

Solution:

1.     Identify the config type to update and retrieve the latest yarn version. When any config type needs to update the whole property set needs to be updated. So, copy the entire property set from existing values.

2.     Update the TAG with unique value for new version. As a best practice use current timestamp as a TAG version

3.     Edit the properties as needed and then update the config type

4.     The new version of the config type must be added and applied to the cluster

We can use REST APIs or a wrapper script (/var/lib/ambari-server/resources/scripts/configs.py or /var/lib/ambari-server/resources/scripts/configs.sh) to edit configurations.

Implementation:

            Step 1: Go to this URL https://sparktest.azurehdinsight.net/api/v1/clusters/sparktest and identify the latest YARN Version. Find the screenshot below

                        YARN Queue Manager:

                       

                       

                       

Cluster Config URL:

                           

            Step 2: Update the property mentioned above as required

            Step 3: Using REST API we can submit the changes

curl -u "admin:pass" -H "X-Requested-By: ambari" -X PUT "http://hn0-sparketest:8080/api/v1/clusters/sparktestdev" -d '[{"Clusters":{
   "desired_config":[{
     "type":"capacity-scheduler",
     "tag":"version1573054562219",
     "properties":{
        "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.33",
            "yarn.scheduler.capacity.maximum-applications" : "10000",
            "yarn.scheduler.capacity.node-locality-delay" : "0",
            "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
            "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
            "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
            "yarn.scheduler.capacity.root.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.capacity" : "100",
            "yarn.scheduler.capacity.root.default.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.default.capacity" : "50",
            "yarn.scheduler.capacity.root.default.maximum-capacity" : "80",
            "yarn.scheduler.capacity.root.default.priority" : "0",
            "yarn.scheduler.capacity.root.default.state" : "RUNNING",
            "yarn.scheduler.capacity.root.default.user-limit-factor" : "10",
            "yarn.scheduler.capacity.root.priority" : "0",
            "yarn.scheduler.capacity.root.quality_analytics.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.quality_analytics.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.quality_analytics.capacity" : "40",
            "yarn.scheduler.capacity.root.quality_analytics.maximum-capacity" : "100",
            "yarn.scheduler.capacity.root.quality_analytics.minimum-user-limit-percent" : "100",
            "yarn.scheduler.capacity.root.quality_analytics.ordering-policy" : "fair",
            "yarn.scheduler.capacity.root.quality_analytics.ordering-policy.fair.enable-size-based-weight" : "false",
            "yarn.scheduler.capacity.root.quality_analytics.priority" : "0",
            "yarn.scheduler.capacity.root.quality_analytics.state" : "RUNNING",
            "yarn.scheduler.capacity.root.quality_analytics.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.queues" : "default,quality_analytics,thriftsvr",
            "yarn.scheduler.capacity.root.thriftsvr.capacity" : "10",
            "yarn.scheduler.capacity.root.thriftsvr.maximum-capacity" : "40",
            "yarn.scheduler.capacity.root.thriftsvr.priority" : "0",
            "yarn.scheduler.capacity.root.thriftsvr.user-limit-factor" : "10"
    },

    "service_config_version_note":"New config version"}]}}]'