Dynamically Change the HDInsight Yarn Capacity Allocation

Dynamically Change the HDInsight Yarn Capacity Allocation

Requirement:

The fundamental unit of YARN is a queue. The user can submit a job to a specific queue. Each queue has a capacity defined by cluster admin and accordingly share of resources are allocated to the queue.

Requirement here is to re-define the YARN memory allocation using scheduled job.

Solution:

1.     Identify the config type to update and retrieve the latest yarn version. When any config type needs to update the whole property set needs to be updated. So, copy the entire property set from existing values.

2.     Update the TAG with unique value for new version. As a best practice use current timestamp as a TAG version

3.     Edit the properties as needed and then update the config type

4.     The new version of the config type must be added and applied to the cluster

We can use REST APIs or a wrapper script (/var/lib/ambari-server/resources/scripts/configs.py or /var/lib/ambari-server/resources/scripts/configs.sh) to edit configurations.

Implementation:

            Step 1: Go to this URL https://sparktest.azurehdinsight.net/api/v1/clusters/sparktest and identify the latest YARN Version. Find the screenshot below

                        YARN Queue Manager:

                       

                       

                       

Cluster Config URL:

                           

            Step 2: Update the property mentioned above as required

            Step 3: Using REST API we can submit the changes

curl -u "admin:pass" -H "X-Requested-By: ambari" -X PUT "http://hn0-sparketest:8080/api/v1/clusters/sparktestdev" -d '[{"Clusters":{
   "desired_config":[{
     "type":"capacity-scheduler",
     "tag":"version1573054562219",
     "properties":{
        "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.33",
            "yarn.scheduler.capacity.maximum-applications" : "10000",
            "yarn.scheduler.capacity.node-locality-delay" : "0",
            "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
            "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
            "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
            "yarn.scheduler.capacity.root.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.capacity" : "100",
            "yarn.scheduler.capacity.root.default.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.default.capacity" : "50",
            "yarn.scheduler.capacity.root.default.maximum-capacity" : "80",
            "yarn.scheduler.capacity.root.default.priority" : "0",
            "yarn.scheduler.capacity.root.default.state" : "RUNNING",
            "yarn.scheduler.capacity.root.default.user-limit-factor" : "10",
            "yarn.scheduler.capacity.root.priority" : "0",
            "yarn.scheduler.capacity.root.quality_analytics.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.quality_analytics.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.quality_analytics.capacity" : "40",
            "yarn.scheduler.capacity.root.quality_analytics.maximum-capacity" : "100",
            "yarn.scheduler.capacity.root.quality_analytics.minimum-user-limit-percent" : "100",
            "yarn.scheduler.capacity.root.quality_analytics.ordering-policy" : "fair",
            "yarn.scheduler.capacity.root.quality_analytics.ordering-policy.fair.enable-size-based-weight" : "false",
            "yarn.scheduler.capacity.root.quality_analytics.priority" : "0",
            "yarn.scheduler.capacity.root.quality_analytics.state" : "RUNNING",
            "yarn.scheduler.capacity.root.quality_analytics.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.queues" : "default,quality_analytics,thriftsvr",
            "yarn.scheduler.capacity.root.thriftsvr.capacity" : "10",
            "yarn.scheduler.capacity.root.thriftsvr.maximum-capacity" : "40",
            "yarn.scheduler.capacity.root.thriftsvr.priority" : "0",
            "yarn.scheduler.capacity.root.thriftsvr.user-limit-factor" : "10"
    },

    "service_config_version_note":"New config version"}]}}]'

Databricks Failure Alert Automation Using Powershell

Issue:

I got multiple databricks work space and received many complaints from end user on availability. Also as a admin, i was not aware or notified any availability issue in advance.

Solution:

To overcome this communication gap, i have written a powershell script to check databricks work space for every 5 minutes and notify me when service is not reachable or accessible.

Powershell Script:

clear

Try

{

#Install-Module -Name DatabricksPS

$Out = 0

$accessToken = "Admin Access Token "

$apiUrl = "https://XYZ.azuredatabricks.net"

Set-DatabricksEnvironment -AccessToken $accessToken -ApiRootUrl $apiUrl -ErrorAction Stop

Get-DatabricksJobRun -JobID 17410 | Get-DatabricksJobRunOutput -ErrorAction Stop

}

catch

{

Write-Output "Something threw an exception"

$emailSmtpServer = "mailrelay.xyz.com"

$emailFrom = “databricks@xyz.com"

$recipients = "user@xyz.com"

$emailSubject = "Alert! List of DataBricks Environment Unavailable"

$message = " Kindly check this environment availability - $apiUrl  - Here is the message $_.Exception.Message"

[string[]]$emailTo = $recipients.Split(',')

Send-MailMessage -To $emailTo -From $emailFrom -Subject $emailSubject -Body $message -SmtpServer $emailSmtpServer

}

Key Concepts in MDX

The purpose of MDX is to make accessing data from multiple dimensions easier and more sensitive.

Dimensions, Levels, Members, and Measures

Most languages used for data definition and manipulation, such as SQL, are designed to retrieve data in two dimensions: a column dimension and a row dimension. The following diagram illustrates a traditional relational database, used to store order information.

Each table represents two-dimensional data. At the intersection of each row and column is a single element of data, called a field. The specific columns to be viewed in an SQL query are specified with a SELECT statement, and the rows to be retrieved are limited by a WHERE clause.

Multidimensional data, on the other hand, can be represented by structures with more than two dimensions. These structures, called cubes, have multiple dimensions. At the intersection of dimensions in a cube, there may be more than one element of data, called a measure. The following diagram illustrates a cube that employs three dimensions, Route, Service and Time; and two measures, Packages and Last. Each dimension is broken down into different levels, each of which is broken down further into members. For example, the Source dimension supplies the Eastern Hemisphere level, which is broken down into four members, Africa, Asia, Australia, and Europe.

As you can see, the querying of even simple data out of a multidimensional data source can be a complex task. A cube can have more than three dimensions, for example, or it may only have one dimension.

The concepts of cubes, dimensions, levels, members, and measures are important to the understanding of MDX syntax.

About MDX

MDX Overview

This article introduces Multidimensional Expressions (MDX) and explains some of the concepts behind its structure and syntax.


Introduction to MDX

MDX, an acronym for Multidimensional Expressions, is a syntax that supports the definition and manipulation of multidimensional objects and data. MDX is similar in many ways to the Structured Query Language (SQL) syntax, but is not an extension of the SQL language; in fact, some of the functionality that is supplied by MDX can be supplied, although not as efficiently or intuitively, by SQL.
As with an SQL query, each MDX query requires a data request (the SELECT clause), a starting point (the FROM clause), and a filter (the WHERE clause). These and other keywords provide the tools used to extract specific portions of data from a cube for analysis. MDX also supplies a robust set of functions for the manipulation of retrieved data, as well as the ability to extend MDX with user-defined functions.
MDX, like SQL, provides data definition language (DDL) syntax for managing data structures. There are MDX commands for creating (and deleting) cubes, dimensions, measures, and their subordinate objects.

Key Concept and Dimensions are continued in the next post

About Trigger

Trigger

A trigger is a database object that is bound to a table. It is similar to a stored procedure.Triggers are often referred to as a "special kind of stored procedure".

Where we Use Triggers ?

There are many reasons to use triggers. To avoid unnecessary delete in the table.Also used to log the table details.If you want to do other stuff while in progress.

Tables can have multiple triggers. The CREATE TRIGGER statement can be defined with the FOR UPDATE, FOR INSERT, or FOR DELETE clauses to target a trigger to a specific class of data modification actions. When FOR UPDATE is specified, the IF UPDATE (column_name) clause can be used to target a trigger to updates affecting a particular column.

SQL Server 2000 greatly enhances trigger functionality, extending the capabilities of the triggers you already know and love, and adding a whole new type of trigger, the "Instead Of" trigger.

SQL Server 2000 has many types of triggers:

1. After Trigger
2. Multiple After Triggers
3. Instead Of Triggers
4. Mixing Triggers Type

Introduction to Microsoft SQL Server 2000

Microsoft SQL Server 2000 is a full-featured RDBMS that offers a variety of administrative tools to ease the burdens of database development, maintenance and administration. The tools available in sql server 2000 are Enterprise Manager, Query Analyzer, SQL Profiler, Service Manager and Data Transformation Services.

Enterprise Manager is the main administrative console for SQL Server installations. It provides you with a graphical view of all of the SQL Server installations on your network. You can perform high-level administrative functions that affect one or more servers, schedule common maintenance tasks or create and modify the structure of individual databases.

Query Analyzer offers a quick and dirty method for performing queries against any of your SQL Server databases.It's a great way to quickly pull information out of a database in response to a user request, test queries before implementing them in other applications, create/modify stored procedures and execute administrative tasks.

SQL Profiler provides a window into the inner workings of your database. You can monitor many different event types and observe database performance in real time. SQL Profiler allows you to capture and replay system "traces" that log various activities. It's a great tool for optimizing databases with performance issues or troubleshooting particular problems.

Service Manager is used to control the MSSQLServer (the main SQL Server process), MSDTC (Microsoft Distributed Transaction Coordinator) and SQLServerAgent processes. An icon for this service normally resides in the system tray of machines running SQL Server. You can use Service Manager to start, stop or pause any one of these services.

Data Transformation Services (DTS) provide an extremely flexible method for importing and exporting data between a Microsoft SQL Server installation and a large variety of other formats. The most commonly used DTS application is the "Import and Export Data" wizard found in the SQL Server program group.

Dotnet for beginner

About ASP

ASP is a server side scripting technology that enables scripts to be executed by an Internet server.
ASP is a Microsoft Technology
ASP stands for Active Server Pages
ASP is a program that runs inside IIS
IIS stands for Internet Information Services
IIS comes as a free component with Windows 2000
IIS is also a part of the Windows NT 4.0 Option Pack
The Option Pack can be downloaded from Microsoft
PWS is a smaller - but fully functional - version of IIS
PWS can be found on your Windows 95/98 CD

What is an ASP File?

An ASP file is just the same as an HTML file
An ASP file can contain text, HTML, XML, and scripts
Scripts in an ASP file are executed on the server
An ASP file has the file extension ".asp"

How does it Work?

When a browser requests an HTML file, the server returns the file
When a browser requests an ASP file, IIS passes the request to the ASP engine on the server
The ASP engine reads the file, line by line, and executes the scripts in the file
Finally, the ASP file is returned to the browser as plain HTML

What is ASP+?

ASP+ is the same as ASP.NET.
ASP+ is just an early name used by Microsoft when they developed ASP.NET.

What is ASP.NET?

ASP 3.0 is the latest version of ASP, but there will never be an ASP 4.0 version.
ASP.NET is the next generation ASP, but it's not an upgraded version of ASP. ASP.NET is an entirely new paradigm for server-side ASP scripting.
ASP.NET is a part of the .NET Framework. Microsoft spent three years rewriting ASP.NET from the ground up, and ASP.NET is not fully backward compatible with ASP 3.0.
You can read more about the differences between ASP and ASP.NET in the next chapter of this tutorial.

Differences between ASP and ASP.NET ?

ASP. net supports object oriented techniques.
ASP.net is compiled, rather than interpreted, so overall performance of asp.net application is much better.
ASP.net supports two new mode of authentication, form based and passport based authentication
ASP.net supports ADO.net.