Showing posts with label Microsoft Azure. Show all posts
Showing posts with label Microsoft Azure. Show all posts

Wednesday, December 23, 2020

PowerShell commands in Azure -Part 1

As I have been working with PowerShell for a while ,below are some of the common commands to work with PowerShell in Azure 

In this blog I cover the following topics 

  • How to Install Az Module in PowerShell
  • How to check the versions of Az available in PowerShell
  • How to connect to Azure using PowerShell
  • How to create or remove resource locks using PowerShell
  • How to create a policydefinition and Assign them to a resourcegroup using PowerShell

1. Install PowerShell module in local machine 

    Install-Module PowerShellGet -force -Scope CurrentUser 

2.Install AzureRM/Az packages : the latest recommended version is the az one 

    Install-Module -Name AzureRM -AllowClobber -Scope CurrentUser--older version 

    Install-Module -Name Az -AllowClobber -Scope CurrentUser

3 Get the list of Az versions installed in the machine 

Get-InstalledModule -Name Az -AllVersions

3 connect to Azure account 

Connect-AzAccount 

4 Get the list of resource groups within the subscription 

Get-AzResourceGroup

5 Get the list of resources within the resourcegroup

Get-AzResource -ResourceGroup <resource group name >

Implementing Resource Locks using PowerShell

Locks prevent the resources within a resource groups from being modified or deleted.There are two types of locks available .CanNotDelete and ReadOnly Locks 

CanNotDelete Locks : This will not allow any user to delete any resources within the resource group .When we try to delete any resource within the group ,it will throw an error 

ReadOnly Locks : This will prevent users from modifying any of the resources within the resource group .

The locks can be implemented both at the resource group level or we can set it up for individual resources within the resource group .

Now we can create locks using the portal /PowerShell/Azure CLI as well .Below are the commands which will help you get ,create and remove locks at both the resource group /or a particular resource within the resource group 

For the commands Below I have created a resource group called myrg180988 which has several resources including a SQL server database and I am trying to create two type of locks one which is a ReadOnly lock at the SQL server database resource and a CanNotDelete lock at the entire resource group level .Lets see how this can be accomplished using PowerShell commands 

#Step 1 : connecting to azure account -Connect-AzAccount

#Step 2 :removing the lock

Remove-AzResourceLock -LockName "mylock180988" -ResourceGroupName "myrg180988" -ResourceName "mydb180988/myDB" -ResourceType "Microsoft.Sql/servers/databases"

#Step 3 : adding a new lock at the DB level 

New-AzresourceLock -LockLevel ReadOnly -LockNotes "This will prevent the locks from being modified" -LockName "mylock180988" -ResourceGroupName "myrg180988" -ResourceName "mydb180988/myDB" -ResourceType "Microsoft.Sql/servers/databases"

#Step 4 : List the locks available 

Get-AzResourceLock -ResourceGroupName "myrg180988" 

#Step 5 : Setting a delete lock on the entire resourcegroup

New-AzResourceLock -LockLevel CanNotDelete -LockName "mylckn180988" -ResourceGroupName "myrg180988"

#Step 6 : Listing the resourcelocks available for the resourcegroup

Get-AzResourceLock -ResourceGroupName "myrg180988" |Select-Object -Property ResourceGroupName,ResourceName,LockName,Properties

#Step 8 : Removing the resource level lock 

Remove-AzResourceLock -LockName "mylock180988" -ResourceGroupName "myrg180988" -ResourceName "mydb180988/myDB" -ResourceType "Microsoft.Sql/servers/databases"

#Step 9 : Listing the resource locks again 

Get-AzResourceLock -ResourceGroupName "myrg180988" |Select-Object -Property ResourceGroupName,ResourceName,LockName,Properties

#Step 10 : removing the lock at the resource group level

Remove-AzResourceLock -LockName "mylckn180988" -ResourceGroupName "myrg180988"

#Step 11 : Listing the locks again

Get-AzResourceLock -ResourceGroupName "myrg180988" |Select-Object -Property ResourceGroupName,ResourceName,LockName,Properties

Policy Creation /Assignment using PowerShell

to Demo this am creating a new resourcegroup and then creating a policydefinition and assign this definition to the resourcegroup .For instance I want to create a policy which will ensure all my sql servers have threat detection enabled .So if you goto the portal and check for the policies related to SQL servers you can see this policy .We need to assign this policy into our resourcegroup .for that we need to create a policy definition using AzPolicyDefinition  command  which captures this policy details from microsoft and then we will use a AzPolicyAssignment command to assign this policy to the resource group created 





##Step 1 : Creating a resource group called mytestpolicy and assigning it into a Variable 

$myrggrp=Get-AzResourceGroup -Name "mytestpolicy" -Location "South India"

echo $myrggrp

#Step 2 : Creating a Policy Definition and assigning it to a variable

$mypolicydef= Get-AzPolicyDefinition |Where-Object{$_.Properties.DisplayName -eq "Deploy Threat Detection on SQL servers"}

echo $mypolicydef

#Step 3 : Assign this policy definition to the new resource groups 

New-AzPolicyAssignment -Name "audit-sql-threat-detection" -DisplayName "This will ensure the SQL servers have threat detection enabled" -Scope $myrggrp.ResourceId -PolicyDefinition $mypolicydef -Location "South India" -AssignIdentity

Note :the AssignIdentity is a property which we need to give manually when we assign policy through scripting ,which will manage the identity of the policy .when giving this AssignIdentity we need to specify a location as well 

#Step 4 : To check whether the resources are compliant to the policy or not 

Get-AzPolicyState -ResourceGroupName $myrggrp -PolicyAssignmentName  "audit-sql-threat-detection" -Filter 'IsCompliant eq false'

This will return the Resource groups where the policy is violated 

#Step 5 : delete the policyassignment 

Remove-AzPolicyAssignment -Name "audit-sql-threat-detection" -Scope "/subscriptions/9c350b81-c8d7-40f1-831b-11a0ea6eda3c/resourceGroups/mytestpolicy"

Note : for removing policyassignment we should give the complete path of the resourcegroup from the subscription 

#Step6 : delete the policyDefinition

Remove-AzPolicyDefinition -Name $mypolicydef.Name -SubscriptionId "9c350b81-c8d7-40f1-831b-11a0ea6eda3c"






Tuesday, January 28, 2020

Storage in Azure

Azure supports different storage options for both structured and unstructured data like specific formatted data ,binary files ,video /image files etc .Lets look at the different options and when and where to use a particular option .

Blobs

Blob storage is mainly used for storing text or binary files such as an image /video files .One practical application of blob storage is when you have a website through which you want users to upload their passport size photo which you then want to store it in Azure .Since Database is relational it cannot store the image file and hence blob storage will be a good option .As soon as the user clicks submit button the image goes and gets uploaded in the blob storage with a unique name (even multiple people upload the file using the same name ,eventually this gets converted into a unique name) and there could be a meta data information that is available in the SQL Database which has information like what is the email id name of the person etc with the link to the image file/the file name as another field .

There are different types of blobs like block blobs ,append blobs and page blobs that can be created

Queue :

Queues can be typically thought of as a messaging system .Lets assume millions of people are accessing our website from different location.In such cases as soon as the user submits the button the data gets added in the Queue where the requests are handled one by one .From the web app ,we could load the image into the queue and from there the data can be pushed to the blob storage which will help for a more robust environment

Files :

File storage system is one of the unique features of Azure .We can store any type of files and one major advantage of using a file system is that it can be mounted on any drives in the local /in Azure vm by using the connection string

Tables :

Tables provide a relational database like storage with a noSQL format ie data can be entered as a key value pair which will then be captured and stored in the table To view this storage account contents we need to connect to azure subscription through visual studio and click on the table to view the data stored in it .

Storage Account :

For creating any of the storage options listed above ,we need to have a storage account in Azure .This can be created using the portal by clicking on add resource or using azure cli/Powershell or even any client libraries

Once the storage account is created ,we can open it and we can see the options blob,queue,file or table there and based on the specific requirement we could go for the type .If it is a Blob storage we have to create containers which are nothing but logical organization of different files (folder like) and for files we need to again organize them in folders

Some Interesting options when creating the storage account is the replication which has the following options

LRS (Locally Redundant Storage)

This is a replication method where if the storage account created in one data center within the region is replicated into the same data centre .This is less expensive option when compared to the other mode of replications .If the data is not time sensitive /if it is an archival data or test data then LRS will be the right option to go with

ZRS (Zone Redundant Storage)

This is a replication method where if the storage account is in a particular geography in a region then the replication will be in a different data center in the same geography .This type of replications helps recovery if there is a failure in the data center

GRS (Geo Redundant Storage )

This is a replication method where if the storage account is in a particular region,the replication will be in another region .This would help in data recovery when there is a calamity /disaster where in the entire data in a region is destructive .This is an expensive option and if the data is complex and sensitive and we are looking for a 100% recoverability ,then one should choose a GRS replication

Read Only GRS 

Here the replication is same as GRS ,but during a disaster or if the region servers are down ,the data that is replicated in the other region will be read only .This is less expensive than GRS storage option

Access Tier 

There are two tiers when you create a storage account

Hot Tier : Optimized for storing the data that is accessed frequently .More expensive than cool /archive tier

Cool Tier : Optimized for storing the data that is infrequently used .

Archive Tier : Used for storing archival files like historical log files etc which will never be accessed .This is very cheap when compared to other two tiers .

The tier needs to be chosen appropriately based on the development requirements .








Azure SQL Database vs Azure SQL DW

There is always a question when going through the resources available in Azure when you see both the components Azure SQL Database and Azure SQL DW .May be because of this confusion ,Microsoft renamed Azure SQL DW resource as Azure Synapse Analytics .

But looking closely at practical usage of both the resources ,it is completely different from one another in terms of Azure pricing ,performance and the unique purpose by which one cannot be used instead of the other .

Lets examine the features one by one

Azure SQL Database is a DaaS(Database as a service) using the SQL Server Engine where as Azure SQL Data warehouse can parallel process huge volumes of data .

When do we use Azure SQL Database ?

Azure SQL Databases are commonly used when we have huge OLTP transactions which are normalized and the results might need a quick turn around time as well .

For eg : if we are building the back-end for a banking /consumer products website then ,the main criteria will be to meet all the ACID properties and the tables are all highly normalized with a quicker DML turnaround.In such scenarios it is good to go with the a SQL Server /the Azure equivalent of it which is Azure SQL Server


When do we use Azure SQL DW 

Azure SQL DW mainly works on the OLAP environment where huge volumes of denormalized data is stored .The data could be organized in Dim/Fact method using either star or snowflake schema as in a data warehouse .Transaction updates are very less in OLAP environment and hence these type of warehouses are mainly built for building reports such as annual report for sales ,monthly report on revenues ,yearly budget analysis and so on .

Size :The max size limit for Azure SQL DB is 4 TB whereas for Azure SQL DW there is no size limit

Pricing : In Azure SQL DB the pricing is based on DTU (Data Transaction Unit) where as in Azure SQL DW the pricing is DWU (Data Warehousing Unit) .At a high level DWU is more expensive than DTU (which is actually based on number of transactions only )

Parellel Connections : The number of concurrent sessions in Azure SQL DB is much higher than the warehouse .It can handle 6400 concurrent logins and 30000 concurrent sessions where as Azure SQL DW can handle only 1024 active connections .

Concurrent Queries : Azure SQL DB can execute 6400 concurrent queries at a time where as in Azure SQL DW a maximum of 128 concurrent queries get executed and the remaining are queued up

Polybase : Azure SQL DW supports polybase where as Azure SQL DB does not

Encryption : Azure SQL DW data is not encrypted where as Azure SQL DB supports encryption for the sensitive data

Replication : Azure SQL Database lets us replicate the Data using Geo Redundant storage whereas an Azure SQL DW does not have replication mechanism

Hope the article was helpful in understanding the basic differences between these two resources in the cloud .Based on the business needs ,one can decide whether to go with Azure SQL DB /Azure SQL DW

Note : the Azure SQL DW is renamed now in the portal(portal.azure.com) as  Azure Synapse Analytics