tag:blogger.com,1999:blog-3022982869287424222024-03-18T21:54:30.168-07:00#cloud blogDBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.comBlogger467125tag:blogger.com,1999:blog-302298286928742422.post-9472578030354133362022-02-19T06:42:00.009-08:002022-02-19T06:42:56.648-08:00How much storage am i using?<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgrh7t5BWtg-yNsHoQYJyvrsFLzOcIbbu1GxLRCzf2SaDpTONpbzopbrXyOBy5SwlG_J-GOPgJR9FYkMFgXNZmQ5kRUUIj31LucCFx9-Us0qbmNPAvebdZvLr4zjkVX5la8rteFiO05G-BWsn6bCuhhGCGxxvi47daMELQgDz3CavngNm1She4_qyXX=s640" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="426" data-original-width="640" height="213" src="https://blogger.googleusercontent.com/img/a/AVvXsEgrh7t5BWtg-yNsHoQYJyvrsFLzOcIbbu1GxLRCzf2SaDpTONpbzopbrXyOBy5SwlG_J-GOPgJR9FYkMFgXNZmQ5kRUUIj31LucCFx9-Us0qbmNPAvebdZvLr4zjkVX5la8rteFiO05G-BWsn6bCuhhGCGxxvi47daMELQgDz3CavngNm1She4_qyXX=s320" width="320" /></a></div><br /><p>We were recently asked how much Azure storage we are actually using - initially I thought this was going to be a simple thing to find out - but actually it turned out to be a little more complex than I thought so here are a few notes about how i went about it.</p><p>Still kind of feels like I missed some easier method of doing this - if so please let me know....</p><p>My first thought was that this is something that is just available in the portal - indeed it is for a single storage account just on the insights blade - it gives a total along with a breakdown of the different storage types</p><p><br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjB96wkpr9oWoaXbkAw7CwRmLtEJwtbR_8AYD2qyVuJPPCRV_sEZvOBX7-nahfGOc6TlDIHQ7q1zWyZl-5194TjI_R3WubBzypuoblJajcZFOxgySIaEkPNF9yctWN0uUOwHvmZcVBUkcDn8BZz2MsaiE_6h8qUQ3dmK7Pq8LoFv1G5XH_Pkkb9Wwuw=s1649" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="712" data-original-width="1649" height="138" src="https://blogger.googleusercontent.com/img/a/AVvXsEjB96wkpr9oWoaXbkAw7CwRmLtEJwtbR_8AYD2qyVuJPPCRV_sEZvOBX7-nahfGOc6TlDIHQ7q1zWyZl-5194TjI_R3WubBzypuoblJajcZFOxgySIaEkPNF9yctWN0uUOwHvmZcVBUkcDn8BZz2MsaiE_6h8qUQ3dmK7Pq8LoFv1G5XH_Pkkb9Wwuw=s320" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">If i then go to Azure Monitor to look at the storage insights I thought this would then allow me to just scale up that view and get the overall picture. Navigating to there I see this:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi7Me_7H9aggKylbRrrtSEux_32RQ9J1O-6ySvxItxjtsFPJOcof61jFVGbdfYu-vofw3LBBYZQfhb_WHUJ7XBFnYuk-OI0Rnf2U-6KFbenUkZpgJ0Y106MD68ccBlOEc0bSilJs7pSdyknxVucAS6t7oZMa_kBrq9zZE0fwgxwNBowEd0dje3EX2ix=s1806" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="822" data-original-width="1806" height="146" src="https://blogger.googleusercontent.com/img/a/AVvXsEi7Me_7H9aggKylbRrrtSEux_32RQ9J1O-6ySvxItxjtsFPJOcof61jFVGbdfYu-vofw3LBBYZQfhb_WHUJ7XBFnYuk-OI0Rnf2U-6KFbenUkZpgJ0Y106MD68ccBlOEc0bSilJs7pSdyknxVucAS6t7oZMa_kBrq9zZE0fwgxwNBowEd0dje3EX2ix=s320" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: left;"><br /></div>So it looks like this might give me what i want - there is a button to download to excel which should then allow me to just sum everything up. However there is an issue - we seem to have too many storage accounts for it to cope with and the report - the built in one is limited to 200 resources - this means I would have to manually filter multiple times to report across everything - likely to be error prone and very time consuming.<br /><br /><p></p><p>So i thought I'll just create my own copy of the report and tweak it to remove the limits - easy enough. I then discovered that this had 2 issues.</p><p>1. There is a hard limit internally in azure workbooks of 1000 items - after that data is just not shown</p><p>2. A lot of the time I was hitting insights api errors - not sure if that was due to throttling or some sort of capacity issues at the time but it was not reliable</p><p>Then I had a rethink - maybe this is available in resource graph query? (well short answer to that is that it isn't)</p><p>So what to do - it looked like I was going to have to write some sort of script either with azcli, powershell or something calling rest API's in order to get all this data.</p><p>Of these options the easiest for me is azcli - so I wrote a short script to extract the data I needed.</p><p>This is 'cheap and cheerful' no error checking and a real bare bones script but it gets the job done. The bonus being that you can just run it in cloud shell (bash) from a browser.</p><p>So here is the script:</p><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff; font-weight: bold;">for </span>ider in <span style="color: #bb4444;">`</span>az account list |jq -r <span style="color: #bb4444;">'.[].id'`</span>
<span style="color: #aa22ff; font-weight: bold;">do </span>
az account <span style="color: #aa22ff;">set</span> --subscription <span style="color: darkgoldenrod;">$ider</span>
<span style="color: #aa22ff; font-weight: bold;">for </span>rec in <span style="color: #bb4444;">`</span>az resource list --resource-type Microsoft.Storage/storageAccounts |jq -r <span style="color: #bb4444;">'.[].id'`</span>
<span style="color: #aa22ff; font-weight: bold;">do</span>
az monitor metrics list --resource <span style="color: darkgoldenrod;">$rec</span> --interval PT1H --metric <span style="color: #bb4444;">"UsedCapacity"</span> |jq -r <span style="color: #bb4444;">'.value[].id,.value[].timeseries[].data[].average'</span> |paste - - -d <span style="color: #bb4444;">","</span> >>rich.txt
<span style="color: #aa22ff; font-weight: bold;">done</span>
<span style="color: #aa22ff; font-weight: bold;">done</span>
</pre></div>
<p>There is nothing particularly clever here - i just loop through all subscriptions, within each subscription loop through all the storage accounts and then extract the metrics information and send the output to a plain text file.</p><p>I make use of jq to extract values form the json data that is returned - this is a really nice utility for that - just needs a big of getting your head round with the syntax.</p><p>The end result of that is a plain text file with 2 columns - the full resource if of the storage account (containing the sub,resource group and storage account name) along with the size in bytes of the used capacity.</p><p>All i then need to do is sum up the 2nd column (the total bytes used) and convert into a suitable measurement unit - in our case PB.....</p><p>A useful exercise - but still feels like it should be easier - lifting the workbook limit would make this far simpler for example. I also found that my approach is much slower at fetching the data that the workbook - so its somehow using a much more efficient API to fetch the data than the command line.</p><p>If nothing else introducing you to jq is definitely something worth knowing.</p><p>P.S. the above report does not include storage for disks attached to VM's - these are handled differently as we use managed storage accounts for all VM disks</p><p><br /></p><p><br /></p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com1tag:blogger.com,1999:blog-302298286928742422.post-91588868087861069752022-01-07T02:41:00.000-08:002022-01-07T02:41:29.087-08:00Azure carbon footprint calculator<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhesfvIf5zblQjnJvqkqEl_bGGvnTS4GcQwvjofDDL_t8mwGsEu5hoKS5yBLTE1ondwguBCV6BPsZSxQbB86A5FvQiLhfPxyB7qcE4AmdKJoYiMS6vHQCo-e0_d0dLnWBMxxDVflmfsLvA814ZtPCTF-7FaoFux7mc-5DPrU2U_C--K3hgiLRl9qx4I=s640" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="426" data-original-width="640" height="213" src="https://blogger.googleusercontent.com/img/a/AVvXsEhesfvIf5zblQjnJvqkqEl_bGGvnTS4GcQwvjofDDL_t8mwGsEu5hoKS5yBLTE1ondwguBCV6BPsZSxQbB86A5FvQiLhfPxyB7qcE4AmdKJoYiMS6vHQCo-e0_d0dLnWBMxxDVflmfsLvA814ZtPCTF-7FaoFux7mc-5DPrU2U_C--K3hgiLRl9qx4I=s320" width="320" /></a></div><br /> <p></p><p><br /></p><p>Sustainability is rightly at the top of many companies agendas right now, for us as an energy producer we are acutely aware of our responsibilities here and we have made a number of key statements and directives of how our generation assets and the wider businesses will make improvements here.</p><p>I'm in IT though and we can sometimes feel removed from what impacts we can directly have in this area. We've been looking in to how we can contribute and there are a number of initiatives and improvements planned. What I wanted to talk about here though is Azure and what we are doing here.</p><p>So where to start? As the eminent Victorian scientist Lord Kelvin put it:</p><p><b>"If you cannot measure it, you cannot improve it"</b></p><p>So how do we measure it? - we have loads of information on what we consume in Azure from all our billing reports - but how does that equate to actual carbon and other greenhouse gas emissions? Until last year we could only guess as we had no insight in to what our usage was producing. That all changed though when Microsoft released their sustainability calculator (currently still in public preview) :</p><p><a href="https://aka.ms/SustainabilityCalculator">https://aka.ms/SustainabilityCalculator</a></p><p>This free tool (though you will need power bi pro to be able to use it) shows the carbon emissions from our company specific usage of Azure (you can see our real screens below - with figures blanked out..)</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEghqnlMTzCB_r6dJigRWLkOuFgxfqfejtiMmIJlOxLw7KbbhNm8Ygk5-eyoF1URfluyEjihvVgFjPhbNmXVQzX6frIMLPiHTDI0QjxVQWeRImzUDxocleMRhoY5owyAAv3TEnZyT9Cj2NTa09kOElZiWnKZepPxTnzx2HiZki3C6w4_ITwFlYbB_XhE=s1881" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="830" data-original-width="1881" height="176" src="https://blogger.googleusercontent.com/img/a/AVvXsEghqnlMTzCB_r6dJigRWLkOuFgxfqfejtiMmIJlOxLw7KbbhNm8Ygk5-eyoF1URfluyEjihvVgFjPhbNmXVQzX6frIMLPiHTDI0QjxVQWeRImzUDxocleMRhoY5owyAAv3TEnZyT9Cj2NTa09kOElZiWnKZepPxTnzx2HiZki3C6w4_ITwFlYbB_XhE=w400-h176" width="400" /></a></div><div><br /></div><div>It's not just a high level set of figures though we can drill down by a number of dimensions - namely subscription, azure service, region and then year/month to allow us to then analyse where our generation is coming from.</div><div><br /></div><div>This is a huge step forward in our ability to see the impact of our usage - however it didn't go as far as we wanted. We had a vision to take it down to the application level - so we can understand the real world impact of individual applications.</div><div><br /></div><div>In our deployment model the resource group is the logical unit that encompasses the application so we needed to get the figures at that level - but how is that possible we are not given the information at that level? ( I have given this feedback to the MS product team so it may make it into the dataset at some future date hopefully). Well the answer is to combine it with information we do have about our usage - the billing data.</div><div><br /></div><div>For example we know we are spending 'x' on virtual machines for resource group 'y'. Resource group 'y' makes up 50% of the total virtual machine costs for subscription 'z' so we can make the rough assumption that therefore 50% of the emissions for subscription 'z' belong to resource group 'y'. In our model the resource group directly correlates to application 'a' so we know how much carbon application 'a' is generating by then linking back to the actual figures from the above reports (this won't be 100% accurate as some machine types will be more efficient than others, reservations may be involved etc - but it's good enough for what we want to achieve here).</div><div><br /></div><div>So to produce a report per application we just need to take:</div><div><br /></div><div>1. An extract from the report above (downloadable as excel from the report above)</div><div>2. The billing/usage data for our company from the cost management services (again downloadable/extractable in many formats)</div><div>3. Application related information - from our CMDB</div><div><br /></div><div>We have all of this data it just needs to be combined and then a nice front end stuck on it.</div><div><br /></div><div>So that's what we did - we took the 3 datasets above and loaded them into a relational database - in our case we used snowflake as it was available and already contained some of the information we needed (any RDBMS will do here though - it just needs to support some minor analytic functions to do some of the calculations ).</div><div><br /></div><div>I won't cover the whole process of how that was done as it would take too long to write up all the steps - essentially though it's just taking the data extracts and loading them into some simple relational tables - it's not a hugely complicated process.</div><div><br /></div><div>Once that data is loaded I then needed to create the SQL to combine the data into an easily consumable form for the front end. The simple way to do this is just to create a view that does all the heavy lifting . The view that I then created looks like this:</div><div><br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiATMTaO6zsRE1cP2pMVuvvg8xHMzc5FUyq44zTrbnWOfN94QT911EkS0Th6Jxkc1rtBBm8bJlMK6w9s_m06T6djYSiRNbb1umDbgahhHbkWCwfWaqtY5F9KAX61SI_grpQXyCysWF0nhhyUsNsPWi7Mq4Ggy6EyqW7vF_8Sn4pFTzNXzV8WfJMIk6B=s1146" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="673" data-original-width="1146" height="235" src="https://blogger.googleusercontent.com/img/a/AVvXsEiATMTaO6zsRE1cP2pMVuvvg8xHMzc5FUyq44zTrbnWOfN94QT911EkS0Th6Jxkc1rtBBm8bJlMK6w9s_m06T6djYSiRNbb1umDbgahhHbkWCwfWaqtY5F9KAX61SI_grpQXyCysWF0nhhyUsNsPWi7Mq4Ggy6EyqW7vF_8Sn4pFTzNXzV8WfJMIk6B=w400-h235" width="400" /></a></div><br /><div><br /></div><div>The underlying SQL for that looking like this:</div><div><br /><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #aa22ff; font-weight: bold;">CREATE</span> <span style="color: #aa22ff; font-weight: bold;">VIEW</span> VW_AZURE_USAGE <span style="color: #aa22ff; font-weight: bold;">AS</span> <span style="color: #aa22ff; font-weight: bold;">with</span> <span style="color: #aa22ff; font-weight: bold;">usage</span> <span style="color: #aa22ff; font-weight: bold;">as</span> (
<span style="color: #aa22ff; font-weight: bold;">select</span>
subscriptionname,
subscriptionid,
resourcelocation,
resourcegroup,
EAM_ID,
applicationname,
tot,
ratio_to_report(tot) over (partition <span style="color: #aa22ff; font-weight: bold;">by</span> subscriptionname) <span style="color: #aa22ff; font-weight: bold;">as</span> perc
<span style="color: #aa22ff; font-weight: bold;">from</span>
(
<span style="color: #aa22ff; font-weight: bold;">Select</span>
subscriptionname,
subscriptionid,
resourcelocation,
resourcegroup,
EAM_ID,
applicationname,
round(<span style="color: #aa22ff; font-weight: bold;">sum</span>(costinbillingcurrency)) <span style="color: #aa22ff; font-weight: bold;">as</span> tot
<span style="color: #aa22ff; font-weight: bold;">from</span>
(
<span style="color: #aa22ff; font-weight: bold;">SELECT</span>
AZ.SUBSCRIPTIONNAME,
AZ.RESOURCEGROUP,
AZ.subscriptionid,
RM.EAM_ID,
PE.<span style="color: darkgoldenrod;">"NAME"</span> <span style="color: #aa22ff; font-weight: bold;">AS</span> ApplicationName,
PE.LOB_PARENT <span style="color: #aa22ff; font-weight: bold;">AS</span> LineOfBusiness,
PE.OWNERSHIP_BU,
PE.BUID,
AZ.resourcelocation,
AZ.COSTINBILLINGCURRENCY
<span style="color: #aa22ff; font-weight: bold;">FROM</span>
T_RAW_AZURE_COST AZ
<span style="color: #aa22ff; font-weight: bold;">LEFT</span> <span style="color: #aa22ff; font-weight: bold;">JOIN</span> (
<span style="color: #aa22ff; font-weight: bold;">SELECT</span>
<span style="color: #aa22ff; font-weight: bold;">owner</span>,
owner_email,
eam_id,
RESOURCEGROUP,
subname
<span style="color: #aa22ff; font-weight: bold;">FROM</span>
T_AZURE_RES_MAPPING
) RM <span style="color: #aa22ff; font-weight: bold;">ON</span> <span style="color: #aa22ff; font-weight: bold;">upper</span>(RM.RESOURCEGROUP) <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">upper</span>(AZ.RESOURCEGROUP)
<span style="color: #aa22ff; font-weight: bold;">AND</span> <span style="color: #aa22ff; font-weight: bold;">UPPER</span>(RM.subname) <span style="color: #666666;">=</span> <span style="color: #aa22ff; font-weight: bold;">UPPER</span>(AZ.SUBSCRIPTIONNAME)
<span style="color: #aa22ff; font-weight: bold;">LEFT</span> <span style="color: #aa22ff; font-weight: bold;">JOIN</span> (
<span style="color: #aa22ff; font-weight: bold;">SELECT</span>
NAME,
EAM_ID,
LOB_PARENT,
OWNERSHIP_BU,
BUID
<span style="color: #aa22ff; font-weight: bold;">FROM</span>
T_UPMX_PORTFOLIO_ELEMENTS
) PE <span style="color: #aa22ff; font-weight: bold;">ON</span> to_char(RM.EAM_ID) <span style="color: #666666;">=</span> to_char(PE.EAM_ID)
<span style="color: #aa22ff; font-weight: bold;">where</span>
AZ.metercategory <span style="color: #666666;">=</span> <span style="color: #bb4444;">'Virtual Machines'</span>
<span style="color: #aa22ff; font-weight: bold;">and</span> AZ.resourcelocation <span style="color: #666666;">=</span> <span style="color: #bb4444;">'westeurope'</span>
)
<span style="color: #aa22ff; font-weight: bold;">group</span> <span style="color: #aa22ff; font-weight: bold;">by</span>
subscriptionname,
subscriptionid,
resourcelocation,
resourcegroup,
EAM_ID,
applicationname
)
)
<span style="color: #aa22ff; font-weight: bold;">select</span>
subscriptionname,
resourcegroup,
<span style="color: #aa22ff; font-weight: bold;">scope</span>,
emission_mtco2e,
perc,
eam_id,
applicationname,
tot,(perc <span style="color: #666666;">*</span> emission_mtco2e) <span style="color: #aa22ff; font-weight: bold;">as</span> co2output
<span style="color: #aa22ff; font-weight: bold;">from</span>
<span style="color: #aa22ff; font-weight: bold;">usage</span>
<span style="color: #aa22ff; font-weight: bold;">left</span> <span style="color: #aa22ff; font-weight: bold;">join</span> T_AZURE_EMISSION EM <span style="color: #aa22ff; font-weight: bold;">on</span> EM.SUBSCRIPTION_ID <span style="color: #666666;">=</span> SUBSCRIPTIONID
<span style="color: #aa22ff; font-weight: bold;">where</span>
AZURE_REGION <span style="color: #666666;">=</span> <span style="color: #bb4444;">'West Europe'</span>
<span style="color: #aa22ff; font-weight: bold;">and</span> AZURE_SERVICE <span style="color: #666666;">=</span> <span style="color: #bb4444;">'Virtual Machines'</span>
<span style="color: #aa22ff; font-weight: bold;">and</span> reportdate <span style="color: #666666;">=</span> <span style="color: #bb4444;">'2021-08-01'</span>
<span style="color: #aa22ff; font-weight: bold;">and</span> applicationname <span style="color: #aa22ff; font-weight: bold;">is</span> <span style="color: #aa22ff; font-weight: bold;">not</span> <span style="color: #aa22ff; font-weight: bold;">null</span>;
</pre></div>
</div><div><br /></div><div><br /></div>The view is maybe simpler than it looks at first glance - it's only really made complex by the use of the ratio_to_report analytic function that allows me to easily work out what ratio of the total emissions belongs to each resource group/application.<div><br /></div><div>Once I have that view all I then need to do is make that look good in a front end tool - this is not my forte but Powerbi allows you to fairly easily create a half decent visual even if you don't really know what you are doing....</div><div><br /></div><div>The screen is then just essentially displaying "Select * from view" - the only extra bit I do is convert the co2 value in to something more 'relatable' - in this case tree's growing to absorb the carbon emitted</div><div><br /></div><div>The end result is shown below (redacted to remove the app details - but you still get the idea)</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjdDJwQqYkQmFJEDKIPjR4-6G4xnts1Qo9oHaFSCOsmCRt-Qau8j_pgZjWuQL20VnGG1Pv2UW8mSrlZqUZ5NOl-jbT8nsGUK_muzC_rMmb9jub25y7FH-2WFkNlkeWqxvFJLl_GV0USGDJ8jyJNIQw8sfYguoxfHCQLBZSa8VGNdsVBs5lt9DVBk8Cr=s1693" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="689" data-original-width="1693" height="163" src="https://blogger.googleusercontent.com/img/a/AVvXsEjdDJwQqYkQmFJEDKIPjR4-6G4xnts1Qo9oHaFSCOsmCRt-Qau8j_pgZjWuQL20VnGG1Pv2UW8mSrlZqUZ5NOl-jbT8nsGUK_muzC_rMmb9jub25y7FH-2WFkNlkeWqxvFJLl_GV0USGDJ8jyJNIQw8sfYguoxfHCQLBZSa8VGNdsVBs5lt9DVBk8Cr=w400-h163" width="400" /></a></div><div><br /></div>We then have a view for each application of its environmental impact.<div><br /></div><div>So what you might say?</div><div><br /></div><div>Well referring back to the top of the post it all starts with measuring - we now have a baseline and we can see trends over time so we can track (hopefully) improvements.</div><div><br /></div><div>How can I improve though is your likely next question - I don't control how MS runs the datacentre. That is of course true - but you can control how you use services within that - for example you could:</div><div><br /></div><div>1) Right size</div><div>2) Shutdown when not in use</div><div>3) Modernize away from Virtual machines to more massively shared PaaS services where the relative emissions are less</div><div>4) Build non prod environments on demand</div><div><br /></div><div>etc...</div><div><br /></div><div>You get the idea - there are things you can control to influence this - the added bonus being that there is a direct correlation to cost - savings in carbon directly relate to actual monetary savings - this is a "no brainer" to be doing these things as both the company and the environment benefit.<br /><div><br /><p>I would encourage you all to at least get the MS tool installed and see the impact you are already having as a first step and start on your co2 measuring and reducing journey.</p><p><br /></p><p><br /></p></div></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-59874260542059418032021-12-02T06:13:00.002-08:002021-12-02T06:13:54.477-08:00To reserve or not to reserve?<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-0972PZYZR_o/YajUcfdER5I/AAAAAAAAD8A/ETZc9OH4eRcn-Z3JbFMZXy9ofm-9mNc1gCNcBGAsYHQ/s640/shakespeare-g7451fb721_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="640" height="320" src="https://1.bp.blogspot.com/-0972PZYZR_o/YajUcfdER5I/AAAAAAAAD8A/ETZc9OH4eRcn-Z3JbFMZXy9ofm-9mNc1gCNcBGAsYHQ/s320/shakespeare-g7451fb721_640.jpg" width="320" /></a></div><br /> <p></p><p>To reserve or not to reserve - that is the question... ( as Shakespeare so eloquently put it).</p><p>If you're looking to save money when running applications in Azure then one of the easiest and least impactful ways is to 'reserve' resources. I'll be talking here about IaaS resources but other resources have this capability too - though savings might not be as dramatic as with IaaS.</p><p>A reservation is a commitment to buy essentially - by making that commitment you get it for a much reduced price over the normal pay as you go price - this can be over a 1 or 3 year timeframe and can attract substantial (up to 80% for long reservations on certain machine types).</p><p>Now the problem for us has always been how to effectively manage this - we have a huge volume of servers and the initial approach we followed was for each application owner to agree to a long commitment - we would then scope the reservation to that applications resource group so they were guaranteed as the ones that would get the discount. This works fine in the simple case of a few servers but just does not effectively scale to thousands of servers - there is just way too much overhead to manage.</p><p>Instead we decided to take a different tack - bulk reservations tenant wide.</p><p>This means we can't ensure who will get the reservation discount (as the discount will jump around between eligible servers) but it means as an organisation overall we get the savings without a massive management overhead.</p><p>Now the question is how many servers should we bulk reserve?</p><p>The reservations screen itself does offer some recommendation but this does not seem to take into account partially running machines (of which we have lots) to really give the best guess at how many we should have to maximize the savings. So to remedy this we built a small logic app to store some stats into a SQL database which we can then query in powerbi - this allows us to produce graphs like this:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-SpLuDlF70uM/YajMKPinm3I/AAAAAAAAD7g/O4NQDqzrBG4LRtKDCQo7iyb-R7BkTOLeQCNcBGAsYHQ/s1576/example.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="851" data-original-width="1576" height="173" src="https://1.bp.blogspot.com/-SpLuDlF70uM/YajMKPinm3I/AAAAAAAAD7g/O4NQDqzrBG4LRtKDCQo7iyb-R7BkTOLeQCNcBGAsYHQ/s320/example.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">With this data we can then show the 'minimum' number of servers of a certain size we have running within a time period - this is then a baseline number for reservations we should take out. In the example above we always have at least 225 B2ms machines running so should bulk reserve that many. With some additional maths we can probably up that number further as if we assume that a reservation saves at least 30% over PAYG costs then even if the reservation is unused 30% of the time it still saves money overall when it is used ( I just need to work out that additional bit of maths.....)</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">To give the background on what was built to create this it was actually pretty simple - we had a pre-existing 'utility' Azure SQL database in to which we added another table</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">The definition (generated from SQL mgmt studio) is this</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: left;"><b>/****** Object: Table [dbo].[runningservers] Script Date: 02/12/2021 14:47:15 ******/</b></div><div class="separator" style="clear: both; text-align: left;"><b>SET ANSI_NULLS ON</b></div><div class="separator" style="clear: both; text-align: left;"><b>GO</b></div><div class="separator" style="clear: both; text-align: left;"><b><br /></b></div><div class="separator" style="clear: both; text-align: left;"><b>SET QUOTED_IDENTIFIER ON</b></div><div class="separator" style="clear: both; text-align: left;"><b>GO</b></div><div class="separator" style="clear: both; text-align: left;"><b><br /></b></div><div class="separator" style="clear: both; text-align: left;"><b>CREATE TABLE [dbo].[runningservers](</b></div><div class="separator" style="clear: both; text-align: left;"><b><span style="white-space: pre;"> </span>[id] [int] IDENTITY(1,1) NOT NULL,</b></div><div class="separator" style="clear: both; text-align: left;"><b><span style="white-space: pre;"> </span>[rundate] [datetime] NULL,</b></div><div class="separator" style="clear: both; text-align: left;"><b><span style="white-space: pre;"> </span>[vmsize] [varchar](100) NOT NULL,</b></div><div class="separator" style="clear: both; text-align: left;"><b><span style="white-space: pre;"> </span>[counter] [int] NULL,</b></div><div class="separator" style="clear: both; text-align: left;"><b>PRIMARY KEY CLUSTERED </b></div><div class="separator" style="clear: both; text-align: left;"><b>(</b></div><div class="separator" style="clear: both; text-align: left;"><b><span style="white-space: pre;"> </span>[id] ASC</b></div><div class="separator" style="clear: both; text-align: left;"><b>)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]</b></div><div class="separator" style="clear: both; text-align: left;"><b>) ON [PRIMARY]</b></div><div class="separator" style="clear: both; text-align: left;"><b>GO</b></div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">So this is really simple - just a surrogate primary key along with 3 columns we are interested in - the date the count ran, the vmsize and the count of running machines (note that its running machines and not just existing machines - that's an important distinction)</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Once we have that empty table we just need something to populate it - here I'm just using a very simple logic app - this will perform a resource graph query against all of our estate and then dump the results in this SQL Server table.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">That logic app definition looks like this:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-xkkzoxSB9q8/YajPkLiehGI/AAAAAAAAD7o/sXfWz398It4Y4njQ4BMiMQhhZ-xpGs52gCNcBGAsYHQ/s1047/logicapp.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="851" data-original-width="1047" height="260" src="https://1.bp.blogspot.com/-xkkzoxSB9q8/YajPkLiehGI/AAAAAAAAD7o/sXfWz398It4Y4njQ4BMiMQhhZ-xpGs52gCNcBGAsYHQ/s320/logicapp.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: left;">Steps being</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">1) trigger on a timer every hour</div><div class="separator" style="clear: both; text-align: left;">2) make rest call to resource graph to retreive the counts</div><div class="separator" style="clear: both; text-align: left;">3) parse the json output that is returned</div><div class="separator" style="clear: both; text-align: left;">4) open a loop</div><div class="separator" style="clear: both; text-align: left;">5) for each 'row' in the dataset insert it into the SQL database</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Point 1 is very simple and doesn't warrant more explanation</div><div class="separator" style="clear: both; text-align: left;">Point 2 is essentially this post - <a href="http://dbaharrison.blogspot.com/2021/10/the-logic-app-and-resource-graph-query.html">The logic app and the resource graph query (dbaharrison.blogspot.com)</a></div><div class="separator" style="clear: both; text-align: left;">the only change being the query that is executed - I've included that below as it's a little long - note that it all has to be a 'single' line when pasted in to the http body element - it can't cope with newline characters</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both;"><b>{</b></div><div class="separator" style="clear: both;"><b> "query": "resources| where type =~ 'microsoft.compute/virtualMachines' |project vmSize = coalesce(tostring(properties.hardwareProfile.vmSize), '-'),subscriptionId, powerState = tostring(split(tolower(properties.extended.instanceView.powerState.code), 'powerstate/')[1]),provisioningState = tostring(properties.provisioningState) | project vmSize, status = case( provisioningState =~ 'CREATING', 'Creating', provisioningState =~ 'DELETING', 'Deleting', (provisioningState =~ 'FAILED' and isnotnull(powerState) and isnotempty(powerState)), case( powerState =~ 'RUNNING', 'Running', powerState =~ 'STOPPED', 'Stopped', powerState =~ 'DEALLOCATED', 'Stopped (deallocated)', 'Unknown' ), provisioningState =~ 'FAILED', 'Failed', (provisioningState =~ 'SUCCEEDED' and isnotnull(powerState) and isnotempty(powerState)), case( powerState =~ 'RUNNING', 'Running', powerState =~ 'STOPPED', 'Stopped', powerState =~ 'DEALLOCATED', 'Stopped (deallocated)', powerState =~ 'STARTING', 'Starting', 'Unknown' ), (provisioningState =~ 'UPDATING' and isnotnull(powerState) and isnotempty(powerState)), case( powerState =~ 'DEALLOCATING', 'Deallocating', powerState =~ 'RUNNING', 'Running', powerState =~ 'STARTING', 'Starting', powerState =~ 'STOPPING', 'Stopping', 'Updating' ), 'Unknown' ),subscriptionId | where status=='Running' |summarize count() by vmSize"</b></div><div class="separator" style="clear: both;"><b>}</b></div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Point 3 needs a slight explanation as you'll need the schema value to be able to parse the output from the previous step - this schema looks like this:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both;"><b>{</b></div><div class="separator" style="clear: both;"><b> "properties": {</b></div><div class="separator" style="clear: both;"><b> "count": {</b></div><div class="separator" style="clear: both;"><b> "type": "integer"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "data": {</b></div><div class="separator" style="clear: both;"><b> "items": {</b></div><div class="separator" style="clear: both;"><b> "properties": {</b></div><div class="separator" style="clear: both;"><b> "count_": {</b></div><div class="separator" style="clear: both;"><b> "type": "integer"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "subscriptionId": {</b></div><div class="separator" style="clear: both;"><b> "type": "string"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "vmSize": {</b></div><div class="separator" style="clear: both;"><b> "type": "string"</b></div><div class="separator" style="clear: both;"><b> }</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "required": [</b></div><div class="separator" style="clear: both;"><b> "vmSize",</b></div><div class="separator" style="clear: both;"><b> "count_"</b></div><div class="separator" style="clear: both;"><b> ],</b></div><div class="separator" style="clear: both;"><b> "type": "object"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "type": "array"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "facets": {</b></div><div class="separator" style="clear: both;"><b> "type": "array"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "resultTruncated": {</b></div><div class="separator" style="clear: both;"><b> "type": "string"</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "totalRecords": {</b></div><div class="separator" style="clear: both;"><b> "type": "integer"</b></div><div class="separator" style="clear: both;"><b> }</b></div><div class="separator" style="clear: both;"><b> },</b></div><div class="separator" style="clear: both;"><b> "type": "object"</b></div><div class="separator" style="clear: both;"><b>}</b></div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">So when pasted in the definition should look something like this:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-P1UUcePIjqE/YajRk3NlHAI/AAAAAAAAD7w/7m1-VvXpn1UUd7wzwfiPzKFTD-40yJx-ACNcBGAsYHQ/s1234/rest.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="1234" height="138" src="https://1.bp.blogspot.com/-P1UUcePIjqE/YajRk3NlHAI/AAAAAAAAD7w/7m1-VvXpn1UUd7wzwfiPzKFTD-40yJx-ACNcBGAsYHQ/s320/rest.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: left;">The final part (the inserting of the data) is done using the insert row action - when defined this will look something like this (note I derive the current time from utcnow(), the other columns come from the resource graph query):</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ySCtjm7Mwoo/YajR89mkNNI/AAAAAAAAD74/ReUBZDdP0N4A0yXi8xJDhUrp8qd-6vsxwCNcBGAsYHQ/s1284/insert.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="690" data-original-width="1284" height="172" src="https://1.bp.blogspot.com/-ySCtjm7Mwoo/YajR89mkNNI/AAAAAAAAD74/ReUBZDdP0N4A0yXi8xJDhUrp8qd-6vsxwCNcBGAsYHQ/s320/insert.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">So at this point everything is done and we insert fresh data every hour ( I know reservations are checked every 30 mins but our shutdown tags always operate on round numbers of hours - you could up the frequency to every 30 mins if need be). You could also add in the extra dimension of subscription to the query/insert if that is relevant for you.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Now all that remains is to visualize the data - you can simply do it in SQL if that suits you or you can graph it up in powerbi or whatever your weapon of choice is. I'm by no means an expert at powerbi but i could manage to get the simple graph of what i needed to do in just a few clicks.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">We're holding fire on bulk reservations for a little while to complete two other important activities which directly link into reservations:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">1) Make sure the machines autoshutdown tags are set correctly - in many cases starting/stopping machines can result in fewer running hours and even greater discount than reservations</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">2) rightsizing and right'series'ing - making sure that the machine is not too big - but also making sure it's on the latest series of machines - for example make sure Dv2 machines are moved up to Dv4 (or even v5) if possible - nothing that newer series may not have reservation options (yet)</div><div><br /></div>Once those are complete we will proceed with a bulk reservation which should result in a huge saving (as will point 1 and point 2 above).<div><br /></div><div>Once reservations are in place - we could then start graphing that too to check the reservations are aligned with the usage.</div><div><br /></div><div><br /><p><br /></p></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-71430886962873644632021-11-02T10:52:00.000-07:002021-11-02T10:52:02.862-07:00An end to end process for complete resource graph extracts<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-F-tf17knpKs/YYF6iT7LyEI/AAAAAAAAD7M/UZff5Aj1iP0sjIKE9QSg5CKaIuMaastjwCLcBGAsYHQ/s640/glue-g08de84ab2_640.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="595" height="320" src="https://1.bp.blogspot.com/-F-tf17knpKs/YYF6iT7LyEI/AAAAAAAAD7M/UZff5Aj1iP0sjIKE9QSg5CKaIuMaastjwCLcBGAsYHQ/s320/glue-g08de84ab2_640.png" width="298" /></a></div><br /><p>So this week I discovered that resource graph (by current favourite Azure tool) has a quite surprising limitation, what is this you ask?</p><p>Well if you have a reasonable size estate in Azure your resource count will very easily be in thousands - if you want to extract all of this as a report from resource graph you'll likely do this as a first pass in the resource graph console in the portal. You'll write the query and all the data is returned to the screen - everything looks good. You then decide to extract this to csv to do some more analysis in Excel - all looks good.</p><p>However what has silently failed is that not everything is extracted - in my tests the limit seemed to be 5000 items - anything above that is just not present the extract at all - but it doesn't error....</p><p>Now i couldn't find a mention of this exact limitation in the docs, however all of the various other API's into resource graph explicit mention limits and the need to make multiple fetches to return the entire dataset back.</p><p>So how to generate my nice excel extract with everything in?</p><p>Do multiple extracts in the GUI and then combine somehow?</p><p>What i actually decided to do was build something to generate a (full) report using some of the native PaaS components, this gives me the end result i want but also gives me the chance to make use of the tooling and teach myself some bits along the way.</p><p>Below is what i ended up building (still needs making a bit more user friendly I think but gets the job done), it taught me a lot of really useful stuff actually about how the components can be used and interact together. I've documented all those below and hopefully some of the explanation/tricks is useful for your use cases too.</p><p>So the basic flow of what I built is summarized in this screenshot from logic apps</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ynqr_ds0XCo/YYFk0uZ8XwI/AAAAAAAAD5k/10PVa25xJzMwP1pKDuuglzY22QMFNyPswCLcBGAsYHQ/s989/rich1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="734" data-original-width="989" height="237" src="https://1.bp.blogspot.com/-ynqr_ds0XCo/YYFk0uZ8XwI/AAAAAAAAD5k/10PVa25xJzMwP1pKDuuglzY22QMFNyPswCLcBGAsYHQ/s320/rich1.PNG" width="320" /></a></div><br /><p>To explain the 5 steps at a high level :</p><p>1) When a MS form is submitted (the form being the front end to submit the request in to my tool - I'll come on to that later)</p><p>2) Extract the specific element from the form that contains the resource graph query that I want to run</p><p>3) Call my function app code that is the main brains of the setup (and actually contains real code - albeit in PowerShell)</p><p>4) take the data and turn it into csv format</p><p>5) Email out the data as an attachment to an email</p><p><br /></p><p>Now when i created this I actually started with point 3) and the bits to call it and send the results out came later. This is the part that caused me the most frustration (and taught me the most) so this is the bit I'll start off by explaining.</p><p>So lets go through creating the function app - the basic use case it answers it to receive a resource graph query and output the results (all of them)</p><p>Now the initial stage of this is exactly the same as one of my earlier explanations <a href="http://dbaharrison.blogspot.com/2021/02/serverless-cmdb-extract-from-azure.html">Serverless CMDB extract from Azure (dbaharrison.blogspot.com)</a> - there is no point duplicating all of that - just stop at the point where I start adding in extra PowerShell modules and then continue from here on (make sure the managed identity part is done - that needs to be there to allow the function to query the data)</p><p>Now we need to create a new function - just be sure to make the trigger http and give it a name</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-pq--lCjRp9E/YYFpAGhPlyI/AAAAAAAAD5s/5X0EtcvnDuUQoFnLEYjE8mzzGYd2YammACLcBGAsYHQ/s1033/rich2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="874" data-original-width="1033" height="271" src="https://1.bp.blogspot.com/-pq--lCjRp9E/YYFpAGhPlyI/AAAAAAAAD5s/5X0EtcvnDuUQoFnLEYjE8mzzGYd2YammACLcBGAsYHQ/s320/rich2.PNG" width="320" /></a></div><br /><p>Once that's created we then need to paste in the magic code (that took way longer that it should of to develop) - I'll include that direct below and then explain some of the detail of why it's written like this.</p><p>Note there is some basic debug that can be removed and the whole thing could maybe do with a clean up - but the basics of what I need it to do are here - don't be too quick to judge :-)</p><p><br /></p><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">using namespace System.Net
<span style="color: #008800; font-style: italic;"># Input bindings are passed in via param block.</span>
<span style="color: #aa22ff; font-weight: bold;">param</span>(<span style="color: darkgoldenrod;">$Request</span>, <span style="color: darkgoldenrod;">$TriggerMetadata</span>)
<span style="color: #008800; font-style: italic;"># Write to the Azure Functions log stream.</span>
<span style="color: #aa22ff;">Write-Host</span> <span style="color: #bb4444;">"PowerShell HTTP trigger function processed a request."</span>
<span style="color: darkgoldenrod;">$endpoint</span> = <span style="color: darkgoldenrod;">$env:MSI_ENDPOINT</span>
<span style="color: darkgoldenrod;">$endpoint</span>
<span style="color: darkgoldenrod;">$secret</span> = <span style="color: darkgoldenrod;">$env:MSI_SECRET</span>
<span style="color: darkgoldenrod;">$secret</span>
<span style="color: darkgoldenrod;">$header</span> = <span style="border: 1px solid rgb(255, 0, 0);">@</span>{<span style="color: #bb4444;">'Secret'</span> = <span style="color: darkgoldenrod;">$secret</span>}
<span style="color: #008800; font-style: italic;"># Interact with query parameters or the body of the request.</span>
<span style="color: darkgoldenrod;">$passed_input</span> = <span style="color: darkgoldenrod;">$Request</span>.Body.v_input
<span style="color: darkgoldenrod;">$TokenURI</span> = <span style="color: #bb4444;">"https://management.azure.com/&api-version=2017-09-01"</span>
<span style="color: darkgoldenrod;">$authenticationResult</span> = <span style="color: #aa22ff;">Invoke-RestMethod</span> -Method Get -Headers <span style="color: darkgoldenrod;">$header</span> -Uri (<span style="color: darkgoldenrod;">$endpoint</span> +<span style="color: #bb4444;">'?resource='</span> +<span style="color: darkgoldenrod;">$TokenURI</span>)
<span style="color: darkgoldenrod;">$authenticationResult</span>
<span style="color: #008800; font-style: italic;"># Use Key Vault AuthN Token to create Request Header</span>
<span style="color: darkgoldenrod;">$requestHeader</span> = <span style="border: 1px solid rgb(255, 0, 0);">@</span>{ Authorization = <span style="color: #bb4444;">"Bearer </span><span style="color: #bb6688; font-weight: bold;">$(</span><span style="border: 1px solid rgb(255, 0, 0);">$</span><span style="color: #bb6688; font-weight: bold;">authenticationResult.access_token)</span><span style="color: #bb4444;">"</span> }
<span style="color: darkgoldenrod;">$passed_input</span>
<span style="color: #aa22ff;">write-host</span> <span style="color: #bb4444;">"$requestHeader"</span>
<span style="color: darkgoldenrod;">$currentUri</span> = <span style="color: #bb4444;">'https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-03-01'</span>
<span style="color: darkgoldenrod;">$staticUri</span> =<span style="color: #bb4444;">'https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-03-01'</span>
<span style="color: darkgoldenrod;">$moredata</span> = <span style="color: #bb4444;">"true"</span>
<span style="color: darkgoldenrod;">$content</span> = <span style="color: #aa22ff; font-weight: bold;">while</span> (<span style="color: #666666;">-not</span> <span style="color: #880000;">[string]</span><span style="border: 1px solid rgb(255, 0, 0);">::</span>IsNullOrEmpty(<span style="color: darkgoldenrod;">$moredata</span>)) {
<span style="color: #008800; font-style: italic;"># API Call</span>
<span style="color: #aa22ff;">Write-Host</span> <span style="color: #bb4444;">"`r`nQuerying $currentUri..."</span> -ForegroundColor Yellow
<span style="color: #aa22ff;">write-host</span> <span style="color: darkgoldenrod;">$passed_input</span>
<span style="color: darkgoldenrod;">$apiCall</span> = <span style="color: #aa22ff;">Invoke-WebRequest</span> -Method <span style="color: #bb4444;">"POST"</span> -Uri <span style="color: darkgoldenrod;">$currentUri</span> -ContentType <span style="color: #bb4444;">"application/json"</span> -ErrorAction Stop -Body <span style="color: darkgoldenrod;">$passed_input</span> -Headers <span style="color: darkgoldenrod;">$requestHeader</span>
<span style="color: darkgoldenrod;">$nextLink</span> = <span style="color: darkgoldenrod;">$null</span>
<span style="color: darkgoldenrod;">$currentUri</span> = <span style="color: darkgoldenrod;">$null</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: darkgoldenrod;">$apiCall</span>.Content) {
<span style="color: #008800; font-style: italic;"># Check if any data is left</span>
<span style="color: darkgoldenrod;">$counter</span> = <span style="color: darkgoldenrod;">$apiCall</span>.Content | <span style="color: #aa22ff;">ConvertFrom-Json</span> | <span style="color: #aa22ff;">Select-Object</span> <span style="color: #bb4444;">'count'</span>
<span style="color: darkgoldenrod;">$totrec</span> = <span style="color: darkgoldenrod;">$apiCall</span>.Content | <span style="color: #aa22ff;">ConvertFrom-Json</span> | <span style="color: #aa22ff;">Select-Object</span> <span style="color: #bb4444;">'totalRecords'</span>
<span style="color: #008800; font-style: italic;">#$counter</span>
<span style="color: #008800; font-style: italic;">#$totrec</span>
<span style="color: #008800; font-style: italic;">#$moredata = $null</span>
<span style="color: darkgoldenrod;">$nextLink</span> = <span style="color: darkgoldenrod;">$apiCall</span>.Content | <span style="color: #aa22ff;">ConvertFrom-Json</span> | <span style="color: #aa22ff;">Select-Object</span> <span style="color: #bb4444;">'$skipToken'</span>
<span style="color: #aa22ff; font-weight: bold;">if</span> (<span style="color: darkgoldenrod;">$nextLink</span>.<span style="color: #bb4444;">'$skipToken'</span> <span style="color: #666666;">-eq</span> <span style="color: darkgoldenrod;">$null</span>) {<span style="color: darkgoldenrod;">$moredata</span> = <span style="color: darkgoldenrod;">$null</span>}
<span style="color: darkgoldenrod;">$currentUri</span> = <span style="color: darkgoldenrod;">$staticUri</span> + <span style="color: #bb4444;">'&$skipToken='</span> + <span style="color: darkgoldenrod;">$nextLink</span>.<span style="color: #bb4444;">'$skipToken'</span>
<span style="color: #aa22ff;">write-host</span> <span style="color: #bb4444;">"$nextLink"</span>
<span style="color: #aa22ff;">Write-host</span> <span style="color: #bb4444;">"$nextLink"</span>.<span style="color: #bb4444;">'$skipToken'</span>
<span style="color: #aa22ff;">Write-host</span> <span style="color: #bb4444;">"$currentUri"</span>
<span style="color: #008800; font-style: italic;">#Write-Host "$apiCall.Content | ConvertFrom-Json"</span>
<span style="color: #aa22ff;">write-host</span> <span style="color: #bb4444;">"$counter"</span>
<span style="color: #aa22ff;">Write-Host</span> <span style="color: #bb4444;">"$totrec"</span>
<span style="color: #008800; font-style: italic;">#$apiCall.Content | ConvertFrom-Json</span>
<span style="color: darkgoldenrod;">$returner</span> = <span style="color: darkgoldenrod;">$apiCall</span>.Content | <span style="color: #aa22ff;">ConvertFrom-Json</span> | <span style="color: #aa22ff;">Select-Object</span> <span style="color: #bb4444;">'data'</span>
<span style="color: darkgoldenrod;">$returner</span>.data
<span style="color: darkgoldenrod;">$newquery</span> = <span style="color: darkgoldenrod;">$passed_input</span> |<span style="color: #aa22ff;">ConvertFrom-Json</span>
<span style="color: #aa22ff;">write-host</span> <span style="color: #bb4444;">"$newquery"</span>
<span style="color: #008800; font-style: italic;">#$newquery.options = '{"$skipToken": "' + "$nextLink".'$skipToken' + '"}"' | convertto-json</span>
<span style="color: darkgoldenrod;">$newquery</span>.options.<span style="color: #bb4444;">'$skipToken'</span> = <span style="color: darkgoldenrod;">$nextLink</span>.<span style="color: #bb4444;">'$skipToken'</span>
<span style="color: #aa22ff;">write-host</span> <span style="color: #bb4444;">"$newquery"</span>
<span style="color: darkgoldenrod;">$passed_input</span> = <span style="color: darkgoldenrod;">$newquery</span> |<span style="color: #aa22ff;">convertto-json</span>
<span style="color: #aa22ff;">write-host</span> <span style="color: darkgoldenrod;">$passed_input</span>
<span style="color: #008800; font-style: italic;">#$moredata = $null</span>
}
<span style="color: #008800; font-style: italic;">#Write-Host "$apiCall.Content | ConvertFrom-Json"</span>
}
<span style="color: darkgoldenrod;">$content</span>
<span style="color: #008800; font-style: italic;"># Associate values to output bindings by calling 'Push-OutputBinding'.</span>
<span style="color: #aa22ff;">Push-OutputBinding</span> -Name Response -Value (<span style="color: #880000;">[HttpResponseContext]</span><span style="border: 1px solid rgb(255, 0, 0);">@</span>{
StatusCode = <span style="color: #880000;">[HttpStatusCode]</span><span style="border: 1px solid rgb(255, 0, 0);">::</span>OK
Body = <span style="color: darkgoldenrod;">$content</span>
})
</pre></div>
<p>You should be able to cut and paste (and improve) that quite easily from the above.</p><p>Now I'll try and explain some of the specific steps a bit more so you can understand what's going on - I've marked a few sections on the function below and I'll explain these further - hope this is somewhat readable.....</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-wcDIO4IWaFs/YYFrGLuXHMI/AAAAAAAAD50/zCtTjXbxzso0GsrCEqciriokyyuZerWvwCLcBGAsYHQ/s994/rich3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="994" data-original-width="666" height="320" src="https://1.bp.blogspot.com/-wcDIO4IWaFs/YYFrGLuXHMI/AAAAAAAAD50/zCtTjXbxzso0GsrCEqciriokyyuZerWvwCLcBGAsYHQ/s320/rich3.PNG" width="214" /></a></div><br /><p>A) This is the code related to the managed identity that the function has and being able to make use of this. These special environment variables are created when a managed identity is attached and contain the endpoint to talk to along with the secret key required to obtain a bearer token. You'll see in C) how they are used - we make a rest request to the endpoint mentioned in the one variable, passing the secret from the other variable - the returned request then contains the bearer token that we can then make use of to authorise us against further Azure services.</p><p>B) is referencing the input variable that is passed in to the query - in my case this is called v_input and this has to be passed in the body of the http request that is submitted.</p><p>D) Is the complete nightmare piece of code - it's actually very short in the end (even shorter if you remove the noise) but it warrants an extra bit of explanation here.</p><p>First up is that the 'nextlink' type of functionality that is present in most Microsoft API's these days - however in this case IT WORKS DIFFERENTLY - aaaarghhhhh. Normally a request would return some sort of token that you then pass in subsequent requests to retrieve the next chunk of data - this is generally done by appending that token to the end of the request - so something like https://api.blah&?token=nextdatatoken and just continuing to iterate until the token returns as null and the dataset is complete.</p><p>However in this case the token is returned but is not usable - instead you have to explicitly change the query and add the token into the query options!!!!!</p><p>As an example my query may look like this initially</p><p>{</p><p> "v_input": "{\"query\": \"Resources | where type == 'microsoft.compute/virtualmachines'|project id,properties.storageProfile.osDisk.osType\", \"options\": {</p><p> \"$skipToken\": \"\" }}"</p><p>}</p><p>To get the next chunk I have to somehow alter that query to push the $skipToken I got from the first pass into it and then submit that 'new' query</p><p>Next complaint - why has the damn variable name got a $ at the front - MS are seemingly making this deliberately annoying</p><p>Next complaint (and to be honest this feels like some sort of oversight from MS here) - the nextlink functionality only starts working if you have the id column in your output!!!! What the hell is that about - in most of my queries i don't have this - and if i did include it it would likely have duplicates anyway due to the nature of the queries - why isn't this just on by default - what has the id column got to do with anything at all?</p><p>OK ranting over but this caused me a lot of frustration working out what was going on</p><p>Finally point E) - returning the complete dataset out of the function - phew! :-)</p><p><br /></p><p>Now if we come back to the overall end to end flow - lets quickly cover point 4 and point 5 from my original picture.</p><p>Point 4 is maybe the simplest link in the chain - all it does is convert the output of the function into a CSV table - all that looks like is this:</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-afgkoDAmf7k/YYFx9tJuUeI/AAAAAAAAD6A/ANwM2DSF8SkkCxC0Sr9eE7l0VjEamoqKQCLcBGAsYHQ/s1334/rich5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-afgkoDAmf7k/YYFx9tJuUeI/AAAAAAAAD6A/ANwM2DSF8SkkCxC0Sr9eE7l0VjEamoqKQCLcBGAsYHQ/s1334/rich5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a><a href="https://1.bp.blogspot.com/-hPxMGM2N2p4/YYFx9YMSBVI/AAAAAAAAD58/mOpnrThaFUUjASeSTcBJHbkVjbOE0FohQCLcBGAsYHQ/s1366/rich4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="526" data-original-width="1366" height="123" src="https://1.bp.blogspot.com/-hPxMGM2N2p4/YYFx9YMSBVI/AAAAAAAAD58/mOpnrThaFUUjASeSTcBJHbkVjbOE0FohQCLcBGAsYHQ/s320/rich4.PNG" width="320" /></a></div></div><br /><br />Point 5 is not that much more complex - just sending an email -the key part is to have the attachments part including the output from step 4. The end result is then a basic email with a csv attachment called argquery.csv<div><br /></div><div><p><br /></p><div style="text-align: center;"><img border="0" data-original-height="629" data-original-width="1334" height="151" src="https://1.bp.blogspot.com/-afgkoDAmf7k/YYFx9tJuUeI/AAAAAAAAD6A/ANwM2DSF8SkkCxC0Sr9eE7l0VjEamoqKQCLcBGAsYHQ/s320/rich5.PNG" width="320" /></div><p></p><p><br /></p><p>At the moment the code just points at a hardcoded email address (me) - but can easily be made a variable that is passed in step 1.</p><p>Talking of step 1 lets finish off by going back to the start. How can we trigger this flow - what's the front end to the user?</p><p>Well I'm not a front end guy - so any website I might try and do are just going to look like they were written in 1995 so I ruled that out pretty quickly. So what to use?</p><p>MS forms I hadn't really used - so I thought lets try that - learn something new.</p><p>So to access this go to <a href="https://forms.office.com">https://forms.office.com</a> and click the create form option - that will look something like this</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-R6l-jg2xsoA/YYF1BvoqF8I/AAAAAAAAD6M/XWzV09O_aN0piNvUvbtLdjB14cEridxlQCLcBGAsYHQ/s1167/rich6.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="781" data-original-width="1167" height="214" src="https://1.bp.blogspot.com/-R6l-jg2xsoA/YYF1BvoqF8I/AAAAAAAAD6M/XWzV09O_aN0piNvUvbtLdjB14cEridxlQCLcBGAsYHQ/s320/rich6.PNG" width="320" /></a></div><div><br /></div><div>Now I just have a single question here where the resource graph query needs to be pasted - very simple and self explanatory to do.</div><div><br /></div><div>The runtime then looks something like this ( with query pasted in)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-d_MFQ-jk5T4/YYF1vnNjyaI/AAAAAAAAD6Y/R2he43O-uB4Dlq6HQ_FBlzuTe0qH7KODACLcBGAsYHQ/s971/rich7.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="704" data-original-width="971" height="232" src="https://1.bp.blogspot.com/-d_MFQ-jk5T4/YYF1vnNjyaI/AAAAAAAAD6Y/R2he43O-uB4Dlq6HQ_FBlzuTe0qH7KODACLcBGAsYHQ/s320/rich7.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>When submitted the user then sees this screen</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-sr_WB0C9-us/YYF2BFWs7MI/AAAAAAAAD6g/YwAu2o8TbqMOCJFtlBvQ9vxefODRdlLcQCLcBGAsYHQ/s1012/rich8.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="579" data-original-width="1012" height="183" src="https://1.bp.blogspot.com/-sr_WB0C9-us/YYF2BFWs7MI/AAAAAAAAD6g/YwAu2o8TbqMOCJFtlBvQ9vxefODRdlLcQCLcBGAsYHQ/s320/rich8.PNG" width="320" /></a></div><br /><div><br /></div><div>Then within a few seconds an email is sent out with the results</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-pxMRB4z96LI/YYF2YlBtanI/AAAAAAAAD6o/JmXmZVl9QfMUjMaUQ912x2J5ArRYvoZngCLcBGAsYHQ/s501/rich9.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="251" data-original-width="501" height="160" src="https://1.bp.blogspot.com/-pxMRB4z96LI/YYF2YlBtanI/AAAAAAAAD6o/JmXmZVl9QfMUjMaUQ912x2J5ArRYvoZngCLcBGAsYHQ/s320/rich9.PNG" width="320" /></a></div><div><br /></div><div><br /></div><div>To explain how the form bit is linked to the logic app lets show the screenshots from the logic app for those parts</div><div><br /></div><div>First showing the reference to the form I created</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-VEfIsf6KCBQ/YYF5A1_Xq0I/AAAAAAAAD6w/M75qMDBOstsGlNHpHuWYTDdcqUqlf65bgCLcBGAsYHQ/s1420/rich10.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="210" data-original-width="1420" height="47" src="https://1.bp.blogspot.com/-VEfIsf6KCBQ/YYF5A1_Xq0I/AAAAAAAAD6w/M75qMDBOstsGlNHpHuWYTDdcqUqlf65bgCLcBGAsYHQ/s320/rich10.PNG" width="320" /></a></div><div><br /></div><div>Then explicitly pulling out the answer to Q1 (the only question in this case)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-XxvWRope5hw/YYF5BOHDJRI/AAAAAAAAD64/18tAUyxM7GsjAosNetcq3WCl7BMIQNtagCLcBGAsYHQ/s1349/rich11.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="234" data-original-width="1349" height="56" src="https://1.bp.blogspot.com/-XxvWRope5hw/YYF5BOHDJRI/AAAAAAAAD64/18tAUyxM7GsjAosNetcq3WCl7BMIQNtagCLcBGAsYHQ/s320/rich11.PNG" width="320" /></a></div><div><br /></div><div>Then we pass that to the argquery function app (but making sure to wrap it in a json() function call)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/--HfYGhMtZuI/YYF5AxzBHoI/AAAAAAAAD60/eThtoq6gC8klsG7hgcr8rhRex3pHDyUnwCLcBGAsYHQ/s1381/rich12.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="274" data-original-width="1381" height="63" src="https://1.bp.blogspot.com/--HfYGhMtZuI/YYF5AxzBHoI/AAAAAAAAD60/eThtoq6gC8klsG7hgcr8rhRex3pHDyUnwCLcBGAsYHQ/s320/rich12.PNG" width="320" /></a></div><div><br /></div><div>So explicitly the code view should look something like this:</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-juW9Edy7s5Y/YYF5BWrThSI/AAAAAAAAD68/W3K2bqXjTmcaKSlcSe7v8LWVcUKNOjzgQCLcBGAsYHQ/s797/rich13.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="158" data-original-width="797" height="63" src="https://1.bp.blogspot.com/-juW9Edy7s5Y/YYF5BWrThSI/AAAAAAAAD68/W3K2bqXjTmcaKSlcSe7v8LWVcUKNOjzgQCLcBGAsYHQ/s320/rich13.PNG" width="320" /></a></div><br /><div><br /></div><div>Great - the whole thing works end to end!</div><div><br /></div><div>I learnt a lot doing this (including persistence :-)) - the whole exercise was very worthwhile</div><div><br /></div><div><br /></div><div>Things still to do....</div><div><br /></div><div>1. Let the input into the form be a direct resource graph query (and not one with all the extra damn json brackets etc) - needs to be more user friendly</div><div>2. Tidy up the function code to delete the 'debug' and add better error handling</div><div>3. Use the email address that is captured automatically by the form process as the send to address of the send email step</div><div>4. Potentially do a much better front end....</div><div>5. Complain to MS about how this rest API to ARG is working :-)<br /><p><br /></p><p><br /></p><p><br /></p><p><br /></p><p><br /></p><p><br /></p></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-83285405776989333812021-10-23T13:45:00.002-07:002021-10-23T13:45:22.076-07:00The logic app and the resource graph query<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-OFV5wav1DZM/YXR0THuMLwI/AAAAAAAAD5c/kN2UkEo9iME8vRw9m06UNfPludU5o0XMwCLcBGAsYHQ/s640/sea-gdc62eeb50_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="573" data-original-width="640" height="287" src="https://1.bp.blogspot.com/-OFV5wav1DZM/YXR0THuMLwI/AAAAAAAAD5c/kN2UkEo9iME8vRw9m06UNfPludU5o0XMwCLcBGAsYHQ/s320/sea-gdc62eeb50_640.jpg" width="320" /></a></div><br /><br /><p></p><p>A nice simple example of how to utilize azure resource graph from a logic app. In this case we have a use case from some of our alerting definitions to be able to work out from a resource id value of a virtual machine if that machine is running Linux or windows (as there is some further processing based on that)</p><p>So some sort of alerting rule as created an alert against a resource id something like this</p><p>/subscriptions/subid/resourcegroups/rgname/providers/microsoft.compute/virtualmachines/servername</p><p>I can't tell from that if the machine is Linux or windows so I want to be able to a simple logic app passing that resource id and get the logic app to tell me if that's Linux or windows - this will enable some further logic down the line</p><p>The process is therefore split into these 3 simple steps:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-riTlJf7VLb8/YXQj65nG5xI/AAAAAAAAD3Q/Ex2BCtJn8zkyQq8LYmLvnsAjagNOGnFAwCLcBGAsYHQ/s534/arg1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="498" data-original-width="534" height="298" src="https://1.bp.blogspot.com/-riTlJf7VLb8/YXQj65nG5xI/AAAAAAAAD3Q/Ex2BCtJn8zkyQq8LYmLvnsAjagNOGnFAwCLcBGAsYHQ/s320/arg1.PNG" width="320" /></a></div><br /><p>The first step defines a http endpoint that can be called that takes a resourceid value as an input parameter.</p><p>This is then followed by a second http step - this is a rest api call to the resource graph service which will allow me to find out the os of the target resource.</p><p>The final step is just arbitrary here - it will just email a value of Linux/windows just to prove the thing is working as expected (the actual logic of what happens next would be something entirely different)</p><p>Lets go through each of those bit by bot so we can see how this is built.</p><p><span style="font-size: medium;"><b>Step 1 - create the thing listening for requests</b></span></p><p>So first things first lets create a new logic app, pretty basic stuff just give it a name and some other basics and away we go</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-5TaNmNIk9FU/YXRsrcvx1WI/AAAAAAAAD3Y/vc7us1-s1bAvnLB9sw3vpeMd0O2_XjmkwCLcBGAsYHQ/s923/arg2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="656" data-original-width="923" height="227" src="https://1.bp.blogspot.com/-5TaNmNIk9FU/YXRsrcvx1WI/AAAAAAAAD3Y/vc7us1-s1bAvnLB9sw3vpeMd0O2_XjmkwCLcBGAsYHQ/s320/arg2.PNG" width="320" /></a></div><div><br /></div><div>Review the settings and then create</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-zfe0-XAC2zY/YXRsrgQPvNI/AAAAAAAAD3c/6DGdi6GhPQMpkhKA8vguwPV3Bpi055wOQCLcBGAsYHQ/s611/arg3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="420" data-original-width="611" height="220" src="https://1.bp.blogspot.com/-zfe0-XAC2zY/YXRsrgQPvNI/AAAAAAAAD3c/6DGdi6GhPQMpkhKA8vguwPV3Bpi055wOQCLcBGAsYHQ/s320/arg3.PNG" width="320" /></a></div><div><br /></div><div>Once it's created then we need to define the trigger - in this basic example it's just when it receives a call via http - so I pick the option below.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-p3AZQsXcCo0/YXRsroTQBzI/AAAAAAAAD3g/EPAna-i2wtcrjsvDPXvGkZ4LUhWL8qa8ACLcBGAsYHQ/s1752/arg4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="858" data-original-width="1752" height="157" src="https://1.bp.blogspot.com/-p3AZQsXcCo0/YXRsroTQBzI/AAAAAAAAD3g/EPAna-i2wtcrjsvDPXvGkZ4LUhWL8qa8ACLcBGAsYHQ/s320/arg4.PNG" width="320" /></a></div><div><br /></div><div>I then click on that object</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-MFcF7mkfdLU/YXRssNTqMzI/AAAAAAAAD3k/gIDV2VsaoQYBTHI434k4Gdmie_AdA0kZgCLcBGAsYHQ/s1283/arg5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="447" data-original-width="1283" height="111" src="https://1.bp.blogspot.com/-MFcF7mkfdLU/YXRssNTqMzI/AAAAAAAAD3k/gIDV2VsaoQYBTHI434k4Gdmie_AdA0kZgCLcBGAsYHQ/s320/arg5.PNG" width="320" /></a></div><div><br /></div><div>Then click on the link (that's not obviously a link...) and define what my basic input will look like</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-VzSb7dxeUG0/YXRssXOalFI/AAAAAAAAD3o/jTONCXldVoMb8QWNxHpRkDZNhsp8EQ2NQCLcBGAsYHQ/s1886/arg6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="637" data-original-width="1886" height="108" src="https://1.bp.blogspot.com/-VzSb7dxeUG0/YXRssXOalFI/AAAAAAAAD3o/jTONCXldVoMb8QWNxHpRkDZNhsp8EQ2NQCLcBGAsYHQ/s320/arg6.PNG" width="320" /></a></div><div><br /></div><div>In my case I just want to pass a single parameter called resid containing the resource id - so I just define my example input like below</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-s5Vh0LIADys/YXRssXl10qI/AAAAAAAAD3s/ct2mKJR6bc46jxgsWqSnKZAaSchG7f1kgCLcBGAsYHQ/s898/arg7.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="345" data-original-width="898" height="123" src="https://1.bp.blogspot.com/-s5Vh0LIADys/YXRssXl10qI/AAAAAAAAD3s/ct2mKJR6bc46jxgsWqSnKZAaSchG7f1kgCLcBGAsYHQ/s320/arg7.PNG" width="320" /></a></div><div><br /></div><div>When I save that in then builds that in the proper json definition for me and saves me some effort.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-B3NXM_WGclA/YXRssiObjjI/AAAAAAAAD3w/W7IBX86ftCcgvFiSCXWO2-jwkOETQpgLACLcBGAsYHQ/s708/arg8.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="579" data-original-width="708" height="262" src="https://1.bp.blogspot.com/-B3NXM_WGclA/YXRssiObjjI/AAAAAAAAD3w/W7IBX86ftCcgvFiSCXWO2-jwkOETQpgLACLcBGAsYHQ/s320/arg8.PNG" width="320" /></a></div><div><br /></div><div>When I now save it the endpoint that is callable is generated - so I could call it at this point but it's not going to actually do anything (well other than reply that it got the request OK - but it actually has no further steps to do yet)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-TkAQqhs3qcs/YXRss_YA3fI/AAAAAAAAD30/tbVhUlmLGmEKc3-fBn6KqKMJF_DbxXvwgCLcBGAsYHQ/s748/arg9.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="539" data-original-width="748" height="231" src="https://1.bp.blogspot.com/-TkAQqhs3qcs/YXRss_YA3fI/AAAAAAAAD30/tbVhUlmLGmEKc3-fBn6KqKMJF_DbxXvwgCLcBGAsYHQ/s320/arg9.PNG" width="320" /></a></div><br /><p><span style="font-size: medium;"><b>Step 2 - define the resource graph bit</b></span></p><p>So next is to do the call out to resource graph - first up we want to add an action to follow after the original http request comes in.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-jaBxvP5t_xo/YXRs2kL6-bI/AAAAAAAAD34/Fuz1gBifYpkiFB9Z37iufJM40g_2itV-QCLcBGAsYHQ/s333/arg10.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="249" data-original-width="333" height="239" src="https://1.bp.blogspot.com/-jaBxvP5t_xo/YXRs2kL6-bI/AAAAAAAAD34/Fuz1gBifYpkiFB9Z37iufJM40g_2itV-QCLcBGAsYHQ/s320/arg10.PNG" width="320" /></a></div><div><br /></div><div>Here I choose http</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-GiZU1Qneiok/YXRs2v6ixEI/AAAAAAAAD38/qpJZTG0NyRUFJC5GXtP1VLXd9fRUegSmgCLcBGAsYHQ/s1478/arg11.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="572" data-original-width="1478" height="124" src="https://1.bp.blogspot.com/-GiZU1Qneiok/YXRs2v6ixEI/AAAAAAAAD38/qpJZTG0NyRUFJC5GXtP1VLXd9fRUegSmgCLcBGAsYHQ/s320/arg11.PNG" width="320" /></a></div><div><br /></div>Which then opens up this screen<div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-PQDKh68XitE/YXRs2szB69I/AAAAAAAAD4A/Y6DYoHth7sA__xgn2DOd07I2h2_KmHLMACLcBGAsYHQ/s1458/arg12.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="614" data-original-width="1458" height="135" src="https://1.bp.blogspot.com/-PQDKh68XitE/YXRs2szB69I/AAAAAAAAD4A/Y6DYoHth7sA__xgn2DOd07I2h2_KmHLMACLcBGAsYHQ/s320/arg12.PNG" width="320" /></a></div><div><br /></div><div>Now we want to populate some values here - so we need to:</div><div><br /></div><div>change the method to POST</div><div>enter the uri for resource graph which is (at the time of writing) <a href="https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-03-01">https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-03-01</a></div><div>And then the clever bit for the resource graph query - in my simple case I just return the os based on the input of a resource id - the code for that is as follows:</div><div><br /></div><div><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">{
<span style="color: #bb4444;">"query"</span><span style="border: 1px solid rgb(255, 0, 0);">:</span> <span style="color: #bb4444;">"Resources | where type == 'microsoft.compute/virtualmachines' and id =~'@{triggerBody()?['resid']}'|project properties.storageProfile.osDisk.osType"</span>
}
</pre></div>
<br /></div><div>Note the 'resid' part is then the variable that is passed through from the original call</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-aiHebDT4vUk/YXRs2-oNuSI/AAAAAAAAD4E/RRW69g2-hOgLkbGlOBl324Hq2NUT4llZwCLcBGAsYHQ/s1455/arg13.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="785" data-original-width="1455" height="173" src="https://1.bp.blogspot.com/-aiHebDT4vUk/YXRs2-oNuSI/AAAAAAAAD4E/RRW69g2-hOgLkbGlOBl324Hq2NUT4llZwCLcBGAsYHQ/s320/arg13.PNG" width="320" /></a></div><div><br /></div><div>You can see that resid is then the special variable in the screen and is obviously different from the rest of the text.</div><div><br /></div><div>Note here that I haven't included any kind of subscription filter so the query will run against everything in your tenant.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-p7lnzQnna98/YXRs3KKVIAI/AAAAAAAAD4I/URAsod6Dq5Yk5iKBOdqhn11WxKpc-ArKACLcBGAsYHQ/s758/arg14.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="653" data-original-width="758" height="276" src="https://1.bp.blogspot.com/-p7lnzQnna98/YXRs3KKVIAI/AAAAAAAAD4I/URAsod6Dq5Yk5iKBOdqhn11WxKpc-ArKACLcBGAsYHQ/s320/arg14.PNG" width="320" /></a></div><div><br /></div><div>If you run the code now you'll find that it fails - as it's not authorized to run against resource graph yet - it's essentially an anonymous request that will fail. To do this in the simplest way I'm going to use a managed identity - i.e. the logic app itself gets an identity and we assign that permissions to be able to run resource graph. First step in that is to enable a managed identity in this screen:</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-u4ISqx2BN5U/YXRs3T2x6oI/AAAAAAAAD4M/xfyMRGPAQ8oUsJCyE1_ijoDdrf5gtg3fQCLcBGAsYHQ/s1111/arg15.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="705" data-original-width="1111" height="203" src="https://1.bp.blogspot.com/-u4ISqx2BN5U/YXRs3T2x6oI/AAAAAAAAD4M/xfyMRGPAQ8oUsJCyE1_ijoDdrf5gtg3fQCLcBGAsYHQ/s320/arg15.PNG" width="320" /></a></div><div><br /></div>We click it on and we see the message below</div><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-sCDjhI_vouU/YXRs3iuQIPI/AAAAAAAAD4Q/i_HfBLGdGdQJq7OCHNg7KrL1r7JFZa3wwCLcBGAsYHQ/s1171/arg16.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="296" data-original-width="1171" height="81" src="https://1.bp.blogspot.com/-sCDjhI_vouU/YXRs3iuQIPI/AAAAAAAAD4Q/i_HfBLGdGdQJq7OCHNg7KrL1r7JFZa3wwCLcBGAsYHQ/s320/arg16.PNG" width="320" /></a></div><div><br /></div><div>Now when we save the identity is created in Azure AD.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-n1qx4weMD7U/YXRs30WQcrI/AAAAAAAAD4U/nt9HGAXofP8bRkg9TH71_gkFq0bCOCY_ACLcBGAsYHQ/s630/arg17.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="630" height="254" src="https://1.bp.blogspot.com/-n1qx4weMD7U/YXRs30WQcrI/AAAAAAAAD4U/nt9HGAXofP8bRkg9TH71_gkFq0bCOCY_ACLcBGAsYHQ/s320/arg17.PNG" width="320" /></a></div><div><br /></div><div>Now we just have to assign that identity permissions. In my case I want to give read access to everything in the entire tenant. To do that I go to my tenant root group in management groups and assign it the reader role - the final screen of that is shown below. I won't show the whole process as it's fairly straightforward and may vary depending on if you want to just grant access to only some subscriptions.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-yJwcAqT8AMU/YXRs32CvcyI/AAAAAAAAD4Y/x3HYlfN8mXMJPi0qQivegz8TxyyTjhnDQCLcBGAsYHQ/s1206/arg18.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="515" data-original-width="1206" height="137" src="https://1.bp.blogspot.com/-yJwcAqT8AMU/YXRs32CvcyI/AAAAAAAAD4Y/x3HYlfN8mXMJPi0qQivegz8TxyyTjhnDQCLcBGAsYHQ/s320/arg18.PNG" width="320" /></a></div><div><br /></div><div>Once that's in place we just need to make use of it - so we choose the add new parameter option and choose the authentication option.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ZzsHa3H74qY/YXRs39RC3tI/AAAAAAAAD4c/0sHEtBAEYp0BwLplxZN_F8_w6E1y8SFFACLcBGAsYHQ/s814/arg19.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="341" data-original-width="814" height="134" src="https://1.bp.blogspot.com/-ZzsHa3H74qY/YXRs39RC3tI/AAAAAAAAD4c/0sHEtBAEYp0BwLplxZN_F8_w6E1y8SFFACLcBGAsYHQ/s320/arg19.PNG" width="320" /></a></div><p>From the drop down choose 'managed identity' - i.e. the thing we created a minute ago</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-1i83Z9vrFRQ/YXRtBrrq6-I/AAAAAAAAD4o/rJJIGgA08A8wlAhAT14tCy-kiTrEgsybQCLcBGAsYHQ/s786/arg20.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="367" data-original-width="786" height="149" src="https://1.bp.blogspot.com/-1i83Z9vrFRQ/YXRtBrrq6-I/AAAAAAAAD4o/rJJIGgA08A8wlAhAT14tCy-kiTrEgsybQCLcBGAsYHQ/s320/arg20.PNG" width="320" /></a></div><div><br /></div><div>Now things should work - so we save it and then run it. Note to choose the with payload option so we can pass the resource id value in</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-k1AANtfaZnc/YXRtBnzdFUI/AAAAAAAAD4w/tLcVmpSvZfgPQm4F47jyAqZOmQwsVLbzgCLcBGAsYHQ/s549/arg21.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="159" data-original-width="549" height="93" src="https://1.bp.blogspot.com/-k1AANtfaZnc/YXRtBnzdFUI/AAAAAAAAD4w/tLcVmpSvZfgPQm4F47jyAqZOmQwsVLbzgCLcBGAsYHQ/s320/arg21.PNG" width="320" /></a></div><div><br /></div><div><br /></div><div>The screen then pops open and we just need to pass in the value in the following format:</div><div><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">{
<span style="color: #bb4444;">"resid"</span> <span style="border: 1px solid rgb(255, 0, 0);">:</span> <span style="color: #bb4444;">"full-res-id-here-with-slashes////"</span>
}
</pre></div>
<br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-KnX8nPCao_k/YXRtBnhDmkI/AAAAAAAAD4s/VKLBcYzN2R4WhJvbTxjAcxnLiNGLwkb_ACLcBGAsYHQ/s738/arg22.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="738" data-original-width="567" height="320" src="https://1.bp.blogspot.com/-KnX8nPCao_k/YXRtBnhDmkI/AAAAAAAAD4s/VKLBcYzN2R4WhJvbTxjAcxnLiNGLwkb_ACLcBGAsYHQ/s320/arg22.PNG" width="246" /></a></div><div><br /></div><div>Now when it runs and we navigate to look at the output of the run - we can see the returned value of 'Windows' in the screenshot below.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-LyDaHrbEEIo/YXRtCaL4olI/AAAAAAAAD40/9H92ZqAj5nMFfjaiGk0BhrJ8Ge73QdbQwCLcBGAsYHQ/s1872/arg23.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="694" data-original-width="1872" height="119" src="https://1.bp.blogspot.com/-LyDaHrbEEIo/YXRtCaL4olI/AAAAAAAAD40/9H92ZqAj5nMFfjaiGk0BhrJ8Ge73QdbQwCLcBGAsYHQ/s320/arg23.PNG" width="320" /></a></div><div><br /></div><div><span style="font-size: medium;"><b>Step 3 - email it</b></span></div><div><br /></div><div>To complete things off and show how we reference the returned value I add a simple send email step as the last action</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-6TcxiMWdY4M/YXRtC59BbsI/AAAAAAAAD44/H9xo6QVIkiAivVbh5GlMmcVeUfbPwBQwACLcBGAsYHQ/s1385/arg24.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="719" data-original-width="1385" height="166" src="https://1.bp.blogspot.com/-6TcxiMWdY4M/YXRtC59BbsI/AAAAAAAAD44/H9xo6QVIkiAivVbh5GlMmcVeUfbPwBQwACLcBGAsYHQ/s320/arg24.PNG" width="320" /></a></div><div><br /></div><div>The only clever bit here is the syntax for referencing the variable - that looks like this:</div><div><br /></div><div><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">body(<span style="color: #bb4444;">'HTTP'</span>)?[<span style="color: #bb4444;">'data'</span>][0]?[<span style="color: #bb4444;">'properties_storageProfile_osDisk_osType'</span>]
</pre></div>
</div><div><br /></div><div>Which if you paste it in then looks like this on screen</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-fi5gHYvEJMo/YXRtDU3o9-I/AAAAAAAAD48/Rs_U2Af362w2_WfXtoVCd7Oo1g9kp0dmQCLcBGAsYHQ/s577/arg25.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="247" data-original-width="577" height="137" src="https://1.bp.blogspot.com/-fi5gHYvEJMo/YXRtDU3o9-I/AAAAAAAAD48/Rs_U2Af362w2_WfXtoVCd7Oo1g9kp0dmQCLcBGAsYHQ/s320/arg25.PNG" width="320" /></a></div><div><br /></div>If i now trigger it again with the payload i get a nice email through in my inbox looking like this:<br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-azemO_TJqFY/YXRtD1Z6riI/AAAAAAAAD5A/yivOxFu1n5Q9ewsTorO5Ts7vViaJCIUXACLcBGAsYHQ/s467/arg26.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="385" data-original-width="467" height="264" src="https://1.bp.blogspot.com/-azemO_TJqFY/YXRtD1Z6riI/AAAAAAAAD5A/yivOxFu1n5Q9ewsTorO5Ts7vViaJCIUXACLcBGAsYHQ/s320/arg26.PNG" width="320" /></a></div><p><br /></p><p>And that's it job done - a nice basic example to show you how resource graph can be called - hope that's useful!</p><p><br /></p></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com5tag:blogger.com,1999:blog-302298286928742422.post-11764716529468659022021-10-23T01:35:00.003-07:002021-10-23T01:35:43.155-07:00Azure resource graph - extractjson example<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-isFlVen14pw/YXPJIXRsX7I/AAAAAAAAD3I/wWYsVukWMC0PMzcs54Hh9lpmGKa9RaVBQCLcBGAsYHQ/s1280/brown-paperbags-g170aa9bcd_1280.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1280" data-original-width="1203" height="320" src="https://1.bp.blogspot.com/-isFlVen14pw/YXPJIXRsX7I/AAAAAAAAD3I/wWYsVukWMC0PMzcs54Hh9lpmGKa9RaVBQCLcBGAsYHQ/s320/brown-paperbags-g170aa9bcd_1280.png" width="301" /></a></div><br /> <p></p><p><br /></p><p>After another long gap I'm finally putting pen to paper (well fingers to keyboard). This is just a short note to hope fully get me back in the saddle. This one is on resource graph query - one of the most useful additions to the Azure space in recent times.</p><p>In this example i just want to do a simple report to do the following:</p><p>Show me a complete list of all resource groups, including a specific tag value, the subscription name as well as the chain of management group hierarchy the resource group is in.</p><p>The first part of that is simple and doesn't really warrant even writing down - however the second part needs a little bit of extra work - it's still fairly simple but there are not too many examples of how to do this so hopefully this is useful for someone.</p><p>To create the report the data is just held in 2 resource graph locations (well actually it's the same one twice...)</p><p>I'll post the query here first and then explain the 'tricky' parts afterwards.</p><p></p><p>xx</p><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">resourcecontainers
|where type== <span style="color: #bb4444;">"microsoft.resources/subscriptions/resourcegroups"</span>
| project name,sub=toupper(subscriptionId),coalesce(tags.Owner,tags.owner),coalesce(tags.Owner_Email,tags.Owner_email,tags.owner_email),tags.EAM_ID
|join kind=leftouter (ResourceContainers | where type==<span style="color: #bb4444;">'microsoft.resources/subscriptions'</span>
| project SubName=name,sub=toupper(subscriptionId),parent=tostring(properties.managementGroupAncestorsChain)) on <span style="color: darkgoldenrod;">$left</span>.sub==<span style="color: darkgoldenrod;">$right</span>.sub
| project-away sub1
|extend parent1=extractjson(<span style="color: #bb4444;">"$.[0].displayName"</span>,parent)
|extend parent2=extractjson(<span style="color: #bb4444;">"$.[1].displayName"</span>,parent)
|extend parent3=extractjson(<span style="color: #bb4444;">"$.[2].displayName"</span>,parent)
|project-away parent<span> </span>
</pre></div>
<p>So to pick out the points of note:</p><p>1) The 'coalesce' function returns the first non null value from a series of inputs - this is very useful as for our tags there is some mixed case of the tag names (which needs correcting) and resource graph is case sensitive - using coaesce with the 3 variations of tag name means we will always have a value displayed.</p><p>2) You'll notice i have toupper() around subscription id's - again this is due to case sensitivity - without this the data won't match - this is a major gotcha when working with resource graph....</p><p>3) And this is the bit that was new to me the 'properties.managementGroupAncestorsChain' value of a subscription object is a json object with an embedded array - this then becomes tricky to access data out of. So in this case i explicitly make it a string with tostring() (this has to be done to avoid an error later on) and name that string 'parent'. Once i have that string I then use the extractjson function to pull values out of it. In the case above the syntax $.[0].displayName is saying :</p><p>Start from the root ($) , for the first element in the array ([0]) show me the displayName value - in this case the immediate parent management group of the subscription.</p><p>I then repeat that for the second and third element in the array to give the grand parent and great grand parent values.</p><p>Once you've got your head round the syntax it's actually quite easy.</p><p>There you go - short and sweet - hope that's useful....</p><p>Picture is of a 'take out' (like extract the json out is take out) - that was obvious right?</p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-35876371477112165492021-05-13T04:06:00.000-07:002021-05-13T04:06:13.051-07:00What's locking it out?<p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-UVHIu5vJpL4/YJ0H-7cYlUI/AAAAAAAAD0I/8fMmso23ceMTGVltuesWiZkAhmq9nq-1QCLcBGAsYHQ/s640/connection-4805610_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="640" height="320" src="https://1.bp.blogspot.com/-UVHIu5vJpL4/YJ0H-7cYlUI/AAAAAAAAD0I/8fMmso23ceMTGVltuesWiZkAhmq9nq-1QCLcBGAsYHQ/s320/connection-4805610_640.jpg" /></a></div><br /><p><br /></p><p>This is a post pulling together a few elements/features of Azure to demonstrate how to solve an issue, even if the core use case does not affect you the way the problem is solved can be applied to loads of other situations.</p><p>In this case my problem being that my AADDS account is getting locked out (for the unfamiliar AADDS is Microsoft's 'managed' active directory service) - so essentially something is trying to authenticate against AD with wrong password - this is causing the account to be locked out - but where is this coming from?</p><p>Normally in traditional AD this is easy to find out - you can just check event logs - however in the managed AADDS you have no direct access to the domain controllers and you can't even use event viewer to connect to the remote managed domain controller machine - that is also not allowed - so how can we find this out?</p><p>Well the first step is to enable the AADDS PaaS service to gather the logs and then send it to somewhere we can access.</p><p>Like many other PaaS services this is enabled from the diagnostics blade of the service - in the screenshot below you can see that screen for AADDS. In my case I'm actually sending the data to three distinct places - a storage account, a log analytics workspace and an event hub. There are reasons for all of these - but for this particular use case the only one I actually need is the log analytics one.</p><p>I then choose to send all the available data sources to this location (the list of available data you can see displayed at the bottom of the screenshot)</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-trhssKGe9BM/YJ0AFssRpSI/AAAAAAAADzY/N2bpSl6dBXAEYmDPDbM0qJRD-7Dh_EH8wCLcBGAsYHQ/s1893/threeCapture.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="819" data-original-width="1893" src="https://1.bp.blogspot.com/-trhssKGe9BM/YJ0AFssRpSI/AAAAAAAADzY/N2bpSl6dBXAEYmDPDbM0qJRD-7Dh_EH8wCLcBGAsYHQ/s320/threeCapture.PNG" width="320" /></a></div><br /><p></p><p>Now for those of you not familiar with it log analytics is a log ingestion and querying platform - along the lines of Splunk or the ELK stack. Basically you send a whole load of log info into it - you can then query those logs using a SQL like language called 'Kusto' (or KQL).</p><p>So for our use case it enables us to search for lockout records and then display the details of those to identify where it's coming from.</p><p>If we browse to our log analytics workspace - we can see that by activating the above diagnostics Azure has created some new 'tables' (I call them tables as I'm not sure what the correct term is and my background is in databases)</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-JLmPZqE5EPg/YJ0AFh-0FyI/AAAAAAAADzc/ygZGf-xZJikczl-6-kAt9TGe2ph3Y_QjgCLcBGAsYHQ/s959/threeCapture2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="856" data-original-width="959" src="https://1.bp.blogspot.com/-JLmPZqE5EPg/YJ0AFh-0FyI/AAAAAAAADzc/ygZGf-xZJikczl-6-kAt9TGe2ph3Y_QjgCLcBGAsYHQ/s320/threeCapture2.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">Now we have those tables we just need to write a query to find the lockout records - you can see that in the screenshot below</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ehDiV-UeJvY/YJ0AFvO17FI/AAAAAAAADzg/syKPL_4ltS0afjbN3zFYs20HYut8YNgdgCLcBGAsYHQ/s1902/threeCapture3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="624" data-original-width="1902" src="https://1.bp.blogspot.com/-ehDiV-UeJvY/YJ0AFvO17FI/AAAAAAAADzg/syKPL_4ltS0afjbN3zFYs20HYut8YNgdgCLcBGAsYHQ/s320/threeCapture3.PNG" width="320" /></a></div><div><br /></div><div><br /></div><div>I've also pasted the query text here to make it easy to copy</div><div><br /><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">AADDomainServicesAccountManagement
| where TimeGenerated >= ago(1d)
| where OperationName has <span style="color: #bb4444;">"4740"</span>
| parse ResultDescription with * <span style="color: #bb4444;">"Account Name:"</span> AccountName <span style="color: #bb4444;">"Account Name:"</span> id <span style="color: #bb4444;">"Additional Information:"</span> dummy <span style="color: #bb4444;">"Caller Computer Name:"</span> device
| project TimeGenerated, id, device, ResultDescription
| sort by TimeGenerated desc
</pre></div>
</div><div><br /></div><div>That query I think is fairly clear (even if you don't know Kusto), it's looking at the data for the last 1 day where a lockout has occurred (id 4740) and then showing the results. The only confusing line is the one starting with parse - that is taking the resultdescription 'column' (which has a lot of formatted text) and pulling out various parts of that - to give us the id and the computer it's coming from. This took me a while to get right and now I come to write this up I find it hard to explain myself now I'm not 'in the moment' of writing it..... Anyway it does some magic to pull out the bits of data we need :-)</div><div><br /></div><div>So at this point we have an easy query to find the lockouts - but how to make this available to a wider audience? We'll we have a few options:</div><div><br /></div><div>1) Share the query round with everyone</div><div>2) Put the query on a dashboard and share that</div><div>3) Save the query as Mquery and publish via powerbi</div><div>4) Stick it in a workbook</div><div>5) etc etc (sure there are lots of other ways to do this).</div><div><br /></div><div>What I'm going to do is go with option 4 (the workbook) - I really like this feature and I don't think it's getting the love it deserves :-)</div><div><br /></div><div>So what I'll do is create a new workbook with a parameter (the id to check) and then display the results of that in the workbook</div><div><br /></div><div>The end result will look like the screen below (which can be directly linked to) - users can enter an id value at the top - the Kusto query will then run in the background and display the records below</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-2NilJrDKliA/YJ0AGd9OumI/AAAAAAAADzk/YtlTij5ANLI_XDPjGr952gwzgEXqwVw1gCLcBGAsYHQ/s1117/threeCapture4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="323" data-original-width="1117" src="https://1.bp.blogspot.com/-2NilJrDKliA/YJ0AGd9OumI/AAAAAAAADzk/YtlTij5ANLI_XDPjGr952gwzgEXqwVw1gCLcBGAsYHQ/s320/threeCapture4.PNG" width="320" /></a></div><div><br /></div><div>I won't go through step by step building the workbook but just summarize the two main things you need to create:</div><div><br /></div><div>1) The parameter the user will pass in</div><div><br /></div><div>So choose the parameter option from the add menu and define it as below - it's very simple</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-loanAXwWoTs/YJ0AGsy6jyI/AAAAAAAADzo/jDC-PbBCunc8HcfyLN3ogmGh0Cnt7AJlgCLcBGAsYHQ/s1441/threeCapture5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="721" data-original-width="1441" src="https://1.bp.blogspot.com/-loanAXwWoTs/YJ0AGsy6jyI/AAAAAAAADzo/jDC-PbBCunc8HcfyLN3ogmGh0Cnt7AJlgCLcBGAsYHQ/s320/threeCapture5.PNG" width="320" /></a></div><div><br /></div><div>2) Define the query to run</div><div><br /></div><div>Next we add a 'query', set the data source to logs, resource type to log analytics and the workspace to be the one we were just querying above - then we just need to paste in the query exactly as it appears above but just with one additional line to pick up the parameter the user will enter (note the format in squiggly braces below)</div><div><br /></div><div><br /></div><div><!--HTML generated using hilite.me--><div style="background: rgb(248, 248, 248); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">AADDomainServicesAccountManagement
| where TimeGenerated >= ago(1d)
| where OperationName has <span style="color: #bb4444;">"4740"</span>
| where ResultDescription contains <span style="color: #bb4444;">"{ID}"</span>
| parse ResultDescription with * <span style="color: #bb4444;">"Account Name:"</span> AccountName <span style="color: #bb4444;">"Account Name:"</span> id <span style="color: #bb4444;">"Additional Information:"</span> dummy <span style="color: #bb4444;">"Caller Computer Name:"</span> device
|project TimeGenerated,id,device,ResultDescription |sort by TimeGenerated desc
</pre></div>
</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-_h3NEKsCLW8/YJ0AGr0bM5I/AAAAAAAADzs/w8uEBv6xLgMJjBOypW-FFwDP0Kg2jm-IgCLcBGAsYHQ/s1724/threeCapture6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="586" data-original-width="1724" src="https://1.bp.blogspot.com/-_h3NEKsCLW8/YJ0AGr0bM5I/AAAAAAAADzs/w8uEBv6xLgMJjBOypW-FFwDP0Kg2jm-IgCLcBGAsYHQ/s320/threeCapture6.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>And that's it - when the workbook is run user enters a user id that has an issue and the query will run displaying where the lockout is being triggered from. The complexity of how to do all that is hidden and the user just has a simple interface to use.<div><br /></div><div>Enjoy :-)<br /><br /><p><br /></p></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com1tag:blogger.com,1999:blog-302298286928742422.post-5899000568366318032021-03-21T14:51:00.004-07:002021-03-22T01:13:26.629-07:00Using Azure to check a website is up - part 2 (internal website)<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-LJ1kJvGIpek/YFe_y_Hls2I/AAAAAAAADy4/-3U7gX2IiOsATHVat3iaN61aFot3pUnPACLcBGAsYHQ/s640/stormtrooper-1995015_640.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" src="https://1.bp.blogspot.com/-LJ1kJvGIpek/YFe_y_Hls2I/AAAAAAAADy4/-3U7gX2IiOsATHVat3iaN61aFot3pUnPACLcBGAsYHQ/s320/stormtrooper-1995015_640.jpg" width="320" /></a></div><br /><p>So following on in quick succession (before I forget what I did) - here are the steps to set up monitoring for an internal website using application insights. The catch here is that you cant do it directly in app insights - instead you have to write something that will exist in your private networking space that will do the test and send the results to the app insights workspace.</p><p>In my case I'm sticking with function apps and powershell as for me at least this is the easiest option here.</p><p>The actual powershell code doing the test I've 'borrowed' from Niels from his excellent post here : <a href="https://swimburger.net/blog/azure/run-availability-tests-using-powershell-and-azure-application-insights">https://swimburger.net/blog/azure/run-availability-tests-using-powershell-and-azure-application-insights</a></p><p>The extra bits I've done are to wrap this in a function app (so its serverless) and deploy that into an 'App Service Environment' (ASE) so it has integration into our private address space.</p><p>Niels does a great job of explaining his powershell so I'm not going to even attempt to explain any of that - just refer to his post for more info.</p><p>So lets build the function and set up the test</p><p>First up we click create new function app - give it a name and choose powershell as the language.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-omLiRiomM0k/YFe4_NVFgMI/AAAAAAAADxM/prcpsWOUywcKPpyAiGrcCLUg9Yaj3F5UgCLcBGAsYHQ/s1250/int1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="909" data-original-width="1250" src="https://1.bp.blogspot.com/-omLiRiomM0k/YFe4_NVFgMI/AAAAAAAADxM/prcpsWOUywcKPpyAiGrcCLUg9Yaj3F5UgCLcBGAsYHQ/s320/int1.PNG" width="320" /></a></div><div><br /></div><div>Then we do one of the key steps - when we choose a region we actually have to scroll right to the top and pick our ASE (this post assumes you already have one - if you don't you'll have to sort that part out first). I personally find this screen a little confusing - feels like there should be a radio button for ASE here - putting it in the region drop down feels like its hidden to me.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-9nnisBDoZ0g/YFe4_XeCf2I/AAAAAAAADxQ/OtlmARZKMEgfFSqjp7kIAgOna6zlvDTPwCLcBGAsYHQ/s934/int2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="762" data-original-width="934" src="https://1.bp.blogspot.com/-9nnisBDoZ0g/YFe4_XeCf2I/AAAAAAAADxQ/OtlmARZKMEgfFSqjp7kIAgOna6zlvDTPwCLcBGAsYHQ/s320/int2.PNG" width="320" /></a></div><div><br /></div><div>After we pick the ASE - we can see the public azurewebsites.net name is changed to the internal domain associated with our ASE</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-u9udroMSuEU/YFe4_DR1ukI/AAAAAAAADxU/R1xcRfqa51IqFpYDdYhAoC298PfwA8RlgCLcBGAsYHQ/s956/int3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="350" data-original-width="956" src="https://1.bp.blogspot.com/-u9udroMSuEU/YFe4_DR1ukI/AAAAAAAADxU/R1xcRfqa51IqFpYDdYhAoC298PfwA8RlgCLcBGAsYHQ/s320/int3.PNG" width="320" /></a></div><div><br /></div><div>We then carry on to the next screen and choose the final couple of options</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-gUIIxbF0_BA/YFe5AHT9c8I/AAAAAAAADxY/yHvGoey-Q6sbxBA5wjRXkEThLqjfRDOUQCLcBGAsYHQ/s1122/int4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="926" data-original-width="1122" src="https://1.bp.blogspot.com/-gUIIxbF0_BA/YFe5AHT9c8I/AAAAAAAADxY/yHvGoey-Q6sbxBA5wjRXkEThLqjfRDOUQCLcBGAsYHQ/s320/int4.PNG" width="320" /></a></div><div><br /></div><div>Pick if we want app insights for this function - now here it is just talking about enabling app insights for the function itself - this doesn't have to be the place we will send data to later (in fact in this example it is to keep it simple but they are different things)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-IxKvETS7TZY/YFe5AT7WNzI/AAAAAAAADxc/RSQ2TVGhbME2B-RWK6pTqvWe8hFnZF7DQCLcBGAsYHQ/s1111/int5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="612" data-original-width="1111" src="https://1.bp.blogspot.com/-IxKvETS7TZY/YFe5AT7WNzI/AAAAAAAADxc/RSQ2TVGhbME2B-RWK6pTqvWe8hFnZF7DQCLcBGAsYHQ/s320/int5.PNG" width="320" /></a></div><div><br /></div><div>Then we get a summary and go ahead and create it</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-zhZfng4QX_M/YFe5AawxyhI/AAAAAAAADxg/k4BKo3b_xbcMzL2Kj7F1-9AjQFI0n_4rACLcBGAsYHQ/s825/int6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="825" data-original-width="812" height="320" src="https://1.bp.blogspot.com/-zhZfng4QX_M/YFe5AawxyhI/AAAAAAAADxg/k4BKo3b_xbcMzL2Kj7F1-9AjQFI0n_4rACLcBGAsYHQ/s320/int6.PNG" /></a></div><div><br /></div><div>Once create we need to go and locate the internal ip address it's selected for out function app - you can find this in the screen below</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-xFug0ngumHs/YFe5AqKC0PI/AAAAAAAADxk/euS8vsJjTAclUzry7eB840ZTAoIpjtebwCLcBGAsYHQ/s1051/int7.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="816" data-original-width="1051" src="https://1.bp.blogspot.com/-xFug0ngumHs/YFe5AqKC0PI/AAAAAAAADxk/euS8vsJjTAclUzry7eB840ZTAoIpjtebwCLcBGAsYHQ/s320/int7.PNG" width="320" /></a></div><div><br /></div><div>We then have to add 2 records to dns - one for the normal function address and one for the 'scm' (debug) version of the site.</div><div><br /></div><div>So in my case i add (same ip in both cases)</div><div><br /></div><div>internalwebtest.mydomain.com 10.x.x.x</div><div>internalwebtest.mydomain.com 10.x.x.x</div><div><br /></div><div>Screen shot below shows adding it for the main address.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-GG_3BDubx20/YFe5Aw30wvI/AAAAAAAADxo/gaYfICM8yEkwbY3eDWPWm9mUKdT_9aUHgCLcBGAsYHQ/s1526/int8.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="553" data-original-width="1526" src="https://1.bp.blogspot.com/-GG_3BDubx20/YFe5Aw30wvI/AAAAAAAADxo/gaYfICM8yEkwbY3eDWPWm9mUKdT_9aUHgCLcBGAsYHQ/s320/int8.PNG" width="320" /></a></div><div><br /></div><div>Now that's all set up i now need to create the actual function - so I navigate to that blade and add a new one on a timer trigger set to run every 1 minute as you can see below.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-30cjPZcXLK8/YFe5AzTJoaI/AAAAAAAADxs/je0RUXB01ecDTOa0qbMnI0dK8oW5y8SJgCLcBGAsYHQ/s1874/int9.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="827" data-original-width="1874" src="https://1.bp.blogspot.com/-30cjPZcXLK8/YFe5AzTJoaI/AAAAAAAADxs/je0RUXB01ecDTOa0qbMnI0dK8oW5y8SJgCLcBGAsYHQ/s320/int9.PNG" width="320" /></a></div><div><br /></div><div>Once that's added I then paste in Niels' code - I 've included it here just for completeness but it's essentially unchanged from what Niels posted (other than the function app extras). Note that the app insights workspace it's sending data to is the one the function is pointing at - it picks this directly from the env variable defined in the default function.</div><!-- HTML generated using hilite.me --><div style="background: #272822; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%"><span style="color: #f92672">#</span> <span style="color: #66d9ef">Input</span> <span style="color: #f8f8f2">bindings</span> <span style="color: #66d9ef">are</span> <span style="color: #f8f8f2">passed</span> <span style="color: #66d9ef">in</span> <span style="color: #f8f8f2">via</span> <span style="color: #f8f8f2">param</span> <span style="color: #f8f8f2">block.</span>
<span style="color: #f8f8f2">param(</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Timer)</span>
<span style="color: #f92672">#</span> <span style="color: #66d9ef">Get</span> <span style="color: #f8f8f2">the</span> <span style="color: #66d9ef">current</span> <span style="color: #f8f8f2">universal</span> <span style="color: #f8f8f2">time</span> <span style="color: #66d9ef">in</span> <span style="color: #f8f8f2">the</span> <span style="color: #66d9ef">default</span> <span style="color: #f8f8f2">string</span> <span style="color: #f8f8f2">format.</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">currentUTCtime</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">(</span><span style="color: #66d9ef">Get</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">Date).ToUniversalTime()</span>
<span style="color: #f92672">#</span> <span style="color: #f8f8f2">The</span> <span style="color: #e6db74">'IsPastDue'</span> <span style="color: #f8f8f2">property</span> <span style="color: #66d9ef">is</span> <span style="color: #e6db74">'true'</span> <span style="color: #66d9ef">when</span> <span style="color: #f8f8f2">the</span> <span style="color: #66d9ef">current</span> <span style="color: #66d9ef">function</span> <span style="color: #f8f8f2">invocation</span> <span style="color: #66d9ef">is</span> <span style="color: #f8f8f2">later</span> <span style="color: #66d9ef">than</span> <span style="color: #f8f8f2">scheduled.</span>
<span style="color: #f8f8f2">if</span> <span style="color: #f8f8f2">(</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Timer.IsPastDue)</span> <span style="color: #960050; background-color: #1e0010">{</span>
<span style="color: #66d9ef">Write</span><span style="color: #f92672">-</span><span style="color: #66d9ef">Host</span> <span style="color: #e6db74">"PowerShell timer is running late!"</span>
<span style="color: #960050; background-color: #1e0010">}</span>
<span style="color: #f92672">#</span> <span style="color: #66d9ef">Write</span> <span style="color: #f8f8f2">an</span> <span style="color: #f8f8f2">information</span> <span style="color: #f8f8f2">log</span> <span style="color: #66d9ef">with</span> <span style="color: #f8f8f2">the</span> <span style="color: #66d9ef">current</span> <span style="color: #f8f8f2">time.</span>
<span style="color: #66d9ef">Write</span><span style="color: #f92672">-</span><span style="color: #66d9ef">Host</span> <span style="color: #e6db74">"PowerShell timer trigger function ran! TIME: $currentUTCtime"</span>
<span style="color: #f92672">#</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">MyInvocation.MyCommand.Path</span> <span style="color: #f92672">|</span> <span style="color: #f8f8f2">Split</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">Path</span> <span style="color: #f92672">|</span> <span style="color: #f8f8f2">Push</span><span style="color: #f92672">-</span><span style="color: #66d9ef">Location</span><span style="color: #f8f8f2">;</span>
<span style="color: #f92672">#</span> <span style="color: #66d9ef">Update</span> <span style="color: #f8f8f2">the</span> <span style="color: #f8f8f2">path</span> <span style="color: #f8f8f2">if</span> <span style="color: #f8f8f2">you</span> <span style="color: #f8f8f2">install</span> <span style="color: #f8f8f2">a</span> <span style="color: #f8f8f2">different</span> <span style="color: #66d9ef">version</span> <span style="color: #66d9ef">of</span> <span style="color: #f8f8f2">AI</span> <span style="color: #f8f8f2">NuGet</span> <span style="color: #f8f8f2">package</span>
<span style="color: #f92672">#</span><span style="color: #66d9ef">Add</span><span style="color: #f92672">-</span><span style="color: #66d9ef">Type</span> <span style="color: #f92672">-</span><span style="color: #f8f8f2">Path</span> <span style="color: #f8f8f2">.</span><span style="color: #960050; background-color: #1e0010">\</span><span style="color: #f8f8f2">lib</span><span style="color: #960050; background-color: #1e0010">\</span><span style="color: #f8f8f2">Microsoft.ApplicationInsights.dll;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">InstrumentationKey</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Env:APPINSIGHTS_INSTRUMENTATIONKEY;</span>
<span style="color: #f92672">#</span> <span style="color: #f8f8f2">If</span> <span style="color: #f8f8f2">your</span> <span style="color: #f8f8f2">resource</span> <span style="color: #66d9ef">is</span> <span style="color: #66d9ef">in</span> <span style="color: #f8f8f2">a</span> <span style="color: #f8f8f2">region</span> <span style="color: #66d9ef">like</span> <span style="color: #f8f8f2">Azure</span> <span style="color: #f8f8f2">Government</span> <span style="color: #66d9ef">or</span> <span style="color: #f8f8f2">Azure</span> <span style="color: #f8f8f2">China,</span> <span style="color: #f8f8f2">change</span> <span style="color: #f8f8f2">the</span> <span style="color: #f8f8f2">endpoint</span> <span style="color: #f8f8f2">address</span> <span style="color: #f8f8f2">accordingly.</span>
<span style="color: #f92672">#</span> <span style="color: #f8f8f2">Visit</span> <span style="color: #f8f8f2">https:</span><span style="color: #f92672">//</span><span style="color: #f8f8f2">docs.microsoft.com</span><span style="color: #f92672">/</span><span style="color: #f8f8f2">azure</span><span style="color: #f92672">/</span><span style="color: #f8f8f2">azure</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">monitor</span><span style="color: #f92672">/</span><span style="color: #f8f8f2">app</span><span style="color: #f92672">/</span><span style="color: #f8f8f2">custom</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">endpoints</span><span style="color: #f92672">#</span><span style="color: #f8f8f2">regions</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">that</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">require</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">endpoint</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">modification</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">EndpointAddress</span> <span style="color: #f92672">=</span> <span style="color: #e6db74">"https://dc.services.visualstudio.com/v2/track"</span><span style="color: #f8f8f2">;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Channel</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[Microsoft.ApplicationInsights.Channel.InMemoryChannel]::</span><span style="color: #66d9ef">new</span><span style="color: #f8f8f2">();</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Channel.EndpointAddress</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">EndpointAddress;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TelemetryConfiguration</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration]::</span><span style="color: #66d9ef">new</span><span style="color: #f8f8f2">(</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">InstrumentationKey,</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Channel</span>
<span style="color: #f8f8f2">);</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TelemetryClient</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[Microsoft.ApplicationInsights.TelemetryClient]::</span><span style="color: #66d9ef">new</span><span style="color: #f8f8f2">(</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TelemetryConfiguration);</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TestName</span> <span style="color: #f92672">=</span> <span style="color: #e6db74">"https://internal.website/"</span><span style="color: #f8f8f2">;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TestLocation</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Env:COMPUTERNAME;</span> <span style="color: #f92672">#</span> <span style="color: #f8f8f2">you</span> <span style="color: #f8f8f2">can</span> <span style="color: #f8f8f2">use</span> <span style="color: #66d9ef">any</span> <span style="color: #f8f8f2">string</span> <span style="color: #66d9ef">for</span> <span style="color: #f8f8f2">this</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">OperationId</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">(</span><span style="color: #66d9ef">New</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">Guid).ToString(</span><span style="color: #e6db74">"N"</span><span style="color: #f8f8f2">);</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[Microsoft.ApplicationInsights.DataContracts.AvailabilityTelemetry]::</span><span style="color: #66d9ef">new</span><span style="color: #f8f8f2">();</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.Id</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">OperationId;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.Name</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TestName;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.RunLocation</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TestLocation;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.Success</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #66d9ef">False</span><span style="color: #f8f8f2">;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Stopwatch</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[</span><span style="color: #66d9ef">System</span><span style="color: #f8f8f2">.</span><span style="color: #66d9ef">Diagnostics</span><span style="color: #f8f8f2">.Stopwatch]::</span><span style="color: #66d9ef">New</span><span style="color: #f8f8f2">()</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Stopwatch.</span><span style="color: #66d9ef">Start</span><span style="color: #f8f8f2">();</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">OriginalErrorActionPreference</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ErrorActionPreference;</span>
<span style="color: #f8f8f2">Try</span>
<span style="color: #960050; background-color: #1e0010">{</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ErrorActionPreference</span> <span style="color: #f92672">=</span> <span style="color: #e6db74">"Stop"</span><span style="color: #f8f8f2">;</span>
<span style="color: #f92672">#</span> <span style="color: #f8f8f2">Run</span> <span style="color: #f8f8f2">test</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Response</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">Invoke</span><span style="color: #f92672">-</span><span style="color: #f8f8f2">WebRequest</span> <span style="color: #f92672">-</span><span style="color: #f8f8f2">Uri</span> <span style="color: #e6db74">"https://internal.website/"</span><span style="color: #f8f8f2">;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Success</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Response.StatusCode</span> <span style="color: #f92672">-</span><span style="color: #f8f8f2">eq</span> <span style="color: #ae81ff">200</span><span style="color: #f8f8f2">;</span>
<span style="color: #f92672">#</span> <span style="color: #66d9ef">End</span> <span style="color: #f8f8f2">test</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.Success</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Success;</span>
<span style="color: #960050; background-color: #1e0010">}</span>
<span style="color: #f8f8f2">Catch</span>
<span style="color: #960050; background-color: #1e0010">{</span>
<span style="color: #f92672">#</span> <span style="color: #f8f8f2">Submit</span> <span style="color: #f8f8f2">Exception</span> <span style="color: #f8f8f2">details</span> <span style="color: #66d9ef">to</span> <span style="color: #f8f8f2">Application</span> <span style="color: #f8f8f2">Insights</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.Message</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">_.Exception.Message;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ExceptionTelemetry</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[Microsoft.ApplicationInsights.DataContracts.ExceptionTelemetry]::</span><span style="color: #66d9ef">new</span><span style="color: #f8f8f2">(</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">_.Exception);</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ExceptionTelemetry.Context.</span><span style="color: #66d9ef">Operation</span><span style="color: #f8f8f2">.Id</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">OperationId;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ExceptionTelemetry.Properties[</span><span style="color: #e6db74">"TestName"</span><span style="color: #f8f8f2">]</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TestName;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ExceptionTelemetry.Properties[</span><span style="color: #e6db74">"TestLocation"</span><span style="color: #f8f8f2">]</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TestLocation;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TelemetryClient.TrackException(</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ExceptionTelemetry);</span>
<span style="color: #960050; background-color: #1e0010">}</span>
<span style="color: #f8f8f2">Finally</span>
<span style="color: #960050; background-color: #1e0010">{</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Stopwatch.Stop();</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.Duration</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Stopwatch.Elapsed;</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability.</span><span style="color: #66d9ef">Timestamp</span> <span style="color: #f92672">=</span> <span style="color: #f8f8f2">[DateTimeOffset]::UtcNow;</span>
<span style="color: #f92672">#</span> <span style="color: #f8f8f2">Submit</span> <span style="color: #f8f8f2">Availability</span> <span style="color: #f8f8f2">details</span> <span style="color: #66d9ef">to</span> <span style="color: #f8f8f2">Application</span> <span style="color: #f8f8f2">Insights</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TelemetryClient.TrackAvailability(</span><span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">Availability);</span>
<span style="color: #f92672">#</span> <span style="color: #66d9ef">call</span> <span style="color: #f8f8f2">flush</span> <span style="color: #66d9ef">to</span> <span style="color: #f8f8f2">ensure</span> <span style="color: #f8f8f2">telemetry</span> <span style="color: #66d9ef">is</span> <span style="color: #f8f8f2">sent</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">TelemetryClient.Flush();</span>
<span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">ErrorActionPreference</span> <span style="color: #f92672">=</span> <span style="color: #960050; background-color: #1e0010">$</span><span style="color: #f8f8f2">OriginalErrorActionPreference;</span>
<span style="color: #960050; background-color: #1e0010">}</span>
</pre></div>
<div><br /></div><div>To make the code work it needs access to the dll mentioned in Niels' post - I've downloaded that as he explains and now I need to make it available to the app.</div><div><br /></div><div>To do this I navigate to the Kudu (advanced tools link) from the portal and get the screen below</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-aIDEgLlT9vg/YFe7LggO9hI/AAAAAAAADyE/4O4KuAnlfi0DLHuOs9Xj33L5wbVyS8nPgCLcBGAsYHQ/s1164/int10.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="773" data-original-width="1164" src="https://1.bp.blogspot.com/-aIDEgLlT9vg/YFe7LggO9hI/AAAAAAAADyE/4O4KuAnlfi0DLHuOs9Xj33L5wbVyS8nPgCLcBGAsYHQ/s320/int10.PNG" width="320" /></a></div><div><br /></div><div>To authenticate I need the details from the deployment blade - note that the username you use you don't specify the part before the '/' - so in my case here the username is $internalwebtest</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-lxfHr3NS3WY/YFe7LskzXII/AAAAAAAADyM/H-vebeJcxJ45atPIXMRP3Y73CDgnJvV9gCLcBGAsYHQ/s1667/int11.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="769" data-original-width="1667" src="https://1.bp.blogspot.com/-lxfHr3NS3WY/YFe7LskzXII/AAAAAAAADyM/H-vebeJcxJ45atPIXMRP3Y73CDgnJvV9gCLcBGAsYHQ/s320/int11.PNG" width="320" /></a></div><div><br /></div><div>Once authenticated we need to navigate to the folder where out function app code sits and then upload the dll by dragging and dropping from file explorer - see example below.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-HIllvfwE1pU/YFe7Lme6pHI/AAAAAAAADyI/u88vegD8r_8-RHpE6RuGgSYz499TKakgACLcBGAsYHQ/s1715/int12.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="934" data-original-width="1715" src="https://1.bp.blogspot.com/-HIllvfwE1pU/YFe7Lme6pHI/AAAAAAAADyI/u88vegD8r_8-RHpE6RuGgSYz499TKakgACLcBGAsYHQ/s320/int12.PNG" width="320" /></a></div><div><br /></div><div>Now at this point the code should have been working but in my case I got this error:</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-DVgQH2uU6wc/YFe7Mc1S6aI/AAAAAAAADyQ/i3IzWJSwAHY60xfF89daqfBlwFaGsC9pgCLcBGAsYHQ/s937/int13.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="43" data-original-width="937" src="https://1.bp.blogspot.com/-DVgQH2uU6wc/YFe7Mc1S6aI/AAAAAAAADyQ/i3IzWJSwAHY60xfF89daqfBlwFaGsC9pgCLcBGAsYHQ/s320/int13.PNG" width="320" /></a></div><div><br /></div><div>This is because our cert authority is not one that Microsoft function apps 'trust' - so it doesn't like the cert and throws this error. The fix is to upload the certs into its trusted store so that it will be happy.</div><div><br /></div><div>I initially tried to upload a combined cert with a root and intermediate cert - this didn't seem to take and was still throwing the error. I then uploaded them separately and it was then fine - you can see the uploaded files below - the only extra part we have to do is explicitly set in the function app code to load these (this seems a little odd as we kind of did this by uploading them didn't we?). Anyway to do this we just need the thumbprint values from the screen below</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-lETnO3p9iRc/YFe7Mbx859I/AAAAAAAADyU/F9AXGD4cwNo4XE13sRFwVSYUjTBFzQIZwCLcBGAsYHQ/s1599/int14.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="843" data-original-width="1599" src="https://1.bp.blogspot.com/-lETnO3p9iRc/YFe7Mbx859I/AAAAAAAADyU/F9AXGD4cwNo4XE13sRFwVSYUjTBFzQIZwCLcBGAsYHQ/s320/int14.PNG" width="320" /></a></div><div><br /></div><div>We then add a new app setting called WEBSITE_LOAD_ROOT_CERTIFICATES and in here paste the thumbprint values (separated by a comma if there is more than one)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-_jVLAIFPCdY/YFe7MuUdvqI/AAAAAAAADyY/1tjTjt26Jpowu8LiCrezJc0mCjJNkI9wwCLcBGAsYHQ/s1846/int15.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="686" data-original-width="1846" src="https://1.bp.blogspot.com/-_jVLAIFPCdY/YFe7MuUdvqI/AAAAAAAADyY/1tjTjt26Jpowu8LiCrezJc0mCjJNkI9wwCLcBGAsYHQ/s320/int15.PNG" width="320" /></a></div><div><br /></div><div>So it should look something like this:</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-JmLZ7E3rJQo/YFe7M-Gd5FI/AAAAAAAADyc/XVOt_BKD15A-K23S-uI-eBragQiMVv8MwCLcBGAsYHQ/s1379/int16.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="388" data-original-width="1379" src="https://1.bp.blogspot.com/-JmLZ7E3rJQo/YFe7M-Gd5FI/AAAAAAAADyc/XVOt_BKD15A-K23S-uI-eBragQiMVv8MwCLcBGAsYHQ/s320/int16.PNG" width="320" /></a></div><div><br /></div><div>Now the script runs without error and we start to see date in app insights - brilliant. You'll see that it looks pretty much the same as for when I demoed this in the last post for a public website - the key difference being there is no edit facility (notice the blank ringed section below) - this is as expected as we are just using this insights workspace to receive the data we create from the function app - we didn't author it in app insights.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-aAUdlhV_DvE/YFe7MxfKyTI/AAAAAAAADyg/g5EAgMpGbI485LBuXfPwL7jZO4AjRl2OgCLcBGAsYHQ/s1914/int17.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="859" data-original-width="1914" src="https://1.bp.blogspot.com/-aAUdlhV_DvE/YFe7MxfKyTI/AAAAAAAADyg/g5EAgMpGbI485LBuXfPwL7jZO4AjRl2OgCLcBGAsYHQ/s320/int17.PNG" width="320" /></a></div><div><br /></div>So there you go - website monitoring for any internal website(including all those 'legacy' on premises ones - you just need network connectivity to be able to reach then for the test.<br /><div><br /></div><div><br /></div><div><br /></div><div><br /></div><br /><p><br /></p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-83656439199970332252021-03-18T15:32:00.002-07:002021-03-20T02:08:59.499-07:00Using Azure to check a website is up - part 1<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-z8lhNpb_rP0/YFPU3B45IpI/AAAAAAAADw4/yqikwKzL6TIGuNKQdO_qoyBc3XDVxCfOQCLcBGAsYHQ/s640/galileo-92350_640.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="514" data-original-width="640" src="https://1.bp.blogspot.com/-z8lhNpb_rP0/YFPU3B45IpI/AAAAAAAADw4/yqikwKzL6TIGuNKQdO_qoyBc3XDVxCfOQCLcBGAsYHQ/s320/galileo-92350_640.jpg" width="320" /></a></div><br /><p></p><p>This is the first in a series of 2 posts dealing with how you can use Azure to monitor if a website is up and running and track the response time of that.</p><p>I've split it into 2 cases - the first (simpler) one is checking if a public facing website is up and running, the second will deal with doing the same thing for a private internal website - that's a longer more complex write up so I'll get the easy one out of the way as it shows some of the basics too which are still relevant to the second part.</p><p>So first up we're going to take advantage of a feature called 'application insights' - this has a whole load of stuff in it that I'm not going to cover here that can do things for PaaS services as well as on premises IaaS services if you install the right additional software and configure it.</p><p>I'm just going to use pretty much the simplest thing that comes with it.</p><p>So first up we create a new app insights object in the portal - so search for app insights and just click new:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-0tZwUNfI_g0/YFPKICnFSpI/AAAAAAAADv4/DFvgOj5FjqYq5zvXUdHvQVN87H4uKSXiACLcBGAsYHQ/s1257/appi1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="335" data-original-width="1257" src="https://1.bp.blogspot.com/-0tZwUNfI_g0/YFPKICnFSpI/AAAAAAAADv4/DFvgOj5FjqYq5zvXUdHvQVN87H4uKSXiACLcBGAsYHQ/s320/appi1.PNG" width="320" /></a></div><div><br /></div><div>Fill in the usual basic information about location and resource group - there is a newer option about choosing between classic and workspace-based. This is just a decision about where the data is stored - if you chose classic it is log analytics as far as I can tell but it's hidden and you can't access it as a separate workspace - if you choose workspace-based then its normal log analytics - this now seems to be the default and is the most sensible option to choose.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-H3jLvWmAzxQ/YFPKIDNp9KI/AAAAAAAADv8/becrgzXZ8j43x5hOdV9dsdTRk3s7U-4rgCLcBGAsYHQ/s1180/appi2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="776" data-original-width="1180" src="https://1.bp.blogspot.com/-H3jLvWmAzxQ/YFPKIDNp9KI/AAAAAAAADv8/becrgzXZ8j43x5hOdV9dsdTRk3s7U-4rgCLcBGAsYHQ/s320/appi2.PNG" width="320" /></a></div><div><br /></div><div>Once that's all validated click create</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-xDfB1dMAcmQ/YFPKICPw65I/AAAAAAAADwA/22aGeHquDoES-Bz5CgDAwK6F8pRI72k6ACLcBGAsYHQ/s984/appi3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="847" data-original-width="984" src="https://1.bp.blogspot.com/-xDfB1dMAcmQ/YFPKICPw65I/AAAAAAAADwA/22aGeHquDoES-Bz5CgDAwK6F8pRI72k6ACLcBGAsYHQ/s320/appi3.PNG" width="320" /></a></div><div><br /></div><div>Wait for it to create and then navigate to the main blade for the newly create resource. Once there click on the availability link shown below:</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-n7tXMm8Zz9w/YFPKIlIWmZI/AAAAAAAADwE/Gty6Y5heaBAfOk1PrQjgg4knAQoL2rA6gCLcBGAsYHQ/s808/appi4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="808" data-original-width="410" height="320" src="https://1.bp.blogspot.com/-n7tXMm8Zz9w/YFPKIlIWmZI/AAAAAAAADwE/Gty6Y5heaBAfOk1PrQjgg4knAQoL2rA6gCLcBGAsYHQ/s320/appi4.PNG" /></a></div><div><br /></div><div>Once in there click 'add test'</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-z96Nbp7rUzY/YFPKI2FSPiI/AAAAAAAADwI/slUS_pQwrgozBigoyWsm-aVJ5S-hZOkewCLcBGAsYHQ/s1595/appi5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="774" data-original-width="1595" src="https://1.bp.blogspot.com/-z96Nbp7rUzY/YFPKI2FSPiI/AAAAAAAADwI/slUS_pQwrgozBigoyWsm-aVJ5S-hZOkewCLcBGAsYHQ/s320/appi5.PNG" width="320" /></a></div><div><br /></div><div><br /></div><div>Now here is where we define what we are actually checking - I give this a name and choose URL ping test (so a simple is the site up and how long does it take) - there is a more complex multi step web test where you could do some sort of navigation round the site - but here I'm just keeping it simple. I then specify the url to check (my blog site) along with how often to check and also where to test from. That's useful as you can check from different regions of the globe to see how it behaves (and I guess how well any CDN system might be working).</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-EelkzPSksFU/YFPKJL0rr2I/AAAAAAAADwM/03LmzGaaALsw2ni-_tYeRvO4loLWZ3uYwCLcBGAsYHQ/s882/appi6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="882" data-original-width="738" height="320" src="https://1.bp.blogspot.com/-EelkzPSksFU/YFPKJL0rr2I/AAAAAAAADwM/03LmzGaaALsw2ni-_tYeRvO4loLWZ3uYwCLcBGAsYHQ/s320/appi6.PNG" /></a></div><div><br /></div><div>Click create and that's it - I now have a test running every 5 minutes to see if the website is up and running - you can see the first results of that below - the test ran once and the website was up and the page was served in 616ms</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-EQBH7M4D06o/YFPKJewHmcI/AAAAAAAADwQ/ifavC5aop749IJI4PxtyZuXL4m3bQXtPQCLcBGAsYHQ/s1597/appi7.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="760" data-original-width="1597" src="https://1.bp.blogspot.com/-EQBH7M4D06o/YFPKJewHmcI/AAAAAAAADwQ/ifavC5aop749IJI4PxtyZuXL4m3bQXtPQCLcBGAsYHQ/s320/appi7.PNG" width="320" /></a></div><br /><p>There are various options then to change the time range and the style of graph plotted - but the basics are all done - my website is now being checked.</p><p>I can then if i want to build alerts on top of that - if website is down do 'X' if website is slower than 'Y' seconds more than 3 times in a row do 'Z' - it's very flexible</p><p>You also get quite a bit of detail of when it ran the test - see pic below:</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-UG1JK1kukoA/YFPNTWNKSMI/AAAAAAAADwg/aFcnb6UWIT4Re50puh2msmg12qPEf5e9gCLcBGAsYHQ/s1824/appi8.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="827" data-original-width="1824" src="https://1.bp.blogspot.com/-UG1JK1kukoA/YFPNTWNKSMI/AAAAAAAADwg/aFcnb6UWIT4Re50puh2msmg12qPEf5e9gCLcBGAsYHQ/s320/appi8.PNG" width="320" /></a></div><div><br /></div><div>You can also see the breakdown and speed of the various locations the 'probes' are running from that check the website.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-DfvMAU2KeUY/YFPQswI0w1I/AAAAAAAADwo/_59TwDadrg0A4tFR8AQ8QCoR4N_fX4V-ACLcBGAsYHQ/s975/appi9.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="447" data-original-width="975" src="https://1.bp.blogspot.com/-DfvMAU2KeUY/YFPQswI0w1I/AAAAAAAADwo/_59TwDadrg0A4tFR8AQ8QCoR4N_fX4V-ACLcBGAsYHQ/s320/appi9.PNG" width="320" /></a></div><div><br /></div><div><br /></div>You can even produce SLA reports on the site availability - thought there is not much to show here in my simple example<div><br /></div><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ovIQB2pv5t4/YFPQ1FQTqnI/AAAAAAAADws/BPVwqzsE1cM0MsTcnEI3cpq-Q4u-LpSswCLcBGAsYHQ/s1885/appi10.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="723" data-original-width="1885" src="https://1.bp.blogspot.com/-ovIQB2pv5t4/YFPQ1FQTqnI/AAAAAAAADws/BPVwqzsE1cM0MsTcnEI3cpq-Q4u-LpSswCLcBGAsYHQ/s320/appi10.PNG" width="320" /></a></div><br /></div><div>This is really useful - you can very rapidly set this up and check your public facing sites to give some really good info on if your public presence is actually up and running and how well it is working for different user bases all over the globe.</div><div><br /></div><div><br /></div><div>Part 2 to follow with how you do this same thing for internal sites.....when I muster up the energy to capture the screenshots and remember all the steps.....</div><div><br /></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-38361329227613299722021-03-13T02:25:00.003-08:002021-03-13T02:25:48.788-08:00Bulk tagging in Azure<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-g7oZEbGZ1es/YEyTCgrxlsI/AAAAAAAADvo/ZbpNFgNmLkMeffTdYMnzfqfuLdRIdLc8wCLcBGAsYHQ/s640/christmas-220177_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="640" height="320" src="https://1.bp.blogspot.com/-g7oZEbGZ1es/YEyTCgrxlsI/AAAAAAAADvo/ZbpNFgNmLkMeffTdYMnzfqfuLdRIdLc8wCLcBGAsYHQ/s320/christmas-220177_640.jpg" /></a></div><br /><p><br /></p><p>We've recently added a whole load of 'on prem' resources to Azure via Azure ARC and we wanted to add some extra tags to those resources to display some more useful information about what they are and who they belong to.</p><p>We already had this mapping information in an excel sheet showing that server x is associated with application y and belong to application owner z etc</p><p>So all we had to do was take the content of this sheet and somehow apply that as tags to the servers once available in Azure.</p><p>To do this again I've just used a simple bit of powershell to read the sheet and then pull out the information into the right format to be assigned to the servers.</p><p>This was a little fiddly as the update-aztag cmdlet ( the thing that lets you update tags) expects the tags to be in the format of a hashtable - this make for a bit of messing around in constructing the input data in the right format.</p><p>I managed to run the whole thing in 'cloud shell' (so no messing around with getting a working powershell environment on my local machine) - this can be accessed by the little rectangle with the triangle in the header bar of the portal or directly by going to <a href="https://shell.azure.com">https://shell.azure.com</a></p><p>Once there you just need to do the following:</p><p>1) copy your excel sheet over (you can just drag/drop into the window) my sheet is of the following format:</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-quZOtL66Cd8/YEyP02mZBQI/AAAAAAAADvY/g8JL86QXnkAtueWBEXSos0HrXmxukuUHQCLcBGAsYHQ/s1876/Capture.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="118" data-original-width="1876" height="40" src="https://1.bp.blogspot.com/-quZOtL66Cd8/YEyP02mZBQI/AAAAAAAADvY/g8JL86QXnkAtueWBEXSos0HrXmxukuUHQCLcBGAsYHQ/w640-h40/Capture.PNG" width="640" /></a></div><br /><p>So I have the following headers : hostname,sub,owneremail,subname,eam,eam,app,lob,desc - i now want to take the data from that and assign that information to the tags on the server mentioned in column A (not the sub/subname things here are not Azure subscriptions - it's for something else :-))</p><p>2) Install the additional module for processing excel - this is not a default thing</p><p>Install-Module -Name ImportExcel -RequiredVersion 4.0.8</p><p>3) Run the following bit of code that will loop through the sheet pulling the details out and applying the tags</p><!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #007020;">import-excel</span> -Path <span style="background-color: #fff0f0;">'tagdata.xlsx'</span> |<span style="color: #008800; font-weight: bold;">ForEach</span>-Object {
<span style="color: #007020;">write-host</span> (<span style="color: #996633;">$_</span>.Hostname)
<span style="color: #996633;">$tags</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{<span style="background-color: #fff0f0;">'VPC_SubscriptionId'</span>= <span style="background-color: #fff0f0;">"</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">_.sub)</span><span style="background-color: #fff0f0;">"</span> <span style="background-color: #ffaaaa; color: red;">;</span> <span style="background-color: #fff0f0;">'Owner_Email'</span>= $(<span style="color: #996633;">$_</span>.owneremail) <span style="background-color: #ffaaaa; color: red;">;</span> <span style="background-color: #fff0f0;">'VPC_SUbscription'</span>= $(<span style="color: #996633;">$_</span>.subname) <span style="background-color: #ffaaaa; color: red;">;</span> <span style="background-color: #fff0f0;">'IteraplanID'</span>= $(<span style="color: #996633;">$_</span>.eam) <span style="background-color: #ffaaaa; color: red;">;</span> <span style="background-color: #fff0f0;">'VPC_App'</span>= $(<span style="color: #996633;">$_</span>.app) <span style="background-color: #ffaaaa; color: red;">;</span> <span style="background-color: #fff0f0;">'LOB'</span>= $(<span style="color: #996633;">$_</span>.LOB) }
<span style="color: #996633;">$RES</span> = <span style="background-color: #fff0f0;">"/subscriptions/subid/resourceGroups/rgid/providers/Microsoft.HybridCompute/machines/"</span>+(<span style="color: #996633;">$_</span>.Hostname)
<span style="color: #007020;">Update-AzTag</span> -ResourceId <span style="color: #996633;">$RES</span> -Tag <span style="color: #996633;">$tags</span> -Operation Merge
<span style="color: #996633;">$RES</span> = <span style="background-color: #fff0f0;">"/subscriptions/subid/resourceGroups/rgid/providers/Microsoft.HybridCompute/machines/"</span>+(<span style="color: #996633;">$_</span>.Hostname)+<span style="background-color: #fff0f0;">".domain"</span>
<span style="color: #007020;">Update-AzTag</span> -ResourceId <span style="color: #996633;">$RES</span> -Tag <span style="color: #996633;">$tags</span> -Operation Merge
}
</pre></div>
<p><br /></p><p>Now due to a quirk in how some of the servers got added into Azure ARC some have domain extension and some don't - the easiest way to deal with that is just to run the tag command twice - then if the domain is not present the first command picks it up, otherwise the second one does - throws a few errors of course but we can just ignore those.</p><p>Process took a little while to loop through everything and there did seem to be a very long delay in the updated data being visible in the portal - some sort of caching effect i think. However the get-aztag result confirmed it was set straight away from that point of view.</p><p>In our case all of the ARC servers are in the same resource group which then makes the script easy - if you have a more complex setup then the script would need amending slightly.</p><p>Same technique can be used for any tagging of course</p><p>End result looks something like this:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-4GrP1fPKDfU/YEySiFhgACI/AAAAAAAADvg/9l9vyzaMEP8lTBSNpillNPv0ibcXXDlzgCLcBGAsYHQ/s1438/capture2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="483" data-original-width="1438" height="214" src="https://1.bp.blogspot.com/-4GrP1fPKDfU/YEySiFhgACI/AAAAAAAADvg/9l9vyzaMEP8lTBSNpillNPv0ibcXXDlzgCLcBGAsYHQ/w640-h214/capture2.PNG" width="640" /></a></div><br /><p>Happy tagging......</p><p><br /></p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com2tag:blogger.com,1999:blog-302298286928742422.post-55183610776983157252021-03-04T15:11:00.001-08:002021-03-04T15:11:27.001-08:00Oracle on Azure - an Inventory<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ixOZzcKC_lw/YEFoR3uQDSI/AAAAAAAADvQ/DE-z5LBV-isH1NJVQcDkr8VxrqYAG-KeACLcBGAsYHQ/s640/space-3262811_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="640" src="https://1.bp.blogspot.com/-ixOZzcKC_lw/YEFoR3uQDSI/AAAAAAAADvQ/DE-z5LBV-isH1NJVQcDkr8VxrqYAG-KeACLcBGAsYHQ/s320/space-3262811_640.jpg" width="320" /></a></div><br /><p></p><p><span style="font-family: inherit;">Well after the roaring success of this <a href="http://dbaharrison.blogspot.com/2021/02/serverless-cmdb-extract-from-azure.html">post</a> I've had a stream of people (well at least one) knocking at my door to do something similar but showing our Oracle installations in Azure.</span></p><p><span style="font-family: inherit;">Now if you're a little bit familiar with Oracle on Azure you'll know there is no PaaS offering here - you have to run it in the traditional way on IaaS (it works just as well as on premises just be very mindful of the IO layout you put in....). The fact that it's just something running on IaaS means though that Azure as a platform has no clue at all that Oracle is running on there - it's not like you can just go and look in the portal for oracle databases.</span></p><p><span style="font-family: inherit;">So trying to find out what you have is not possible right? A whole load of disconnected servers in different subscriptions and networks with no common interface to them - I'm sure you are all asking how can this be done?</span></p><p><span style="font-family: inherit;">Well let me tell you - we can use a wonderful </span>PowerShell<span style="font-family: inherit;"> script to achieve this - the key part being a little cmdlet called Invoke-AzVMRuncommand - as long as the user that runs that command has the 'virtual machine contributor' role it is able to execute commands on any server you are in the context of in Azure - you don't need to be able to reach it via the ip network and you don't even need to authenticate explicitly - the cmdlet is 'allowed' to do this.</span></p><p><span style="font-family: inherit;">So powered by this cmdlet I built a whole wrapper around it to go out and fetch the details of what is installed. Now this discovery is fairly basic - it's not going to tell you options in use - it's just going to deal with the basics of ORACLE_HOME, ORACLE_SID,VERSION, DATABASE NAME and if the database is actually running. For us that is enough for our immediate needs - it's not going to be everything you need for a full audit of what you have but it's a really good start.</span></p><p><span style="font-family: inherit;">I've pasted the script in it's entirety at the end of this post - so if you want to jump ahead and just cut and paste it then go for it - I'll spend the next few paragraphs explaining what it's doing and how it works as I think it's useful to be able to follow my logic and learn some </span>PowerShell<span style="font-family: inherit;"> along the way.</span></p><p><span style="font-family: inherit;">To be able to do the full serverless deployment - please refer back to that earlier post - the steps are exactly the same - the only addition is making sure the function app has virtual machine contributor in addition to everything else and possibly increasing the timeout to a few hours (the cmdlet seems to often hand for quite a while in the actual connecting phase though the script being run on the destination server is almost instant.</span></p><p><span style="font-family: inherit;">OK - onto the explanation - I'll start by again referring here <a href="http://ramblingcookiemonster.github.io/Join-Object/">http://ramblingcookiemonster.github.io/Join-Object/ </a> for the excellent join-object function I make use of in the code - thanks Warren.</span></p><p>The first part of the code is just the standard function app headers and Warrens code so I won't say anything more there and I'll jump on to the more interesting parts.</p><p>The first one is the bit of PowerShell that writes out a bash script (the thing I actually want to run on the Linux machine) - not this was an exercise in understanding all the different types of quotes and escape character in PowerShell - so the end result contains a well formatted bash script - this was no mean feat I can assure you. This end up looking like this:</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ucxYCw0Tb18/YEFbnl32T8I/AAAAAAAADug/XklBqnWNBKQrmdfW8r7uJXbDATpWdyD7wCLcBGAsYHQ/s1215/oracle.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="219" data-original-width="1215" src="https://1.bp.blogspot.com/-ucxYCw0Tb18/YEFbnl32T8I/AAAAAAAADug/XklBqnWNBKQrmdfW8r7uJXbDATpWdyD7wCLcBGAsYHQ/s16000/oracle.PNG" /></a></div><br /><p>The end result being a bash script called (originally enough) script.sh.</p><p>Now it's worth looking at what this will actually do as this is the key thing that tells us if Oracle is installed or not and what that looks like when it is.</p><p>So in the screenshot above - here is an explanation of the lines..... (for a short script this needs quite a bit of explanation.....)</p><p>515 - define that this script should be executed using the bash shell</p><p>516 - Define the internal file separator as just being a newline character - by default it treats spaces/tabs also as a separator and this messes up later processing</p><p>517 - start a loop for every line in the oratab that isn't commented out and doesn't contain the word agent</p><p>519 - pull the ORACLE_HOME out of the oratab line</p><p>520 - pull the database name out of the oratab line</p><p>521 - find the actual version of the software using the sqlplus header (some processing to cope with the way the header changes in some versions)</p><p>522 - check if the database is actually running</p><p>523 - echo out all the above info I found + the word YOOHOO which i will make use of later</p><p>Right now that's out of the way and I've reattached all the quote keys to my keyboard after smashing them all up working out how to create this we can move on in the script to this section</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-YVnula6UVl8/YEFeQigSFpI/AAAAAAAADuw/rAXwqMy5eukY92Dm52OGdwzljm_QcjsywCLcBGAsYHQ/s1731/oracle2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="757" data-original-width="1731" src="https://1.bp.blogspot.com/-YVnula6UVl8/YEFeQigSFpI/AAAAAAAADuw/rAXwqMy5eukY92Dm52OGdwzljm_QcjsywCLcBGAsYHQ/s16000/oracle2.PNG" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p>Here I initialize a few basic variables (couple of arrays and a date value) then I do the following (again explaining some of the lines below)</p><p>530 - fetch all the subscriptions into a variable</p><p>531/532 - loop through those setting the context to each one as we go round the loop</p><p>535 - do a resource graph query to return all the Linux machines in that subscription (now I've added some extra criteria here to filter out some of the noise but you don't have to do that.) I exclude anything that hasn't got ORA in the name of the VM, anything where the name starts with aks (these are machine created by the AKS PaaS service that i don't need to check), anything where the plan is not null (appliance machines will have this set to something - NetApp for example) - and finally only return values where the machine is running as the cmdlet can do nothing against a powered down machine.</p><p>The reason that I look specifically for machines with Ora in is that the dataset was too huge to do everything in one pass (and have it complete in a reasonable time) so I have 2 version of the script running in parallel - one looking for ORA and the other doing the inverse. Hopefully in your case that logic isn't required. Also note that I did this as a different function in a different function app. It seems 2 functions in the same functionapp interfere with each other - it kind of looks like only one function can be executed at a time.....</p><p>537 - open a new loop within the main loop do do something for every VM</p><p>538 - run the magic cmdlet to execute the script i created earlier and store the output in a variable</p><p>541 - define a pattern to look for (the YOOHOO thing I set in the script) - this allows me to pull out the lines of interest from the rest of the noise in the output</p><p>544 - split the output into individual lines</p><p>547-552 parse the output data (this is quite a minor work of art in itself and took me a while to figure out)</p><p>554-558 - expand the array of data i now have with some additional information (like the vm name , subscription etc)</p><p>562 - append the final output from that VM to the FINRESULT array - the loop continues round adding each 'row' to this array</p><p>OK - that's that stage complete - now i need to enrich that dataset with some more info - like tags that contain some extra information</p><p>This is what that section looks like</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-56dqiaXONF8/YEFiE8R8AFI/AAAAAAAADu4/pkDqHY3h9b80j3YB-Kdsi0aRz5YkR-dSQCLcBGAsYHQ/s1730/oracle3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="214" data-original-width="1730" src="https://1.bp.blogspot.com/-56dqiaXONF8/YEFiE8R8AFI/AAAAAAAADu4/pkDqHY3h9b80j3YB-Kdsi0aRz5YkR-dSQCLcBGAsYHQ/s16000/oracle3.PNG" /></a></div><br /><p>So to explain here</p><p>568-572 is doing a resource graph query to pull out various elements of what Azure knows - i do a few joins in resource graph to pull everything out I want</p><p>575 - i then join the ARG array i just got to the earlier FINRESULT array using a column i created called joiner in both sides. This column is a concatenation of vmname, resourcegroup and subscription - the combination of which is unique for the join</p><p>Final stretch now kids - last few lines are here:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-X-dUjdD0rtU/YEFi6XFCSMI/AAAAAAAADvA/6THL88csC8khpwe1qiwybKiDHxtA6ic1gCLcBGAsYHQ/s1544/oracle4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="237" data-original-width="1544" src="https://1.bp.blogspot.com/-X-dUjdD0rtU/YEFi6XFCSMI/AAAAAAAADvA/6THL88csC8khpwe1qiwybKiDHxtA6ic1gCLcBGAsYHQ/s16000/oracle4.PNG" /></a></div><br /><p>And final bit of explanation</p><p>578 - fetch the metadata about all the machine sizes available in West Europe into yet another array variable</p><p>580 - join the previously joined data to this VM metadata (the reason i do this is primarily just to get the cpu count - which as we all know is a key metric for oracle licencing - I also have the VM series from which we can see if hyperthreading is relevant or not.</p><p>582 - output the file to a temporary local file - quoting some of the columns that contain 'funny' characters that will confuse the csv format</p><p>586-587 - write that file out to blob storage with a date stamp</p><p>And there you have it - a list of al of your oracle database across all of your Azure estate - the impossible made possible :-)</p><p><br /></p><p>Caveats though before you all run off and copy the script and do this yourself</p><p>a) only works against Linux - Oracle on Windows is not something we do so script does not cater for that. I guess something similar is possible - you'd maybe just need some PowerShell equivalent of the discovery I did</p><p>b) Only works if the machines are up - we scheduled the script to run during office hours as we heavily make use of shutting down machines to save costs out of normal hours</p><p>c) relies on using the oratab file in the 'normal' way</p><p>d) only be tested on 11.2 and up - older versions may not work the same way</p><p>e) It really only covers basic discovery - if you have a more sophisticated script then I guess you can just call that via this method and parse the output differently.</p><p>A quick sample of the output with some columns hidden and some data redacted is shown below</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-S5yEhrnHjn0/YEFmYc5n1tI/AAAAAAAADvI/p3VjSQqHGAAl7Th5JA8UtL9ELNGg41O4wCLcBGAsYHQ/s1692/oracle5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="131" data-original-width="1692" src="https://1.bp.blogspot.com/-S5yEhrnHjn0/YEFmYc5n1tI/AAAAAAAADvI/p3VjSQqHGAAl7Th5JA8UtL9ELNGg41O4wCLcBGAsYHQ/s16000/oracle5.PNG" /></a></div><br /><p>And here is the script in all its magnificence.....</p><!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #888888;"># Input bindings are passed in via param block.</span>
<span style="color: #008800; font-weight: bold;">param</span>(<span style="color: #996633;">$Timer</span>)
<span style="color: #888888;"># Get the current universal time in the default string format.</span>
<span style="color: #996633;">$currentUTCtime</span> = (<span style="color: #007020;">Get-Date</span>).ToUniversalTime()
<span style="color: #888888;"># The 'IsPastDue' property is 'true' when the current function invocation is later than scheduled.</span>
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$Timer</span>.IsPastDue) {
<span style="color: #007020;">Write-Host</span> <span style="background-color: #fff0f0;">"PowerShell timer is running late!"</span>
}
<span style="color: #888888;"># Write an information log with the current time.</span>
<span style="color: #007020;">Write-Host</span> <span style="background-color: #fff0f0;">"PowerShell timer trigger function ran! TIME: $currentUTCtime"</span>
<span style="color: #008800; font-weight: bold;">function</span> <span style="color: #007020;">Join-Object</span>
{
<span style="color: #888888;"><#</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.SYNOPSIS</span><span style="color: #888888;"></span>
<span style="color: #888888;"> Join data from two sets of objects based on a common value</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.DESCRIPTION</span><span style="color: #888888;"></span>
<span style="color: #888888;"> Join data from two sets of objects based on a common value</span>
<span style="color: #888888;"> For more details, see the accompanying blog post:</span>
<span style="color: #888888;"> http://ramblingcookiemonster.github.io/Join-Object/</span>
<span style="color: #888888;"> For even more details, see the original code and discussions that this borrows from:</span>
<span style="color: #888888;"> Dave Wyatt's Join-Object - http://powershell.org/wp/forums/topic/merging-very-large-collections</span>
<span style="color: #888888;"> Lucio Silveira's Join-Object - http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Left</span>
<span style="color: #888888;"> 'Left' collection of objects to join. You can use the pipeline for Left.</span>
<span style="color: #888888;"> The objects in this collection should be consistent.</span>
<span style="color: #888888;"> We look at the properties on the first object for a baseline.</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Right</span>
<span style="color: #888888;"> 'Right' collection of objects to join.</span>
<span style="color: #888888;"> The objects in this collection should be consistent.</span>
<span style="color: #888888;"> We look at the properties on the first object for a baseline.</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> LeftJoinProperty</span>
<span style="color: #888888;"> Property on Left collection objects that we match up with RightJoinProperty on the Right collection</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> RightJoinProperty</span>
<span style="color: #888888;"> Property on Right collection objects that we match up with LeftJoinProperty on the Left collection</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> LeftProperties</span>
<span style="color: #888888;"> One or more properties to keep from Left. Default is to keep all Left properties (*).</span>
<span style="color: #888888;"> Each property can:</span>
<span style="color: #888888;"> - Be a plain property name like "Name"</span>
<span style="color: #888888;"> - Contain wildcards like "*"</span>
<span style="color: #888888;"> - Be a hashtable like @{Name="Product Name";Expression={$_.Name}}.</span>
<span style="color: #888888;"> Name is the output property name</span>
<span style="color: #888888;"> Expression is the property value ($_ as the current object)</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> Alternatively, use the Suffix or Prefix parameter to avoid collisions</span>
<span style="color: #888888;"> Each property using this hashtable syntax will be excluded from suffixes and prefixes</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> RightProperties</span>
<span style="color: #888888;"> One or more properties to keep from Right. Default is to keep all Right properties (*).</span>
<span style="color: #888888;"> Each property can:</span>
<span style="color: #888888;"> - Be a plain property name like "Name"</span>
<span style="color: #888888;"> - Contain wildcards like "*"</span>
<span style="color: #888888;"> - Be a hashtable like @{Name="Product Name";Expression={$_.Name}}.</span>
<span style="color: #888888;"> Name is the output property name</span>
<span style="color: #888888;"> Expression is the property value ($_ as the current object)</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> Alternatively, use the Suffix or Prefix parameter to avoid collisions</span>
<span style="color: #888888;"> Each property using this hashtable syntax will be excluded from suffixes and prefixes</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Prefix</span>
<span style="color: #888888;"> If specified, prepend Right object property names with this prefix to avoid collisions</span>
<span style="color: #888888;"> Example:</span>
<span style="color: #888888;"> Property Name = 'Name'</span>
<span style="color: #888888;"> Suffix = 'j_'</span>
<span style="color: #888888;"> Resulting Joined Property Name = 'j_Name'</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Suffix</span>
<span style="color: #888888;"> If specified, append Right object property names with this suffix to avoid collisions</span>
<span style="color: #888888;"> Example:</span>
<span style="color: #888888;"> Property Name = 'Name'</span>
<span style="color: #888888;"> Suffix = '_j'</span>
<span style="color: #888888;"> Resulting Joined Property Name = 'Name_j'</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Type</span>
<span style="color: #888888;"> Type of join. Default is AllInLeft.</span>
<span style="color: #888888;"> AllInLeft will have all elements from Left at least once in the output, and might appear more than once</span>
<span style="color: #888888;"> if the where clause is true for more than one element in right, Left elements with matches in Right are</span>
<span style="color: #888888;"> preceded by elements with no matches.</span>
<span style="color: #888888;"> SQL equivalent: outer left join (or simply left join)</span>
<span style="color: #888888;"> AllInRight is similar to AllInLeft.</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> OnlyIfInBoth will cause all elements from Left to be placed in the output, only if there is at least one</span>
<span style="color: #888888;"> match in Right.</span>
<span style="color: #888888;"> SQL equivalent: inner join (or simply join)</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> AllInBoth will have all entries in right and left in the output. Specifically, it will have all entries</span>
<span style="color: #888888;"> in right with at least one match in left, followed by all entries in Right with no matches in left, </span>
<span style="color: #888888;"> followed by all entries in Left with no matches in Right.</span>
<span style="color: #888888;"> SQL equivalent: full join</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"></span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> #Define some input data.</span>
<span style="color: #888888;"> $l = 1..5 | Foreach-Object {</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Name = "jsmith$_"</span>
<span style="color: #888888;"> Birthday = (Get-Date).adddays(-1)</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> $r = 4..7 | Foreach-Object{</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Department = "Department $_"</span>
<span style="color: #888888;"> Name = "Department $_"</span>
<span style="color: #888888;"> Manager = "jsmith$_"</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> #We have a name and Birthday for each manager, how do we find their department, using an inner join?</span>
<span style="color: #888888;"> Join-Object -Left $l -Right $r -LeftJoinProperty Name -RightJoinProperty Manager -Type OnlyIfInBoth -RightProperties Department</span>
<span style="color: #888888;"> # Name Birthday Department </span>
<span style="color: #888888;"> # ---- -------- ---------- </span>
<span style="color: #888888;"> # jsmith4 4/14/2015 3:27:22 PM Department 4</span>
<span style="color: #888888;"> # jsmith5 4/14/2015 3:27:22 PM Department 5</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"> </span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> #Define some input data.</span>
<span style="color: #888888;"> $l = 1..5 | Foreach-Object {</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Name = "jsmith$_"</span>
<span style="color: #888888;"> Birthday = (Get-Date).adddays(-1)</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> $r = 4..7 | Foreach-Object{</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Department = "Department $_"</span>
<span style="color: #888888;"> Name = "Department $_"</span>
<span style="color: #888888;"> Manager = "jsmith$_"</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> #We have a name and Birthday for each manager, how do we find all related department data, even if there are conflicting properties?</span>
<span style="color: #888888;"> $l | Join-Object -Right $r -LeftJoinProperty Name -RightJoinProperty Manager -Type AllInLeft -Prefix j_</span>
<span style="color: #888888;"> # Name Birthday j_Department j_Name j_Manager</span>
<span style="color: #888888;"> # ---- -------- ------------ ------ ---------</span>
<span style="color: #888888;"> # jsmith1 4/14/2015 3:27:22 PM </span>
<span style="color: #888888;"> # jsmith2 4/14/2015 3:27:22 PM </span>
<span style="color: #888888;"> # jsmith3 4/14/2015 3:27:22 PM </span>
<span style="color: #888888;"> # jsmith4 4/14/2015 3:27:22 PM Department 4 Department 4 jsmith4 </span>
<span style="color: #888888;"> # jsmith5 4/14/2015 3:27:22 PM Department 5 Department 5 jsmith5 </span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"></span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> #Hey! You know how to script right? Can you merge these two CSVs, where Path1's IP is equal to Path2's IP_ADDRESS?</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> #Get CSV data</span>
<span style="color: #888888;"> $s1 = Import-CSV $Path1</span>
<span style="color: #888888;"> $s2 = Import-CSV $Path2</span>
<span style="color: #888888;"> #Merge the data, using a full outer join to avoid omitting anything, and export it</span>
<span style="color: #888888;"> Join-Object -Left $s1 -Right $s2 -LeftJoinProperty IP_ADDRESS -RightJoinProperty IP -Prefix 'j_' -Type AllInBoth |</span>
<span style="color: #888888;"> Export-CSV $MergePath -NoTypeInformation</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"></span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> # "Hey Warren, we need to match up SSNs to Active Directory users, and check if they are enabled or not.</span>
<span style="color: #888888;"> # I'll e-mail you an unencrypted CSV with all the SSNs from gmail, what could go wrong?"</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> # Import some SSNs. </span>
<span style="color: #888888;"> $SSNs = Import-CSV -Path D:\SSNs.csv</span>
<span style="color: #888888;"> #Get AD users, and match up by a common value, samaccountname in this case:</span>
<span style="color: #888888;"> Get-ADUser -Filter "samaccountname -like 'wframe*'" |</span>
<span style="color: #888888;"> Join-Object -LeftJoinProperty samaccountname -Right $SSNs `</span>
<span style="color: #888888;"> -RightJoinProperty samaccountname -RightProperties ssn `</span>
<span style="color: #888888;"> -LeftProperties samaccountname, enabled, objectclass</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.NOTES</span><span style="color: #888888;"></span>
<span style="color: #888888;"> This borrows from:</span>
<span style="color: #888888;"> Dave Wyatt's Join-Object - http://powershell.org/wp/forums/topic/merging-very-large-collections/</span>
<span style="color: #888888;"> Lucio Silveira's Join-Object - http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx</span>
<span style="color: #888888;"> Changes:</span>
<span style="color: #888888;"> Always display full set of properties</span>
<span style="color: #888888;"> Display properties in order (left first, right second)</span>
<span style="color: #888888;"> If specified, add suffix or prefix to right object property names to avoid collisions</span>
<span style="color: #888888;"> Use a hashtable rather than ordereddictionary (avoid case sensitivity)</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.LINK</span><span style="color: #888888;"></span>
<span style="color: #888888;"> http://ramblingcookiemonster.github.io/Join-Object/</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.FUNCTIONALITY</span><span style="color: #888888;"></span>
<span style="color: #888888;"> PowerShell Language</span>
<span style="color: #888888;"> #></span>
[<span style="color: #008800; font-weight: bold;">CmdletBinding</span>()]
<span style="color: #008800; font-weight: bold;">Param</span>
(
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span>=<span style="color: #996633;">$true</span>,
<span style="color: #008800; font-weight: bold;">ValueFromPipeLine</span> = <span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[object[]]</span> <span style="color: #996633;">$Left</span>,
<span style="color: #888888;"># List to join with $Left</span>
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span>=<span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[object[]]</span> <span style="color: #996633;">$Right</span>,
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span> = <span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[string]</span> <span style="color: #996633;">$LeftJoinProperty</span>,
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span> = <span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[string]</span> <span style="color: #996633;">$RightJoinProperty</span>,
<span style="color: #003366; font-weight: bold;">[object[]]</span><span style="color: #996633;">$LeftProperties</span> = <span style="background-color: #fff0f0;">'*'</span>,
<span style="color: #888888;"># Properties from $Right we want in the output.</span>
<span style="color: #888888;"># Like LeftProperties, each can be a plain name, wildcard or hashtable. See the LeftProperties comments.</span>
<span style="color: #003366; font-weight: bold;">[object[]]</span><span style="color: #996633;">$RightProperties</span> = <span style="background-color: #fff0f0;">'*'</span>,
[<span style="color: #008800; font-weight: bold;">validateset</span>( <span style="background-color: #fff0f0;">'AllInLeft'</span>, <span style="background-color: #fff0f0;">'OnlyIfInBoth'</span>, <span style="background-color: #fff0f0;">'AllInBoth'</span>, <span style="background-color: #fff0f0;">'AllInRight'</span>)]
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span>=<span style="color: #996633;">$false</span>)]
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Type</span> = <span style="background-color: #fff0f0;">'AllInLeft'</span>,
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Prefix</span>,
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Suffix</span>
)
<span style="color: #008800; font-weight: bold;">Begin</span>
{
<span style="color: #008800; font-weight: bold;">function</span> AddItemProperties(<span style="color: #996633;">$item</span>, <span style="color: #996633;">$properties</span>, <span style="color: #996633;">$hash</span>)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$item</span>)
{
<span style="color: #008800; font-weight: bold;">return</span>
}
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$property</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$properties</span>)
{
<span style="color: #996633;">$propertyHash</span> = <span style="color: #996633;">$property</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[hashtable]</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$null</span> <span style="color: #333333;">-ne</span> <span style="color: #996633;">$propertyHash</span>)
{
<span style="color: #996633;">$hashName</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"name"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[string]</span>
<span style="color: #996633;">$expression</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"expression"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[scriptblock]</span>
<span style="color: #996633;">$expressionValue</span> = <span style="color: #996633;">$expression</span>.Invoke(<span style="color: #996633;">$item</span>)[0]
<span style="color: #996633;">$hash</span>[<span style="color: #996633;">$hashName</span>] = <span style="color: #996633;">$expressionValue</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$itemProperty</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$item</span>.psobject.Properties)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$itemProperty</span>.Name <span style="color: #333333;">-like</span> <span style="color: #996633;">$property</span>)
{
<span style="color: #996633;">$hash</span>[<span style="color: #996633;">$itemProperty</span>.Name] = <span style="color: #996633;">$itemProperty</span>.Value
}
}
}
}
}
<span style="color: #008800; font-weight: bold;">function</span> TranslateProperties
{
[<span style="color: #008800; font-weight: bold;">cmdletbinding</span>()]
<span style="color: #008800; font-weight: bold;">param</span>(
<span style="color: #003366; font-weight: bold;">[object[]]</span><span style="color: #996633;">$Properties</span>,
<span style="color: #003366; font-weight: bold;">[psobject]</span><span style="color: #996633;">$RealObject</span>,
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Side</span>)
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$Prop</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Properties</span>)
{
<span style="color: #996633;">$propertyHash</span> = <span style="color: #996633;">$Prop</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[hashtable]</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$null</span> <span style="color: #333333;">-ne</span> <span style="color: #996633;">$propertyHash</span>)
{
<span style="color: #996633;">$hashName</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"name"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[string]</span>
<span style="color: #996633;">$expression</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"expression"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[scriptblock]</span>
<span style="color: #996633;">$ScriptString</span> = <span style="color: #996633;">$expression</span>.tostring()
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$ScriptString</span> <span style="color: #333333;">-notmatch</span> <span style="background-color: #fff0f0;">'param\('</span>)
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Property '$HashName'`: Adding param(`$_) to scriptblock '$ScriptString'"</span>
<span style="color: #996633;">$Expression</span> = <span style="color: #003366; font-weight: bold;">[ScriptBlock]</span><span style="background-color: #ffaaaa; color: red;">::</span>Create(<span style="background-color: #fff0f0;">"param(`$_)`n $ScriptString"</span>)
}
<span style="color: #996633;">$Output</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{Name =<span style="color: #996633;">$HashName</span><span style="background-color: #ffaaaa; color: red;">;</span> Expression = <span style="color: #996633;">$Expression</span> }
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Found $Side property hash with name </span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Output.Name)</span><span style="background-color: #fff0f0;">, expression:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Output.Expression | out-string)</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #996633;">$Output</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$ThisProp</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$RealObject</span>.psobject.Properties)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$ThisProp</span>.Name <span style="color: #333333;">-like</span> <span style="color: #996633;">$Prop</span>)
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Found $Side property '</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">ThisProp.Name)</span><span style="background-color: #fff0f0;">'"</span>
<span style="color: #996633;">$ThisProp</span>.Name
}
}
}
}
}
<span style="color: #008800; font-weight: bold;">function</span> WriteJoinObjectOutput(<span style="color: #996633;">$leftItem</span>, <span style="color: #996633;">$rightItem</span>, <span style="color: #996633;">$leftProperties</span>, <span style="color: #996633;">$rightProperties</span>)
{
<span style="color: #996633;">$properties</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{}
AddItemProperties <span style="color: #996633;">$leftItem</span> <span style="color: #996633;">$leftProperties</span> <span style="color: #996633;">$properties</span>
AddItemProperties <span style="color: #996633;">$rightItem</span> <span style="color: #996633;">$rightProperties</span> <span style="color: #996633;">$properties</span>
<span style="color: #007020;">New-Object</span> psobject -Property <span style="color: #996633;">$properties</span>
}
<span style="color: #888888;">#Translate variations on calculated properties. Doing this once shouldn't affect perf too much.</span>
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$Prop</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="background-color: #ffaaaa; color: red;">@</span>(<span style="color: #996633;">$LeftProperties</span> + <span style="color: #996633;">$RightProperties</span>))
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$Prop</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[hashtable]</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$variation</span> <span style="color: #008800; font-weight: bold;">in</span> (<span style="background-color: #fff0f0;">'n'</span>,<span style="background-color: #fff0f0;">'label'</span>,<span style="background-color: #fff0f0;">'l'</span>))
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Name'</span>) )
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$Prop</span>.ContainsKey(<span style="color: #996633;">$variation</span>) )
{
<span style="color: #996633;">$Prop</span>.Add(<span style="background-color: #fff0f0;">'Name'</span>,<span style="color: #996633;">$Prop</span>[<span style="color: #996633;">$Variation</span>])
}
}
}
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Name'</span>) <span style="color: #333333;">-or</span> <span style="color: #996633;">$Prop</span>[<span style="background-color: #fff0f0;">'Name'</span>] <span style="color: #333333;">-like</span> <span style="color: #996633;">$null</span> )
{
Throw <span style="background-color: #fff0f0;">"Property is missing a name`n. This should be in calculated property format, with a Name and an Expression:`n@{Name='Something';Expression={`$_.Something}}`nAffected property:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Prop | out-string)</span><span style="background-color: #fff0f0;">"</span>
}
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Expression'</span>) )
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'E'</span>) )
{
<span style="color: #996633;">$Prop</span>.Add(<span style="background-color: #fff0f0;">'Expression'</span>,<span style="color: #996633;">$Prop</span>[<span style="background-color: #fff0f0;">'E'</span>])
}
}
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Expression'</span>) <span style="color: #333333;">-or</span> <span style="color: #996633;">$Prop</span>[<span style="background-color: #fff0f0;">'Expression'</span>] <span style="color: #333333;">-like</span> <span style="color: #996633;">$null</span> )
{
Throw <span style="background-color: #fff0f0;">"Property is missing an expression`n. This should be in calculated property format, with a Name and an Expression:`n@{Name='Something';Expression={`$_.Something}}`nAffected property:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Prop | out-string)</span><span style="background-color: #fff0f0;">"</span>
}
}
}
<span style="color: #996633;">$leftHash</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{}
<span style="color: #996633;">$rightHash</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{}
<span style="color: #888888;"># Hashtable keys can't be null; we'll use any old object reference as a placeholder if needed.</span>
<span style="color: #996633;">$nullKey</span> = <span style="color: #007020;">New-Object</span> psobject
<span style="color: #996633;">$bound</span> = <span style="color: #996633;">$PSBoundParameters</span>.keys <span style="color: #333333;">-contains</span> <span style="background-color: #fff0f0;">"InputObject"</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$bound</span>)
{
<span style="color: #003366; font-weight: bold;">[System.Collections.ArrayList]</span><span style="color: #996633;">$LeftData</span> = <span style="background-color: #ffaaaa; color: red;">@</span>()
}
}
<span style="color: #008800; font-weight: bold;">Process</span>
{
<span style="color: #888888;">#We pull all the data for comparison later, no streaming</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$bound</span>)
{
<span style="color: #996633;">$LeftData</span> = <span style="color: #996633;">$Left</span>
}
<span style="color: #008800; font-weight: bold;">Else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$Object</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Left</span>)
{
<span style="color: #003366; font-weight: bold;">[void]</span><span style="color: #996633;">$LeftData</span>.add(<span style="color: #996633;">$Object</span>)
}
}
}
<span style="color: #008800; font-weight: bold;">End</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$item</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Right</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$item</span>.<span style="color: #996633;">$RightJoinProperty</span>
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$key</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$nullKey</span>
}
<span style="color: #996633;">$bucket</span> = <span style="color: #996633;">$rightHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$bucket</span>)
{
<span style="color: #996633;">$bucket</span> = <span style="color: #007020;">New-Object</span> System.Collections.ArrayList
<span style="color: #996633;">$rightHash</span>.Add(<span style="color: #996633;">$key</span>, <span style="color: #996633;">$bucket</span>)
}
<span style="color: #996633;">$null</span> = <span style="color: #996633;">$bucket</span>.Add(<span style="color: #996633;">$item</span>)
}
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$item</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$LeftData</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$item</span>.<span style="color: #996633;">$LeftJoinProperty</span>
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$key</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$nullKey</span>
}
<span style="color: #996633;">$bucket</span> = <span style="color: #996633;">$leftHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$bucket</span>)
{
<span style="color: #996633;">$bucket</span> = <span style="color: #007020;">New-Object</span> System.Collections.ArrayList
<span style="color: #996633;">$leftHash</span>.Add(<span style="color: #996633;">$key</span>, <span style="color: #996633;">$bucket</span>)
}
<span style="color: #996633;">$null</span> = <span style="color: #996633;">$bucket</span>.Add(<span style="color: #996633;">$item</span>)
}
<span style="color: #996633;">$LeftProperties</span> = TranslateProperties -Properties <span style="color: #996633;">$LeftProperties</span> -Side <span style="background-color: #fff0f0;">'Left'</span> -RealObject <span style="color: #996633;">$LeftData</span>[0]
<span style="color: #996633;">$RightProperties</span> = TranslateProperties -Properties <span style="color: #996633;">$RightProperties</span> -Side <span style="background-color: #fff0f0;">'Right'</span> -RealObject <span style="color: #996633;">$Right</span>[0]
<span style="color: #888888;">#I prefer ordered output. Left properties first.</span>
<span style="color: #003366; font-weight: bold;">[string[]]</span><span style="color: #996633;">$AllProps</span> = <span style="color: #996633;">$LeftProperties</span>
<span style="color: #888888;">#Handle prefixes, suffixes, and building AllProps with Name only</span>
<span style="color: #996633;">$RightProperties</span> = <span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$RightProp</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$RightProperties</span>)
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> (<span style="color: #996633;">$RightProp</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[Hashtable]</span>))
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Transforming property $RightProp to $Prefix$RightProp$Suffix"</span>
<span style="background-color: #ffaaaa; color: red;">@</span>{
Name=<span style="background-color: #fff0f0;">"$Prefix$RightProp$Suffix"</span>
Expression=<span style="color: #003366; font-weight: bold;">[scriptblock]</span><span style="background-color: #ffaaaa; color: red;">::</span>create(<span style="background-color: #fff0f0;">"param(`$_) `$_.'$RightProp'"</span>)
}
<span style="color: #996633;">$AllProps</span> += <span style="background-color: #fff0f0;">"$Prefix$RightProp$Suffix"</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Skipping transformation of calculated property with name </span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">RightProp.Name)</span><span style="background-color: #fff0f0;">, expression:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">RightProp.Expression | out-string)</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #996633;">$AllProps</span> += <span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$RightProp</span>[<span style="background-color: #fff0f0;">"Name"</span>]
<span style="color: #996633;">$RightProp</span>
}
}
<span style="color: #996633;">$AllProps</span> = <span style="color: #996633;">$AllProps</span> | Select -Unique
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Combined set of properties: </span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">AllProps -join ', ')</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #008800; font-weight: bold;">foreach</span> ( <span style="color: #996633;">$entry</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$leftHash</span>.GetEnumerator() )
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$entry</span>.Key
<span style="color: #996633;">$leftBucket</span> = <span style="color: #996633;">$entry</span>.Value
<span style="color: #996633;">$rightBucket</span> = <span style="color: #996633;">$rightHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$rightBucket</span>)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInLeft'</span> <span style="color: #333333;">-or</span> <span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInBoth'</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$leftItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$leftBucket</span>)
{
WriteJoinObjectOutput <span style="color: #996633;">$leftItem</span> <span style="color: #996633;">$null</span> <span style="color: #996633;">$LeftProperties</span> <span style="color: #996633;">$RightProperties</span> | Select <span style="color: #996633;">$AllProps</span>
}
}
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$leftItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$leftBucket</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$rightItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$rightBucket</span>)
{
WriteJoinObjectOutput <span style="color: #996633;">$leftItem</span> <span style="color: #996633;">$rightItem</span> <span style="color: #996633;">$LeftProperties</span> <span style="color: #996633;">$RightProperties</span> | Select <span style="color: #996633;">$AllProps</span>
}
}
}
}
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInRight'</span> <span style="color: #333333;">-or</span> <span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInBoth'</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$entry</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$rightHash</span>.GetEnumerator())
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$entry</span>.Key
<span style="color: #996633;">$rightBucket</span> = <span style="color: #996633;">$entry</span>.Value
<span style="color: #996633;">$leftBucket</span> = <span style="color: #996633;">$leftHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$leftBucket</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$rightItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$rightBucket</span>)
{
WriteJoinObjectOutput <span style="color: #996633;">$null</span> <span style="color: #996633;">$rightItem</span> <span style="color: #996633;">$LeftProperties</span> <span style="color: #996633;">$RightProperties</span> | Select <span style="color: #996633;">$AllProps</span>
}
}
}
}
}
}
echo <span style="background-color: #fff0f0;">'#!/bin/bash'</span> >.\script.sh
echo <span style="background-color: #fff0f0;">"IFS=`$'\n'"</span> >>.\script.sh
echo <span style="background-color: #fff0f0;">'for line in `grep -v \^\# /etc/oratab |grep -v agent `'</span> >>.\script.sh
echo <span style="background-color: #fff0f0;">'do'</span> >>.\script.sh
echo <span style="background-color: #fff0f0;">"VERS=``echo `$line |awk -F: '{print `$2}' ;``" >>.\script.sh</span>
<span style="background-color: #fff0f0;">echo "</span>DB=``echo `$line |awk <span style="color: #333333;">-F</span><span style="background-color: #ffaaaa; color: red;">:</span> <span style="background-color: #fff0f0;">'{print `$1`}'</span> <span style="background-color: #ffaaaa; color: red;">;</span>``" >>.\script.sh
echo <span style="background-color: #fff0f0;">'ACTUALVERS=`export ORACLE_HOME=${VERS};export LD_LIBRARY_PATH=${VERS}/lib;${VERS}/bin/sqlplus -v |tail -2|head -1`'</span> >>.\script.sh
echo <span style="background-color: #fff0f0;">'RUNNING=`ps -ef |grep pmon_${DB} |grep -v grep |wc -l`'</span> >>.\script.sh
echo <span style="background-color: #fff0f0;">'echo YOOHOO,"$DB","$VERS","$ACTUALVERS","$RUNNING"'</span> >>.\script.sh
echo <span style="background-color: #fff0f0;">'done'</span> >>.\script.sh
<span style="color: #996633;">$script:FINRESULT</span> = <span style="background-color: #ffaaaa; color: red;">@</span>()
<span style="color: #996633;">$script:FINRESULT2</span> = <span style="background-color: #ffaaaa; color: red;">@</span>()
<span style="color: #996633;">$date</span> = <span style="color: #007020;">Get-Date</span> -format dd-MM-yyyy
<span style="color: #996633;">$Subscriptions</span> = <span style="color: #007020;">Get-AzSubscription</span>
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$sub</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Subscriptions</span>) {
<span style="color: #007020;">Get-AzSubscription</span> -SubscriptionName <span style="color: #996633;">$sub</span>.Name | <span style="color: #007020;">Set-AzContext</span>
<span style="color: #996633;">$vms</span> = Search-AzGraph -Query <span style="background-color: #fff0f0;">'Resources |where type == "microsoft.compute/virtualmachines" | where name contains "ORA" and properties.storageProfile.osDisk.osType=="Linux" and name !startswith "aks" and isnull(plan) and properties.extended.instanceView.powerState.displayStatus=="VM running" |project resourceGroup,name'</span> -First 5000 -subscription <span style="color: #996633;">$sub</span>.id
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$vm</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$vms</span>) {
<span style="color: #996633;">$output</span> = <span style="color: #007020;">Invoke-AzVMRunCommand</span> -ResourceGroupName <span style="color: #996633;">$vm</span>.resourceGroup -VMName <span style="color: #996633;">$vm</span>.name -ScriptPath <span style="background-color: #fff0f0;">".\script.sh"</span> -CommandId RunShellScript -Verbose
<span style="color: #996633;">$pattern</span> = <span style="background-color: #fff0f0;">'^YOOHOO'</span>
<span style="color: #888888;"># split data into lines, and process:</span>
<span style="color: #996633;">$lines</span> = <span style="color: #996633;">$output</span>.Value.Message.Split(<span style="background-color: #fff0f0;">"`n"</span>)
<span style="color: #996633;">$result</span> = <span style="color: #996633;">$lines</span> | <span style="color: #007020;">Select-Object</span> -Skip 2 | <span style="color: #008800; font-weight: bold;">ForEach</span>-Object {
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$_</span> <span style="color: #333333;">-match</span> <span style="color: #996633;">$pattern</span>)
{
<span style="color: #996633;">$_</span> | <span style="color: #007020;">ConvertFrom-Csv</span> -Header dummy,dbname,path,version,running
}
}
<span style="color: #996633;">$result</span>| <span style="color: #007020;">Add-Member</span> -MemberType NoteProperty <span style="background-color: #fff0f0;">"Server"</span> -Value <span style="color: #996633;">$vm</span>.name
<span style="color: #996633;">$result</span>| <span style="color: #007020;">Add-Member</span> -MemberType NoteProperty <span style="background-color: #fff0f0;">"sub"</span> -Value <span style="color: #996633;">$sub</span>.name
<span style="color: #996633;">$result</span>| <span style="color: #007020;">Add-Member</span> -MemberType NoteProperty <span style="background-color: #fff0f0;">"FLAG"</span> -Value <span style="background-color: #fff0f0;">"ORA"</span>
<span style="color: #996633;">$result</span>| <span style="color: #007020;">Add-Member</span> -MemberType NoteProperty <span style="background-color: #fff0f0;">"RG"</span> -Value <span style="color: #996633;">$vm</span>.resoureGroup
<span style="color: #996633;">$result</span>| <span style="color: #007020;">Add-Member</span> -MemberType NoteProperty <span style="background-color: #fff0f0;">"joiner"</span> -Value <span style="background-color: #fff0f0;">"</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">vm.name.tolower()</span><span style="background-color: #fff0f0;">)</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">vm.resourceGroup.tolower()</span><span style="background-color: #fff0f0;">)</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">sub.Id.tolower()</span><span style="background-color: #fff0f0;">)"</span>
<span style="color: #996633;">$result</span> |ft
<span style="color: #996633;">$script:FINRESULT</span> +=<span style="color: #996633;">$result</span>
}
}
<span style="color: #996633;">$ARG</span> = Search-AzGraph -Query <span style="background-color: #fff0f0;">'(Resources |where type == "microsoft.compute/virtualmachines" </span>
<span style="background-color: #fff0f0;">| project vmname=name,resourceGroup=toupper(resourceGroup),vmsize=properties.hardwareProfile.vmSize,sub=toupper(subscriptionId),os=properties.storageProfile.osDisk.osType,shut=tags.AutoShutdownSchedule,vmid=id,offer=properties.storageProfile.imageReference.offer,sku=properties.storageProfile.imageReference.sku</span>
<span style="background-color: #fff0f0;">| join kind=inner (ResourceContainers |where type=="microsoft.resources/subscriptions/resourcegroups" |project name=toupper(name),itera=tags.IteraplanID,own=tags.Owner,email=tags.Owner_email,sub2=toupper(subscriptionId) ) on $left.resourceGroup==$right.name and $left.sub==$right.sub2) </span>
<span style="background-color: #fff0f0;">| join kind=leftouter (ResourceContainers | where type=="microsoft.resources/subscriptions" | project SubName=name,sub=toupper(subscriptionId)) on $left.sub==$right.sub |</span>
<span style="background-color: #fff0f0;">project vmname=tolower(vmname),resourceGroup=tolower(resourceGroup),sub=tolower(sub),SubName,vmsize,os,itera,shut,own,email,vmid,offer,sku,joiner=strcat(vmname,resourceGroup,sub)'</span> -First 5000 -subscription <span style="color: #996633;">$Subscriptions</span>.id
<span style="color: #996633;">$JOINRESULT</span> = <span style="color: #007020;">Join-Object</span> -Left <span style="color: #996633;">$script:FINRESULT</span> -Right <span style="color: #996633;">$ARG</span> -LeftJoinProperty joiner -RightJoinProperty joiner -Type OnlyIfInBoth
<span style="color: #996633;">$MACHINES</span> = <span style="color: #007020;">get-azvmsize</span> -location westeurope
<span style="color: #996633;">$FINALJOIN</span> = <span style="color: #007020;">Join-Object</span> -Left <span style="color: #996633;">$JOINRESULT</span> -Right <span style="color: #996633;">$MACHINES</span> -LeftJoinProperty vmsize -RightJoinProperty Name -Type OnlyIfInBoth
<span style="color: #996633;">$FINALJOIN</span> | <span style="color: #007020;">Export-Csv</span> -Path .\temporacle.csv -QuoteFields <span style="background-color: #fff0f0;">"dbname"</span>,<span style="background-color: #fff0f0;">"shut"</span>,<span style="background-color: #fff0f0;">"own"</span>,<span style="background-color: #fff0f0;">"email"</span>
<span style="color: #996633;">$Context</span> = <span style="color: #007020;">New-AzStorageContext</span> -StorageAccountName <span style="background-color: #fff0f0;">"account here"</span> -StorageAccountKey <span style="background-color: #fff0f0;">"key here"</span>
<span style="color: #007020;">Set-AzStorageBlobContent</span> -Context <span style="color: #996633;">$Context</span> -Container <span style="background-color: #fff0f0;">"azure"</span> <span style="color: #333333;">-File</span> <span style="background-color: #fff0f0;">"temporacle.csv"</span> -Blob <span style="background-color: #fff0f0;">"azureoracle-$date.csv"</span> -Force
</pre></div>
<p><br /></p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-40337647001725807082021-02-23T14:49:00.000-08:002021-02-23T14:49:11.890-08:00Harry Potter and the Chat Bot<p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-cz9DJLfWxdw/YDQ8BuTEUFI/AAAAAAAADp0/0PuLntFZNWICTPvIA3RL3IbSXkSOZ2CHgCLcBGAsYHQ/s640/harry-potter-5055509_640.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="427" data-original-width="640" src="https://1.bp.blogspot.com/-cz9DJLfWxdw/YDQ8BuTEUFI/AAAAAAAADp0/0PuLntFZNWICTPvIA3RL3IbSXkSOZ2CHgCLcBGAsYHQ/s320/harry-potter-5055509_640.jpg" width="320" /></a></div><p><br /></p><p>So kids birthday in lockdown - what you gonna do? Can't really go anywhere or do anything - so how to make something exciting and memorable? Well I'll tell you what I did - I created them a chatbot. That sounds like worst idea ever right?</p><p>Well I made it a bit more exciting than that - what I built is a 'treasure hunt' powered by a chatbot. I combined some of the easy to use features we get from Azure for chatbots, with a Harry Potter set of Funko miniature figures (using an advent calendar like the below...) </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-XTOwTimumss/YDWGJpzRJ7I/AAAAAAAADuI/xAd1KTMvlKMpH_WAeA1ikIl1--768Hd1wCLcBGAsYHQ/s1600/advent.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1600" height="320" src="https://1.bp.blogspot.com/-XTOwTimumss/YDWGJpzRJ7I/AAAAAAAADuI/xAd1KTMvlKMpH_WAeA1ikIl1--768Hd1wCLcBGAsYHQ/s320/advent.png" /></a></div><br /><p>With Telegram for the front end to the chatbot to make it look nice. All I then needed to do was think of some clever places to hide the figures - and there you have it. Best Birthday Party game ever...</p><p>So the basic game idea is you have to guess 22 of the character names from the Harry Potter film - if you enter a correct one into the chatbot you get a picture back with a clue to where the Funko figure is hidden. The aim being to find all the figures. Once you've found all the figures some of them have a letter stuck on them - these need to be rearranged to make a magic word that when entered will reveal a final clue to reveal the treasure location.</p><p>The end result looks like the below - this is the Telegram front end to it</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-xy6O9zM2Fbk/YDRCPpjPAmI/AAAAAAAADt4/M8L6o9bljw4Ch6atSGV6gaXFtZsp3gafgCLcBGAsYHQ/s762/RPReplay_Final1614036774.gif" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="762" data-original-width="352" height="320" src="https://1.bp.blogspot.com/-xy6O9zM2Fbk/YDRCPpjPAmI/AAAAAAAADt4/M8L6o9bljw4Ch6atSGV6gaXFtZsp3gafgCLcBGAsYHQ/s320/RPReplay_Final1614036774.gif" /></a></div><p></p>So - how did I build this? Well you might be surprised to know that actually the chatbot part is incredibly easy (and free) - what took the time was coming up with the places to hide the figures and then 'clues' to be able to find them. No point making it too easy right?<div><br /></div><div>So lets walk through step by step how to create the above.<br /> </div><div>First thing to do is go to <a href="https://qnamaker.ai">https://qnamaker.ai</a> - this site will allow you to create a 'knowledge base' - all this contains is just a set of questions (in my case character names) and answers (the picture and the clue). So we navigate to the site and login.<br /><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-2itw_M04h9I/YDQ8IGdzKKI/AAAAAAAADp4/Vdw_BCrtlNMbgjZI8ecZzuz_OuuPXoFygCLcBGAsYHQ/s1739/potter1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="698" data-original-width="1739" src="https://1.bp.blogspot.com/-2itw_M04h9I/YDQ8IGdzKKI/AAAAAAAADp4/Vdw_BCrtlNMbgjZI8ecZzuz_OuuPXoFygCLcBGAsYHQ/s320/potter1.PNG" width="320" /></a></div><div><br /></div><div>Then we click create a Q+A service in step 1 of the wizard - this will now create something to back the knowledge base.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-gnOecFmxDu4/YDQ8IH2C4uI/AAAAAAAADp8/PfkJHtR9Zo8PZ1pXGhp3kccQwV91XOmlQCLcBGAsYHQ/s1845/potter2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="642" data-original-width="1845" src="https://1.bp.blogspot.com/-gnOecFmxDu4/YDQ8IH2C4uI/AAAAAAAADp8/PfkJHtR9Zo8PZ1pXGhp3kccQwV91XOmlQCLcBGAsYHQ/s320/potter2.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">This then takes us to the Azure portal where we have to fill in some basics about this Q+A service that we will provision - note in the screenshot below I'm using some of the cheapest pay for features as the free ones were already being used - in your case though this is likely not the case and you can just choose free.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-zH_-EJ3m18Y/YDQ8IBty1YI/AAAAAAAADqA/xYj-g51YL98K0tgQEXHEz801aZ2Tko_jwCLcBGAsYHQ/s1486/potter3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="955" data-original-width="1486" src="https://1.bp.blogspot.com/-zH_-EJ3m18Y/YDQ8IBty1YI/AAAAAAAADqA/xYj-g51YL98K0tgQEXHEz801aZ2Tko_jwCLcBGAsYHQ/s320/potter3.PNG" width="320" /></a></div><div><br /></div><div>Once you've populated all the fields then click create</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-5ZjMESHJP20/YDQ8IxIAk_I/AAAAAAAADqE/PpgOyzA5vgE82DW86dWDPAifqqrzYWBRwCLcBGAsYHQ/s924/potter4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="924" data-original-width="509" height="320" src="https://1.bp.blogspot.com/-5ZjMESHJP20/YDQ8IxIAk_I/AAAAAAAADqE/PpgOyzA5vgE82DW86dWDPAifqqrzYWBRwCLcBGAsYHQ/s320/potter4.PNG" /></a></div><div><br /></div><div>Wait for a couple of minutes and then you should see that its completed.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-UNJZ1Dn-t1Q/YDQ8JOx3nBI/AAAAAAAADqI/EKHBXZXHr1wMrSmcUj_WtcZHPp6b5oP1gCLcBGAsYHQ/s1619/potter5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="589" data-original-width="1619" src="https://1.bp.blogspot.com/-UNJZ1Dn-t1Q/YDQ8JOx3nBI/AAAAAAAADqI/EKHBXZXHr1wMrSmcUj_WtcZHPp6b5oP1gCLcBGAsYHQ/s320/potter5.PNG" width="320" /></a></div><div><br /></div><div>Now when we return the the qnamaker website and move to step 2 we should be able to select the qna service we just created in Azure.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-DyP714PYDsE/YDQ8JMofVyI/AAAAAAAADqM/wOo4IOnvUBYPG5B0bRxNDzN27jG0ijP8wCLcBGAsYHQ/s1466/potter6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="613" data-original-width="1466" src="https://1.bp.blogspot.com/-DyP714PYDsE/YDQ8JMofVyI/AAAAAAAADqM/wOo4IOnvUBYPG5B0bRxNDzN27jG0ijP8wCLcBGAsYHQ/s320/potter6.PNG" width="320" /></a></div><div>You may have to wait a couple of minutes for the service to be full initialised - otherwise you'll get the error below</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-DzbKkIYo-t4/YDQ8JaWevHI/AAAAAAAADqQ/uYjEiYAvFR08_wFm9jY07J1DDL-9CPUVwCLcBGAsYHQ/s665/potter7.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="665" src="https://1.bp.blogspot.com/-DzbKkIYo-t4/YDQ8JaWevHI/AAAAAAAADqQ/uYjEiYAvFR08_wFm9jY07J1DDL-9CPUVwCLcBGAsYHQ/s320/potter7.PNG" width="320" /></a></div><div><br /></div><div>So after coming back couple of mins later we try again - we can now choose the language we want</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-SS2WSAQEKm0/YDQ8JiAY_GI/AAAAAAAADqU/qFXxUQrx4lYl8VjmEJlPVhX4s-QHnL42gCLcBGAsYHQ/s1126/potter8.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="576" data-original-width="1126" src="https://1.bp.blogspot.com/-SS2WSAQEKm0/YDQ8JiAY_GI/AAAAAAAADqU/qFXxUQrx4lYl8VjmEJlPVhX4s-QHnL42gCLcBGAsYHQ/s320/potter8.PNG" width="320" /></a></div><div><br /></div><div>Then give the knowledge base a name</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-128yqvgkWSo/YDQ8JhCUjgI/AAAAAAAADqY/tCBUPvsIgGoZpp5bRPnILi_Jc0B6HeM6wCLcBGAsYHQ/s1345/potter9.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="244" data-original-width="1345" src="https://1.bp.blogspot.com/-128yqvgkWSo/YDQ8JhCUjgI/AAAAAAAADqY/tCBUPvsIgGoZpp5bRPnILi_Jc0B6HeM6wCLcBGAsYHQ/s320/potter9.PNG" width="320" /></a></div><br /><p>Then click create and it will go away and provision it.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-wf7ntUWcHZo/YDQ8PTumVbI/AAAAAAAADqc/LzTms-shmLcK3HprOqhRxAfUkW0LsRomgCLcBGAsYHQ/s1321/potter10.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="271" data-original-width="1321" src="https://1.bp.blogspot.com/-wf7ntUWcHZo/YDQ8PTumVbI/AAAAAAAADqc/LzTms-shmLcK3HprOqhRxAfUkW0LsRomgCLcBGAsYHQ/s320/potter10.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">A minute or so after this screen</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-2CAjARpfXzQ/YDQ8PQV6LtI/AAAAAAAADqg/s5g346ZUFeQaU17vto10jSknoczB-wzLQCLcBGAsYHQ/s740/potter11.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="370" data-original-width="740" src="https://1.bp.blogspot.com/-2CAjARpfXzQ/YDQ8PQV6LtI/AAAAAAAADqg/s5g346ZUFeQaU17vto10jSknoczB-wzLQCLcBGAsYHQ/s320/potter11.PNG" width="320" /></a></div><div><br /></div>The knowledge base is then created and I can start to add the pairs of questions and answers to power the treasure hunt</div><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-M-udCMWuqtw/YDQ8PTBJYDI/AAAAAAAADqk/EL-A1JYBbogvObfv_KfFjYo5XJaZ-Nr2QCLcBGAsYHQ/s1725/potter12.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="272" data-original-width="1725" src="https://1.bp.blogspot.com/-M-udCMWuqtw/YDQ8PTBJYDI/AAAAAAAADqk/EL-A1JYBbogvObfv_KfFjYo5XJaZ-Nr2QCLcBGAsYHQ/s320/potter12.PNG" width="320" /></a></div><div><br /></div><div>I'm adding 2 in the example below to illustrate the point - the rest are then just different characters and clues - so you just work your way through the list you have. So on the left hand side we have the character name and the right the response with the clue.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-m6T_1JM-EO0/YDQ8QM4Pz6I/AAAAAAAADqo/00_0YELs8HYAuJC84QfhmObHuzUrI5mLgCLcBGAsYHQ/s1712/potter13.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="621" data-original-width="1712" src="https://1.bp.blogspot.com/-m6T_1JM-EO0/YDQ8QM4Pz6I/AAAAAAAADqo/00_0YELs8HYAuJC84QfhmObHuzUrI5mLgCLcBGAsYHQ/s320/potter13.PNG" width="320" /></a></div><div><br /></div><div>To make the reply/clue look nice I embed a link to an image of the character as shown below - so I just refer to an internet source of that image. The client program (Telegram) will do the job ok displaying this for me.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-HvJxuK2Za8Q/YDQ8QZmpSAI/AAAAAAAADqs/c0ivekogqlAQDejaN2QUbNoO2GaI9CLTgCLcBGAsYHQ/s1155/potter14.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="252" data-original-width="1155" src="https://1.bp.blogspot.com/-HvJxuK2Za8Q/YDQ8QZmpSAI/AAAAAAAADqs/c0ivekogqlAQDejaN2QUbNoO2GaI9CLTgCLcBGAsYHQ/s320/potter14.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>The other custom question/answer I need to create is for the initial 'instructions' of what the hunt is all about. To do that I create a special question called /start (this is the message a Telegram chatbot sends as its first communication) - this then triggers the reply with the instructions - including a link to an embedded gif to make it look nice in the display</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/--5yCk8jlXiI/YDQ8QTNcuzI/AAAAAAAADqw/nYaFjC8PNmg8f7cIu_4kvKuVEVOhfkTuwCLcBGAsYHQ/s1652/potter15.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="705" data-original-width="1652" src="https://1.bp.blogspot.com/--5yCk8jlXiI/YDQ8QTNcuzI/AAAAAAAADqw/nYaFjC8PNmg8f7cIu_4kvKuVEVOhfkTuwCLcBGAsYHQ/s320/potter15.PNG" width="320" /></a></div><div><br /></div><div>Once I've create all the KB entries I then want to 'save and train' the qnamaker (for the purpose of this its essentially just a save)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-lCo_3CjWtQc/YDQ8QpK62QI/AAAAAAAADq0/d8htL5VGVQ8Yo0N_Lo-p69jxG0qX_vXZACLcBGAsYHQ/s1877/potter16.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="140" data-original-width="1877" src="https://1.bp.blogspot.com/-lCo_3CjWtQc/YDQ8QpK62QI/AAAAAAAADq0/d8htL5VGVQ8Yo0N_Lo-p69jxG0qX_vXZACLcBGAsYHQ/s320/potter16.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-2UGyHX8vkQg/YDQ8RFZ9h-I/AAAAAAAADq4/BnXC0xgsygAWvOI8nnUqLRZYz16PR0mPACLcBGAsYHQ/s1467/potter17.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="457" data-original-width="1467" src="https://1.bp.blogspot.com/-2UGyHX8vkQg/YDQ8RFZ9h-I/AAAAAAAADq4/BnXC0xgsygAWvOI8nnUqLRZYz16PR0mPACLcBGAsYHQ/s320/potter17.PNG" width="320" /></a></div><div>Wait for that to run.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-TOJDTqmqNFE/YDQ8RlSo_lI/AAAAAAAADq8/lebI2Pq26JUHNwOWqxTfU0L0uNASaVl1gCLcBGAsYHQ/s715/potter18.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="715" src="https://1.bp.blogspot.com/-TOJDTqmqNFE/YDQ8RlSo_lI/AAAAAAAADq8/lebI2Pq26JUHNwOWqxTfU0L0uNASaVl1gCLcBGAsYHQ/s320/potter18.PNG" width="320" /></a></div><div><br /></div>Once that is done we get this screen back confirming all is created.</div><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-TnqDG5Td0RY/YDQ8SD_JSlI/AAAAAAAADrA/AXBgDUV8ju8xaO8-RimnalSOkh04bwXbgCLcBGAsYHQ/s1406/potter19.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="687" data-original-width="1406" src="https://1.bp.blogspot.com/-TnqDG5Td0RY/YDQ8SD_JSlI/AAAAAAAADrA/AXBgDUV8ju8xaO8-RimnalSOkh04bwXbgCLcBGAsYHQ/s320/potter19.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>Now lets do a quick test just to make sure the basics of the question/answer are working - I click the test button at the top right. If I then enter a character name - you can see it replies with the text. In the simple web client the embedded image does not get fully displayed.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-MJJFtkZiv4E/YDQ8ZTC4XeI/AAAAAAAADrU/MgqL27p9a4wgewp0mGuElIioZswG4357wCLcBGAsYHQ/s954/potter20.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="933" data-original-width="954" src="https://1.bp.blogspot.com/-MJJFtkZiv4E/YDQ8ZTC4XeI/AAAAAAAADrU/MgqL27p9a4wgewp0mGuElIioZswG4357wCLcBGAsYHQ/s320/potter20.PNG" width="320" /></a></div><div><br /></div>So looking good - now to actually create the chatbot service in Azure - to do this we click the create bot button that displays after we click the save/train button again.</div><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-p05Rc90YGMU/YDQ8ZeO6tLI/AAAAAAAADrM/7Dium7BKrG8rEEc74evrqz2JTnEvKvyggCLcBGAsYHQ/s1128/potter21.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="242" data-original-width="1128" src="https://1.bp.blogspot.com/-p05Rc90YGMU/YDQ8ZeO6tLI/AAAAAAAADrM/7Dium7BKrG8rEEc74evrqz2JTnEvKvyggCLcBGAsYHQ/s320/potter21.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>This then sends us to the Azure portal again where we fill in some basics about where we want the chatbot created.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-n8xBxhvm7As/YDQ8ZcBMZ-I/AAAAAAAADrQ/1ce21z5u45w383VBQTiYuiWCKOvSiZOAQCLcBGAsYHQ/s933/potter22.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="933" data-original-width="723" height="320" src="https://1.bp.blogspot.com/-n8xBxhvm7As/YDQ8ZcBMZ-I/AAAAAAAADrQ/1ce21z5u45w383VBQTiYuiWCKOvSiZOAQCLcBGAsYHQ/s320/potter22.PNG" /></a></div><div><br /></div><div>After that is created we click on go to resource and we are presented with the following screen. Here we want to click the channels option - this is where we pair the chatbot to the Telegram service. By the way I chose Telegram as it had a native support here - you can no doubt build it in other things - but Telegram was the easy option.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Mwazu4qkIc0/YDQ8Z5TWoaI/AAAAAAAADrY/azO3rRgCyxM-fDIE3OHyHy1CDac_eNjkACLcBGAsYHQ/s1827/potter23.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="901" data-original-width="1827" src="https://1.bp.blogspot.com/-Mwazu4qkIc0/YDQ8Z5TWoaI/AAAAAAAADrY/azO3rRgCyxM-fDIE3OHyHy1CDac_eNjkACLcBGAsYHQ/s320/potter23.PNG" width="320" /></a></div><div><br /></div><div>So in the channels screen choose Telegram </div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-u7AB5DrVwC8/YDQ8aBzmaSI/AAAAAAAADrc/mAL3anbX0yErVld-ZHnB4ZkFMMbthJxPgCLcBGAsYHQ/s1462/potter24.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="698" data-original-width="1462" src="https://1.bp.blogspot.com/-u7AB5DrVwC8/YDQ8aBzmaSI/AAAAAAAADrc/mAL3anbX0yErVld-ZHnB4ZkFMMbthJxPgCLcBGAsYHQ/s320/potter24.PNG" width="320" /></a></div><div><br /></div>Then on the Telegram page we just need to link it to out Telegram API key - now we didn't create that yet - so lets jump out to my mobile to do that and then come back to the portal in a minute</div><div><br /></div><div>Now once you have the Telegram app installed you need to add something called the Botfather as a 'contact'. I search for it as shown below.</div><div><br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-HMMSKs9xbYk/YDQ-bRys32I/AAAAAAAADso/MTSqCZZfHzwnnRfXf04x7mndw9hxYM1WACLcBGAsYHQ/s2436/image001.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2436" data-original-width="1125" height="320" src="https://1.bp.blogspot.com/-HMMSKs9xbYk/YDQ-bRys32I/AAAAAAAADso/MTSqCZZfHzwnnRfXf04x7mndw9hxYM1WACLcBGAsYHQ/s320/image001.png" /></a></div><div><br /></div><div><br /></div><div>This is then a chatbot interface to create other chatbots - a slightly odd concept at first - anyway it seems to work well. So i start with the /start option (actually in the app it just does this for you). <div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-1EjzGpw1Gro/YDQ-bSfUqpI/AAAAAAAADsk/MXfslJ7VJN8wQlIg7Kd2uLmwiqf2DcbogCLcBGAsYHQ/s640/image002.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-1EjzGpw1Gro/YDQ-bSfUqpI/AAAAAAAADsk/MXfslJ7VJN8wQlIg7Kd2uLmwiqf2DcbogCLcBGAsYHQ/s320/image002.png" /></a></div></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>Then i say /newbot</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-4kM3p5LeKNw/YDQ-bcs3niI/AAAAAAAADss/P1ifuRPCLAEeg67tzQgiQeTolwbDXIS5ACLcBGAsYHQ/s640/image003.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-4kM3p5LeKNw/YDQ-bcs3niI/AAAAAAAADss/P1ifuRPCLAEeg67tzQgiQeTolwbDXIS5ACLcBGAsYHQ/s320/image003.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">I then follow the prompt to give it a name - the end result of which comes out below including the all important API key (just outside of the screenshot below). This is the 'password' that we configure between the telegram bot and the Azure bot service to allow the two to communicate.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-RThTzMcTqfw/YDQ-cLW4ZlI/AAAAAAAADsw/wVCgbflACZ48cGu2zxHyfpY_WBw46LaKwCLcBGAsYHQ/s640/image004.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-RThTzMcTqfw/YDQ-cLW4ZlI/AAAAAAAADsw/wVCgbflACZ48cGu2zxHyfpY_WBw46LaKwCLcBGAsYHQ/s320/image004.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">So taking that API key we paste it into the channel screen for Telegram in the chat bot.</div></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-iBvw4JulKug/YDQ8aahA3dI/AAAAAAAADrg/Jt1hLMT2oUAzm8tFtg8XK4nw07-9qM2sgCLcBGAsYHQ/s1203/potter25.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="550" data-original-width="1203" src="https://1.bp.blogspot.com/-iBvw4JulKug/YDQ8aahA3dI/AAAAAAAADrg/Jt1hLMT2oUAzm8tFtg8XK4nw07-9qM2sgCLcBGAsYHQ/s320/potter25.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">Click save - and we should get a nice green bar</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Op9zLQpyDnQ/YDQ8ac74EMI/AAAAAAAADrk/A_3TRZy7jhwzSca5z-5L6KI6IaDbJAqXQCLcBGAsYHQ/s1635/potter26.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="491" data-original-width="1635" src="https://1.bp.blogspot.com/-Op9zLQpyDnQ/YDQ8ac74EMI/AAAAAAAADrk/A_3TRZy7jhwzSca5z-5L6KI6IaDbJAqXQCLcBGAsYHQ/s320/potter26.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div>If i then connect to the new bot i just configured in Telegram (you'll need to search for the @botnameyoujustcreated and add it) you will see that its already working - initially I see this screen - I then tap start</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-G82yk4TyAwA/YDQ-cig9AhI/AAAAAAAADs0/RWwDB36uhfknaKogfCTYatJro7KHuOu_gCLcBGAsYHQ/s640/image006.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-G82yk4TyAwA/YDQ-cig9AhI/AAAAAAAADs0/RWwDB36uhfknaKogfCTYatJro7KHuOu_gCLcBGAsYHQ/s320/image006.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">This then sends the /start question to my bot - which then replies with my instructions and the now working video clip.</div><div class="separator" style="clear: both; text-align: center;"><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-mGbhSngZN3s/YDQ-dN96jJI/AAAAAAAADs4/itOi3XhfJtkVFlIHaDHXngaZLHPJlQqRQCLcBGAsYHQ/s640/image008.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-mGbhSngZN3s/YDQ-dN96jJI/AAAAAAAADs4/itOi3XhfJtkVFlIHaDHXngaZLHPJlQqRQCLcBGAsYHQ/s320/image008.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div style="text-align: left;">I can then send a character name and this too is expanded into the picture and the clue</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-mXolTYYt3js/YDQ-djpubRI/AAAAAAAADs8/hhRom9vcNcEJ7lFHlWtv9s48MoLKu_8FQCLcBGAsYHQ/s640/image009.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-mXolTYYt3js/YDQ-djpubRI/AAAAAAAADs8/hhRom9vcNcEJ7lFHlWtv9s48MoLKu_8FQCLcBGAsYHQ/s320/image009.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><span style="text-align: left;">So looking good so far - we're nearly there - what's left is what happens if they type something that isn't a valid character - well by default you get something like the below 'No QnAMaker answers found' - that doesn't sound great and I want to change that</span></div></div></div><br /></div><div style="text-align: center;"><a href="https://1.bp.blogspot.com/-RPBiTse_4Cw/YDQ-lgwMhZI/AAAAAAAADtA/6KtxgARxCDcks7MScVVTIMei4P6nbEBbQCLcBGAsYHQ/s640/image010.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-RPBiTse_4Cw/YDQ-lgwMhZI/AAAAAAAADtA/6KtxgARxCDcks7MScVVTIMei4P6nbEBbQCLcBGAsYHQ/s320/image010.png" /></a></div><br />Now this next bit start to feel a bit like you are actually having to code something (up until now that really hasn't been the case) - this is however pretty simple and you can ignore any of the files and things you see when doing this change. So to get to be able to do this we navigate to the chatbot app screen as shown below (this app was created by the earlier steps - however you won't have had to get to this screen before). We click on app service editor<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-CzTQ2muIT3Y/YDQ8a0g9qOI/AAAAAAAADrs/gICxo4ZwVNIFcpdr9M9HcjpJD7zbxeTXgCLcBGAsYHQ/s1830/potter28.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="866" data-original-width="1830" src="https://1.bp.blogspot.com/-CzTQ2muIT3Y/YDQ8a0g9qOI/AAAAAAAADrs/gICxo4ZwVNIFcpdr9M9HcjpJD7zbxeTXgCLcBGAsYHQ/s320/potter28.PNG" width="320" /></a></div><div><br /></div><div>That then opens this blade in which we click the link</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-BJMPw1daeE0/YDQ8awYpLzI/AAAAAAAADrw/62PF9F1ACxU_27abpxrUvCPdKKa55BzIQCLcBGAsYHQ/s939/potter29.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="167" data-original-width="939" src="https://1.bp.blogspot.com/-BJMPw1daeE0/YDQ8awYpLzI/AAAAAAAADrw/62PF9F1ACxU_27abpxrUvCPdKKa55BzIQCLcBGAsYHQ/s320/potter29.PNG" width="320" /></a></div><div><br /></div><div>This then takes us into a browser based editor - we need to navigate and find the file QnAMakerBaseDialog.cs and specifically the enter highlighted below - this controls what the default response message will be.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-wC1-xDnfxvk/YDQ8ivJxCTI/AAAAAAAADr8/VDlC9XMbK20c655-t3e-ObQF0LbZebHCQCLcBGAsYHQ/s1572/potter30.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="775" data-original-width="1572" src="https://1.bp.blogspot.com/-wC1-xDnfxvk/YDQ8ivJxCTI/AAAAAAAADr8/VDlC9XMbK20c655-t3e-ObQF0LbZebHCQCLcBGAsYHQ/s320/potter30.PNG" width="320" /></a></div><div><br /></div><div>I update the value to something more appropriate and make sure the autosave has kicked in (see saved message at top right)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ozwM0nty0z4/YDQ8imHSd-I/AAAAAAAADsA/ppvcMl8VPsk5LMmppBobxv18mwiq23w4QCLcBGAsYHQ/s1476/potter31.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="584" data-original-width="1476" src="https://1.bp.blogspot.com/-ozwM0nty0z4/YDQ8imHSd-I/AAAAAAAADsA/ppvcMl8VPsk5LMmppBobxv18mwiq23w4QCLcBGAsYHQ/s320/potter31.PNG" width="320" /></a></div><div><br /></div><div>Now the source code is updated I need to recompile the chatbot - to do that I click on the console icon on the right hand side and type build.cmd into that console window which will trigger that to happen.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-nWFgn8dsLTE/YDQ8iuOo7DI/AAAAAAAADr4/b7aqW6DB-Mkc1Buy5Nwm9vohoFTAHyfPwCLcBGAsYHQ/s1672/potter32.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="545" data-original-width="1672" src="https://1.bp.blogspot.com/-nWFgn8dsLTE/YDQ8iuOo7DI/AAAAAAAADr4/b7aqW6DB-Mkc1Buy5Nwm9vohoFTAHyfPwCLcBGAsYHQ/s320/potter32.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">After a few seconds that process is completed and the new app deployed</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-UlX0EIrR8WM/YDQ8jFYfNfI/AAAAAAAADsE/F6K3aPtJtMgXf3-xzEXw0nAiHrJtb8u6gCLcBGAsYHQ/s1660/potter33.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="720" data-original-width="1660" src="https://1.bp.blogspot.com/-UlX0EIrR8WM/YDQ8jFYfNfI/AAAAAAAADsE/F6K3aPtJtMgXf3-xzEXw0nAiHrJtb8u6gCLcBGAsYHQ/s320/potter33.PNG" width="320" /></a></div><br />If I now send a message that has no answer i get the new default reply<br /><p style="text-align: center;"><a href="https://1.bp.blogspot.com/-bwngSyjPcI4/YDQ-lo1QvEI/AAAAAAAADtE/Hu3ktlv5WUcD7MiwP7YH_1vXqac73RnOwCLcBGAsYHQ/s640/image011.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="295" height="320" src="https://1.bp.blogspot.com/-bwngSyjPcI4/YDQ-lo1QvEI/AAAAAAAADtE/Hu3ktlv5WUcD7MiwP7YH_1vXqac73RnOwCLcBGAsYHQ/s320/image011.png" /></a><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">The final finishing touch is to then add the special key word that the kids will find once they solve the Anagram using the letters stuck to some of the characters. In this case the word is 'Alohamora' - and the reply it sends is a cryptic clue of emojis. I make use of the emoji text keywords that Telegram understands - in my case :golf: and :boot: - revealing that the treasure (their birthday presents) is hidden in the boot (trunk for my American friends) of my car ( a VW Golf).</p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-UnFa8HmleY8/YDQ8aiJnAWI/AAAAAAAADro/dt6P2kkkjDAaUko_GOpXdLX5fTq-GqdewCLcBGAsYHQ/s1714/potter27.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="593" data-original-width="1714" src="https://1.bp.blogspot.com/-UnFa8HmleY8/YDQ8aiJnAWI/AAAAAAAADro/dt6P2kkkjDAaUko_GOpXdLX5fTq-GqdewCLcBGAsYHQ/s320/potter27.PNG" width="320" /></a></div></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">So the emoji comes out as below</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-NDhlTBxkdLU/YDWCwhQ5gCI/AAAAAAAADuA/cCqUNUz4bTYQv6VjMef1pGf4aKfYaYpsACLcBGAsYHQ/s640/golfboot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="526" height="320" src="https://1.bp.blogspot.com/-NDhlTBxkdLU/YDWCwhQ5gCI/AAAAAAAADuA/cCqUNUz4bTYQv6VjMef1pGf4aKfYaYpsACLcBGAsYHQ/s320/golfboot.png" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">At that point we're good to go - here's a few pics of the treasure hunt in progress - from starting off and reading the instructions.</div><div class="separator" style="clear: both; text-align: center;"><br /></div></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-OYtto5IcxLc/YDQ-vCRKqwI/AAAAAAAADtg/wT97T8KrSwITZRQ_sPPDB-RRZSNL-QFYwCLcBGAsYHQ/s2016/image001.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1512" data-original-width="2016" src="https://1.bp.blogspot.com/-OYtto5IcxLc/YDQ-vCRKqwI/AAAAAAAADtg/wT97T8KrSwITZRQ_sPPDB-RRZSNL-QFYwCLcBGAsYHQ/s320/image001.jpg" width="320" /></a></div><div><br /></div><div>To finding the initial Harry Potter character</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-2WlQH-NUQ2Q/YDQ-vMUzQaI/AAAAAAAADtk/IY3ny8Xr2O42HvZE-1DXiB79STTt_fKqQCLcBGAsYHQ/s2016/image002.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1512" data-original-width="2016" src="https://1.bp.blogspot.com/-2WlQH-NUQ2Q/YDQ-vMUzQaI/AAAAAAAADtk/IY3ny8Xr2O42HvZE-1DXiB79STTt_fKqQCLcBGAsYHQ/s320/image002.jpg" width="320" /></a></div><div><br /></div><div>And the more well hidden (frozen in a block of Ice) Severus Snape</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-PKKg-Mnhyk0/YDQ-vO0D57I/AAAAAAAADtc/ANVmpYb1g5wv6dYx6zouFhTIfbTxJQWRQCLcBGAsYHQ/s2016/image003.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2016" data-original-width="1512" height="320" src="https://1.bp.blogspot.com/-PKKg-Mnhyk0/YDQ-vO0D57I/AAAAAAAADtc/ANVmpYb1g5wv6dYx6zouFhTIfbTxJQWRQCLcBGAsYHQ/s320/image003.jpg" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>The kids really loved it - hopefully with the instructions above its easy to recreate for anyone with some basic IT skills. It can be adapted to work with with different Funko sets - or can be anything that the kids are into - doesn't even have to be something you go out and buy.</div><div><br /></div><div>Hope this gives someone else inspiration to do something similar!<br /><br /><p><br /></p><p><br /></p></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com1tag:blogger.com,1999:blog-302298286928742422.post-53594626578025776832021-02-17T14:32:00.000-08:002021-02-17T14:32:10.440-08:00The Azure Beehive<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-kB9InkjLscg/YC2ZP-GoqiI/AAAAAAAADpk/H5yfx_GvGA4cmWuVhl8cBDdoUX1L7e2mACLcBGAsYHQ/s640/bee-2519772_640.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="640" height="320" src="https://1.bp.blogspot.com/-kB9InkjLscg/YC2ZP-GoqiI/AAAAAAAADpk/H5yfx_GvGA4cmWuVhl8cBDdoUX1L7e2mACLcBGAsYHQ/s320/bee-2519772_640.png" /></a></div><br /><p></p><p>This is a short post about my new favourite feature in the Azure portal - Workbooks. These allow you to create some nice interactive visuals based on a variety of datasets. Microsoft themselves have some of the portal screens based on this technology and it seems to be something they are really investing in. It kind of takes the dashboards concept to the next level.</p><p>What I'm going to show here is a nice 'honeycomb' visualisation which I think is quite effective.</p><p>For my example I'm going to use a real world thing I've just built - in this case I want to show the main users of traditional netapp fileservices. To discover that I'm making use of the data fetched by the Azure insights agent - this collects metrics to make the performance and map screens work in VM blades of the portal. In my case I also have a number of 'non-Azure' servers configured with Azure ARC - these are also configured with that same MMA agent fetching the data. The data that is being used here, that the agent fetched, is stored across a few log analytics workspaces - I now just want to query across all of them and create a nice visual. </p><p>So here is how I did that:</p><p>To create a workbook don't fall into the trap of going to the workbook screen and trying to add it there - there is no add button....</p><p>Instead navigate to the Azure monitor page, go to the workbooks option and then click create an empty workbook.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-p3mOBFG5Lsw/YC2Q3Gw-PNI/AAAAAAAADo0/ydECyxFnfC0j4c-X-cpcnqtsDxh-_tkrQCLcBGAsYHQ/s1383/workbook1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="545" data-original-width="1383" src="https://1.bp.blogspot.com/-p3mOBFG5Lsw/YC2Q3Gw-PNI/AAAAAAAADo0/ydECyxFnfC0j4c-X-cpcnqtsDxh-_tkrQCLcBGAsYHQ/s320/workbook1.PNG" width="320" /></a></div><div><br /></div><div>Once you have done that we then need to add an element to it</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-N3Nd7XNfH9w/YC2Q2urdZBI/AAAAAAAADow/CKDAsxaKiB0zTlYcfuwKzV2aQy2Daz7HQCLcBGAsYHQ/s1074/workbook2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="520" data-original-width="1074" src="https://1.bp.blogspot.com/-N3Nd7XNfH9w/YC2Q2urdZBI/AAAAAAAADow/CKDAsxaKiB0zTlYcfuwKzV2aQy2Daz7HQCLcBGAsYHQ/s320/workbook2.PNG" width="320" /></a></div><div><br /></div><div>In my case i want to add a 'query'</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-9y-HZecptb4/YC2Q2tnFqaI/AAAAAAAADos/JANiq0OnP5Iq5Rbvdm3LbGv5ZUMmlImNwCLcBGAsYHQ/s555/workbook3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="491" data-original-width="555" src="https://1.bp.blogspot.com/-9y-HZecptb4/YC2Q2tnFqaI/AAAAAAAADos/JANiq0OnP5Iq5Rbvdm3LbGv5ZUMmlImNwCLcBGAsYHQ/s320/workbook3.PNG" width="320" /></a></div><div><br /></div><div>We are no presented with this screen - in which I want to choose the 'logs' option and then 'log analytics', I can then select one or more workspaces for the query to run across (in my case quite a few more) and then I just paste in my pre-written query. I then click the run query button to check it's returning what I want.</div><div><br /></div><div>What's nice here is that it's enabled the multi workspace query - this is possible directly in log analytics by use of the union keyword but it's a little messy to write - especially when you have lots of workspaces - this screen keeps it nice and clean.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-JGWZPYtzAYQ/YC2Q3e8HvWI/AAAAAAAADo4/2DRRgjf9dYMqSAURB9aS5fGGaIYgvkRzQCLcBGAsYHQ/s1538/workbook4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="648" data-original-width="1538" src="https://1.bp.blogspot.com/-JGWZPYtzAYQ/YC2Q3e8HvWI/AAAAAAAADo4/2DRRgjf9dYMqSAURB9aS5fGGaIYgvkRzQCLcBGAsYHQ/s320/workbook4.PNG" width="320" /></a></div><br /></div><div>The query I'm running is shown below - so I'm looking in the VMconnection 'bucket' of data that the agent has been pushing the data into - then I'm saying I'm only interested in traffic to certain IP's (the netapp filers) on port 445 (the file services port). I'm then summing up the amount of data sent across and grouping it by the computer and the netappfiler destination ip. So I get a nice summary of how much data each computer sent to each filer.</div><div><br /></div><div><!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">VMConnection <span style="color: #333333;">|</span> <span style="color: #008800; font-weight: bold;">where</span> DestinationIp <span style="color: #008800; font-weight: bold;">in</span>(<span style="background-color: #fff0f0;">'ip'</span>,<span style="background-color: #fff0f0;">'ip'</span>,<span style="background-color: #fff0f0;">'ip'</span>,<span style="background-color: #fff0f0;">'ip'</span>) <span style="color: #008800; font-weight: bold;">and</span> DestinationPort <span style="color: #333333;">==</span><span style="background-color: #fff0f0;">'445'</span>
<span style="color: #333333;">|</span>summarize <span style="color: #008800; font-weight: bold;">sum</span>(BytesSent)<span style="color: #333333;">+</span><span style="color: #008800; font-weight: bold;">sum</span>(BytesReceived) <span style="color: #008800; font-weight: bold;">by</span> Computer,DestinationIp <span style="color: #333333;">|</span>sort <span style="color: #008800; font-weight: bold;">by</span> Column1 <span style="color: #008800; font-weight: bold;">desc</span>
</pre></div>
</div><div><br /></div>So that on its own is quite useful - but I want to make it look nice.<div><br /></div><div>To do that I switch the visualisation to graph as shown below<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-mPMYziHn0Aw/YC2Q3mgnWwI/AAAAAAAADo8/7alSUnddZ0k6xzlCcqKYVSLMH25lmCbbwCLcBGAsYHQ/s863/workbook5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="863" data-original-width="850" height="320" src="https://1.bp.blogspot.com/-mPMYziHn0Aw/YC2Q3mgnWwI/AAAAAAAADo8/7alSUnddZ0k6xzlCcqKYVSLMH25lmCbbwCLcBGAsYHQ/s320/workbook5.PNG" /></a></div><div><br /></div><div>I then choose the 'hive clusters' option and set the nodeid to the computer column (this is the value used for each hexagon in the output) and I group by the destination ip ( so I get a distinct honeycomb for each filer)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-LgRmXTBIj04/YC2Q4KiyJHI/AAAAAAAADpA/SxhbsufHZesVUNxwjqfdOs81j6zB34RWgCLcBGAsYHQ/s637/workbook6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="474" data-original-width="637" src="https://1.bp.blogspot.com/-LgRmXTBIj04/YC2Q4KiyJHI/AAAAAAAADpA/SxhbsufHZesVUNxwjqfdOs81j6zB34RWgCLcBGAsYHQ/s320/workbook6.PNG" width="320" /></a></div><div><br /></div><div>Next I update the centre content column to switch the number format to bytes and the max significant digits to 0 (screenshot shows 3 but it should be 0...)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-EamO8puW0E8/YC2Q4GSeGeI/AAAAAAAADpI/OWxu6Pej6_wBopK1zLH3kjawnTtgXO3zQCLcBGAsYHQ/s725/workbook7.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="725" data-original-width="725" height="320" src="https://1.bp.blogspot.com/-EamO8puW0E8/YC2Q4GSeGeI/AAAAAAAADpI/OWxu6Pej6_wBopK1zLH3kjawnTtgXO3zQCLcBGAsYHQ/s320/workbook7.PNG" /></a></div><div><br /></div><div>Then at the bottom of that screen i go to the colouring section and choose 'heatmap', I derive the colouring option from the value of the column1 (the sum of the bytes) field and I want the colour palette to go from green to red. I leave the min and max value to auto so it defines them itself based on the data.</div><div><br /></div><div>What that then means is that the highest value comes up as dark red changing in colour down to green for the lowest value in the dataset. That way I get a nice visual representation of the biggest users.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ITK862sKx4U/YC2Q4AhHwbI/AAAAAAAADpE/lcaoGthFPJMveMrEAQb2R93UNewVp0nwwCLcBGAsYHQ/s654/workbook8.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="375" data-original-width="654" src="https://1.bp.blogspot.com/-ITK862sKx4U/YC2Q4AhHwbI/AAAAAAAADpE/lcaoGthFPJMveMrEAQb2R93UNewVp0nwwCLcBGAsYHQ/s320/workbook8.PNG" width="320" /></a></div><div><br /></div><div>The end result looking something like the below - you can see the private ip (redacted) of the filer at the top of each cluster. The honeycomb below then shows the total bytes per computer, I've removed the computer name from the actual display (which spoils the screenshot below a little, but it was too much data to redact....) - but each hexagon would have the computer name in it if you followed the steps above.</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-NHym47sej1s/YC2Q4gsv88I/AAAAAAAADpM/CIS5QzAW8JMM_gjshsceX-kKbk66jsU4ACLcBGAsYHQ/s1840/workbook9.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="584" data-original-width="1840" height="127" src="https://1.bp.blogspot.com/-NHym47sej1s/YC2Q4gsv88I/AAAAAAAADpM/CIS5QzAW8JMM_gjshsceX-kKbk66jsU4ACLcBGAsYHQ/w400-h127/workbook9.PNG" width="400" /></a></div><br /><p>There you go - a nice way of visualising the data</p><p>There are loads more things you can do with workbooks, making then interactive, having tabs, having multiple datasets joined within them, linking out to external data via rest API - they are really powerful - more posts to come!</p></div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-59051049219088841012021-02-15T06:04:00.004-08:002021-02-15T06:04:27.834-08:00Some Azure Resource Graph examples for SQL Server <div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-6Cct77zbOPw/YCp_KNPnabI/AAAAAAAADog/WKiunZuxF6gX_sYBYZTotfNncA3pPia5wCLcBGAsYHQ/s640/boy-1299640_640.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="504" data-original-width="640" src="https://1.bp.blogspot.com/-6Cct77zbOPw/YCp_KNPnabI/AAAAAAAADog/WKiunZuxF6gX_sYBYZTotfNncA3pPia5wCLcBGAsYHQ/s320/boy-1299640_640.png" width="320" /></a></div><br /><p>I've been using resource graph a lot recently and found it hard to find good examples for some of the more fiddly types of queries - primarily where I want to join different sets of data - or manipulate the results</p><p>The two examples below illustrate a couple of techniques that are maybe useful to know and give some example to copy/paste from.</p><p>First up is this one which retrieves details of all the SQL Server installations on IaaS we have (so SQL done in the traditional way)</p><p>The trick here is that I need data from 3 different containers within resource graph so I have to join them. What I found was tricky is that resource graph is case sensitive - and the MS data is often inconsistent between the different containers - so I'm having to toupper() some of the data to get it to match. I also have to join on multiple columns (as names can duplicated - only combination of name/rg/sub is unique). I also pull out some of the embedded properties in the tags attribute.</p><p><br /></p><!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">(Resources |where type == <span style="background-color: #fff0f0;">"microsoft.sqlvirtualmachine/sqlvirtualmachines"</span>
| extend lic=properties.sqlServerLicenseType,
mgmt=properties.sqlManagement,
image=properties.sqlImageOffer,
sku=properties.sqlImageSku,sub=toupper(subscriptionId) | extend rg=toupper(resourceGroup) |project-away id,type,tenantId,kind)
| join kind=inner (ResourceContainers |where type==<span style="background-color: #fff0f0;">"microsoft.resources/subscriptions/resourcegroups"</span> |project name,itera=tags.IteraplanID,own=tags.Owner,email=tags.Owner_email,sub2=toupper(subscriptionId),nm=toupper(name) ) on <span style="color: #996633;">$left</span>.rg==<span style="color: #996633;">$right</span>.nm and <span style="color: #996633;">$left</span>.sub==<span style="color: #996633;">$right</span>.sub2
| join kind=inner (ResourceContainers | where type==<span style="background-color: #fff0f0;">"microsoft.resources/subscriptions"</span> | project SubName=name,sub=toupper(subscriptionId)) on <span style="color: #996633;">$left</span>.sub==<span style="color: #996633;">$right</span>.sub
|project-away subscriptionId,managedBy,plan,properties,tags,identity,zones,extendedLocation,sub,name1,sub2,sub1
</pre></div>
<div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">That gives you something like this (you likely want to change the tag names in the query for your environment)</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-tjKeBv-dlfU/YCp-ONO0u7I/AAAAAAAADoU/akRouIplz5875e1wSdjHXuKeZK6Lvhl-ACLcBGAsYHQ/s1771/arg1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="279" data-original-width="1771" src="https://1.bp.blogspot.com/-tjKeBv-dlfU/YCp-ONO0u7I/AAAAAAAADoU/akRouIplz5875e1wSdjHXuKeZK6Lvhl-ACLcBGAsYHQ/s320/arg1.PNG" width="320" /></a></div><br /><p>Next up is very similar but for Azure SQL (the full PaaS one) - most of the code is very similar - the only additional tricky part is the use of the extract function - here I'm using a regular expression match to pull the server name out of the data - I could find almost no example of someone using that in resource graph.</p><!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;">(Resources |where type == <span style="background-color: #fff0f0;">"microsoft.sql/servers/databases"</span> |extend sub=toupper(subscriptionId)
| join kind=inner (ResourceContainers |where type==<span style="background-color: #fff0f0;">"microsoft.resources/subscriptions/resourcegroups"</span> |project name,itera=tags.IteraplanID,own=tags.Owner,email=tags.Owner_email,sub2=toupper(subscriptionId) ) on <span style="color: #996633;">$left</span>.resourceGroup==<span style="color: #996633;">$right</span>.name and <span style="color: #996633;">$left</span>.sub==<span style="color: #996633;">$right</span>.sub2)
| join kind=inner (ResourceContainers | where type==<span style="background-color: #fff0f0;">"microsoft.resources/subscriptions"</span> | project SubName=name,sub=toupper(subscriptionId)) on <span style="color: #996633;">$left</span>.sub==<span style="color: #996633;">$right</span>.sub
|extend collation=properties.collation, skuname=properties.currentSku.name,capacity=properties.currentSku.capacity,tier=properties.currentSku.tier,family=properties.currentSku.family
,server=extract(<span style="background-color: #fff0f0;">"servers/(([a-z]|-|[0-9])+)"</span>,1,id)
|project-away tenantId,type,subscriptionId,managedBy,sku,plan,properties,tags,identity,zones,extendedLocation,sub,name1,sub2,apiVersion,aliases,id
</pre></div>
<p><br /></p><p>That gives something like the below - same comment as the above for tags.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-N4hINO2jrek/YCp-N9mys_I/AAAAAAAADoQ/FSfdQCeJomsa_iJkr5ALPWCvGpDL68AgQCLcBGAsYHQ/s1795/arg2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="197" data-original-width="1795" src="https://1.bp.blogspot.com/-N4hINO2jrek/YCp-N9mys_I/AAAAAAAADoQ/FSfdQCeJomsa_iJkr5ALPWCvGpDL68AgQCLcBGAsYHQ/s320/arg2.PNG" width="320" /></a></div><p></p><p><br /></p><p>Hopefully that gives you a head start if you are building anything like this.</p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com2tag:blogger.com,1999:blog-302298286928742422.post-62972878233992638212021-02-13T12:42:00.001-08:002021-02-13T12:42:50.950-08:00Serverless CMDB extract from Azure<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-l0diw_Ph-FI/YCg5cGQg8KI/AAAAAAAADoE/xND-O9f2dNgb1BfH0MvCapN7bfAmolIKwCLcBGAsYHQ/s640/dark-4077614_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" src="https://1.bp.blogspot.com/-l0diw_Ph-FI/YCg5cGQg8KI/AAAAAAAADoE/xND-O9f2dNgb1BfH0MvCapN7bfAmolIKwCLcBGAsYHQ/s320/dark-4077614_640.jpg" width="320" /></a></div><br /><div><br /></div><div><br /></div>With a now large and well established Azure estate we have the need to effectively report on what we 'have' in Azure. Now we're a few years in we already have to start thinking about dealing with os versions that are now old. To be able to do this we need a reliable source of the information.<div><br /></div><div>When I started to look into this I thought initially there is a simple solution for this surely - Azure 'knows' what we have I just need to extract that. However when you start to get into the detail you start to find there are gaps in the dataset.....</div><div><br /></div><div>Azure resource graph is the obvious choice to use - but this doesn't have the 'actual' os version info - all it records is the image/sku that the machine as created from - this may be cryptic (or wrong if os version updates have been done) or even null if custom images were used.</div><div><br /></div><div>The heartbeat from the MMA agent will record some of the data in log analytics (but again not totally complete) - and the MMA agent won't work on all operating systems (NVA's and other appliances for example)</div><div><br /></div><div>The vminsights agent does record everything for the os's that it works on - but that is even more limited than the MMA agent.</div><div><br /></div><div>So what to do?</div><div><br /></div><div>Well the best we seem to be able to do is make use of powershell and specifically the get-azvm command - with the -status switch. This only works specifically against a VM directly - so we have to execute it for every VM (thanks to Marcel from Microsoft for the hint on this).</div><div><br /></div><div>As long as the machine is running this will return the 'real' os and version for any type of VM (even appliances such as netapp, checkpoint and the like). For us this still represents not full coverage as we have many machines that are only running at certain times of the day or are shut down and only used exceptionally. However a combination of the powershell data and the resource graph data gives us the coverage we need to enable upgrade planning.<br /><p></p><div class="separator" style="clear: both; text-align: center;"></div><p></p>I mocked up a powershell script to do all the above, however I wanted to build this in a way that didn't rely on a machine to have to execute the powershell on - this is a waste of a VM and the 'old' way to do things.</div><div><br /></div><div>The solution - a powershell functionapp - and that's what I'll explain now.</div><div><br /></div><div>So first up I create a new resource group - I do this up front as we have policies that enforce certain tags have to be in place - this enables our governance on ownership and cost reporting - if I do the RG in line with creating the functionapp i can't specify these.</div><div><br /></div><div>So create an RG wherever you like<br /><br /><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-mrs4E6BjfAE/YCfKSDZjfFI/AAAAAAAADlQ/c-UyXdEUI1UWe6isa3Mz9UGnHNuXa4nmACLcBGAsYHQ/s1537/cmdbrgcreate.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="658" data-original-width="1537" src="https://1.bp.blogspot.com/-mrs4E6BjfAE/YCfKSDZjfFI/AAAAAAAADlQ/c-UyXdEUI1UWe6isa3Mz9UGnHNuXa4nmACLcBGAsYHQ/s320/cmdbrgcreate.PNG" width="320" /></a></div></div><div><br /></div><div>Set tags if you need to (may not be mandatory in your env) </div><div><br /></div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-51x_IySkcec/YCfKSJCQpqI/AAAAAAAADlM/rcKxnjwnK0sGzhgCBTFa-AqTg1CXzZv6ACLcBGAsYHQ/s1190/cmdbrgcreate2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="611" data-original-width="1190" src="https://1.bp.blogspot.com/-51x_IySkcec/YCfKSJCQpqI/AAAAAAAADlM/rcKxnjwnK0sGzhgCBTFa-AqTg1CXzZv6ACLcBGAsYHQ/s320/cmdbrgcreate2.PNG" width="320" /></a></div></div><div><br /></div><div>Wait for that to run (only few secs)</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-pYI8k_UlF8k/YCfKiyRwupI/AAAAAAAADlc/L7y_sE_ZtEMH1dDnNvPlbfPWpv1Ql777wCLcBGAsYHQ/s541/cmdb3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="279" data-original-width="541" src="https://1.bp.blogspot.com/-pYI8k_UlF8k/YCfKiyRwupI/AAAAAAAADlc/L7y_sE_ZtEMH1dDnNvPlbfPWpv1Ql777wCLcBGAsYHQ/s320/cmdb3.PNG" width="320" /></a></div><div><br /></div><div>Next we create a functionapp in that resource group I just added - this is a 'bucket' to put our actual code in. You can see I choose specifically powershell as the language I'll be using - quite a few other languages are available now and you could achieve the same thing in all of them - but for me at least powershell is much easier than the other options</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ptWUtILWOXo/YCfKjEE_vUI/AAAAAAAADlg/vEA1XTX2GhUgXv7-G1BPQYyLm5Tb5zqiQCLcBGAsYHQ/s1045/cmdb4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="808" data-original-width="1045" src="https://1.bp.blogspot.com/-ptWUtILWOXo/YCfKjEE_vUI/AAAAAAAADlg/vEA1XTX2GhUgXv7-G1BPQYyLm5Tb5zqiQCLcBGAsYHQ/s320/cmdb4.PNG" width="320" /></a></div><div><br /></div><div>Next we choose a few other options , I just accepted default storage account, the os currently has to be windows for powershell (which seemed odd when powershell will run on linux). The final one here is quite important based on the size of your dataset as I found out later on when building this. By default the consumption based one (which is what I wanted to use) has a runtime time limit of 10 minutes - any longer than that and its killed. In our case we have so many machines that the script runtime is around 30 minutes so I had to choose a premium service instead which allows longer timeouts. If your script can run in less than 10 mins then just stick with the cheaper consumption model.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-czmwkAUXJnk/YCfKi_j7eMI/AAAAAAAADlk/LKckD7CKsQUsg4kYSdNhq2SY7bW8IGH4wCLcBGAsYHQ/s1048/cmdb5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="721" data-original-width="1048" src="https://1.bp.blogspot.com/-czmwkAUXJnk/YCfKi_j7eMI/AAAAAAAADlk/LKckD7CKsQUsg4kYSdNhq2SY7bW8IGH4wCLcBGAsYHQ/s320/cmdb5.PNG" width="320" /></a></div><div><br /></div><div>Then we get quick summary and click create.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-_Uajk9Ax7rU/YCfKjifWSiI/AAAAAAAADlo/_MPU6L7N780t4w8n6GeW4vHaZvFFZ_JLwCLcBGAsYHQ/s904/cmdb6.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="904" data-original-width="689" height="320" src="https://1.bp.blogspot.com/-_Uajk9Ax7rU/YCfKjifWSiI/AAAAAAAADlo/_MPU6L7N780t4w8n6GeW4vHaZvFFZ_JLwCLcBGAsYHQ/s320/cmdb6.PNG" /></a></div><div><br /></div><div>And wait for completion - again only a few secs.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-fTdYzHozcIU/YCfKj2mPt4I/AAAAAAAADls/U2qrLQZCGgAYP4ouVtcgQwg1nFXKHzZegCLcBGAsYHQ/s1297/cmdb7.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="448" data-original-width="1297" src="https://1.bp.blogspot.com/-fTdYzHozcIU/YCfKj2mPt4I/AAAAAAAADls/U2qrLQZCGgAYP4ouVtcgQwg1nFXKHzZegCLcBGAsYHQ/s320/cmdb7.PNG" width="320" /></a></div><div><br /></div><div>Now we have the function app container we have to create the actual function where our code will sit. There are a few options on how the code can get executed - you can see in the screenshot below. I chose to use the timer trigger - so the script just executes on a schedule - using standard cron style attributes. The other common choice would be a http trigger - this allows you to call the function via a rest api call - very handy for interfacing to a logic app if you want to build some sort of workflow around this extract process.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-zDdWxe4um5I/YCfKj8XFUJI/AAAAAAAADlw/qqoeUiPsLMw78JCUh3gFtTRy-KaS-vCiACLcBGAsYHQ/s1889/cmdb8.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="920" data-original-width="1889" src="https://1.bp.blogspot.com/-zDdWxe4um5I/YCfKj8XFUJI/AAAAAAAADlw/qqoeUiPsLMw78JCUh3gFtTRy-KaS-vCiACLcBGAsYHQ/s320/cmdb8.PNG" width="320" /></a></div><div><br /></div><div>Click add and wait a few seconds and then its created.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-YGX88OeDEe8/YCfKkNi65MI/AAAAAAAADl0/Y5m9FkglSjEObybRcFm4WgbFd43LyHShgCLcBGAsYHQ/s1344/cmdb9.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="345" data-original-width="1344" src="https://1.bp.blogspot.com/-YGX88OeDEe8/YCfKkNi65MI/AAAAAAAADl0/Y5m9FkglSjEObybRcFm4WgbFd43LyHShgCLcBGAsYHQ/s320/cmdb9.PNG" width="320" /></a></div><br /><div>The first thing i want to do is make use of what of the really nice features in here and that is to enable a managed identity - you can see that in the screenshot below. This creates an app registration in azure active directory for the function itself - what that means is that we can then grant rights specifically to the function itself - no messing around with somehow trying to authenticate as service accounts and having to handle all that - it's just done for you.</div><div><br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-JizQzOKgq3A/YCfK7WstkcI/AAAAAAAADmM/6URZnLSD4A8d5AJ9KaDU4rNTNAAwX1KLQCLcBGAsYHQ/s821/cmdb10.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="668" data-original-width="821" src="https://1.bp.blogspot.com/-JizQzOKgq3A/YCfK7WstkcI/AAAAAAAADmM/6URZnLSD4A8d5AJ9KaDU4rNTNAAwX1KLQCLcBGAsYHQ/s320/cmdb10.PNG" width="320" /></a></div><div><br /></div><div>After few secs you'll see it created.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-7IfMrLhAFbc/YCfK7dn8qSI/AAAAAAAADmU/Hhsbm7yuNjY30gg1oARj35Q5h2GKUz4CACLcBGAsYHQ/s641/cmdb11.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="492" data-original-width="641" src="https://1.bp.blogspot.com/-7IfMrLhAFbc/YCfK7dn8qSI/AAAAAAAADmU/Hhsbm7yuNjY30gg1oARj35Q5h2GKUz4CACLcBGAsYHQ/s320/cmdb11.PNG" width="320" /></a></div><div><br /></div><div>To enable the script to be able to read all the details across 'everything' i need to grant it the reader role everywhere - the easiest way to do that is to grant it at the tenant root group level and let it cascade down to everywhere - i do that in this screen. Of course you can do it however it works for you - the function app just needs read rights on whatever you want to report on.</div><div><br /></div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-y5bPK-qTw9I/YCfLQbxwndI/AAAAAAAADnM/YTFgGyp564sOPVeeQL-tKkEG9MFaApD1gCLcBGAsYHQ/s1898/cmdb21.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="721" data-original-width="1898" src="https://1.bp.blogspot.com/-y5bPK-qTw9I/YCfLQbxwndI/AAAAAAAADnM/YTFgGyp564sOPVeeQL-tKkEG9MFaApD1gCLcBGAsYHQ/s320/cmdb21.PNG" width="320" /></a></div><br /></div><div>Right we're getting closer - next up is to make the additional powershell modules needed for my script available to the function app. Now from what I read this is meant to be automatic for any modules starting with az. - for me at least this only worked for az.accounts for some reason - so instead I had to add the modules directly to the function app.</div><div><br /></div><div>Now I'm sure there is more than one way to do this (and maybe a better way) but the method below worked for me.</div><div><br /></div><div>First up launch the kudu tool from the link shown below from within the function app.</div><div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-JNAjE3uUzQk/YCfK7fZobyI/AAAAAAAADmQ/KJxpg0rLAfAv3gc_hDrM8a5Bwvl-beg2gCLcBGAsYHQ/s1229/cmdb12.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="567" data-original-width="1229" src="https://1.bp.blogspot.com/-JNAjE3uUzQk/YCfK7fZobyI/AAAAAAAADmQ/KJxpg0rLAfAv3gc_hDrM8a5Bwvl-beg2gCLcBGAsYHQ/s320/cmdb12.PNG" width="320" /></a></div><div><br /></div><div>Then go to the console link</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-8CypHZ33eMU/YCfK8Er_fUI/AAAAAAAADmY/HIj9TfvQzawNWKsCcszoEFcQX4xG1IcnACLcBGAsYHQ/s1726/cmdb13.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="950" data-original-width="1726" src="https://1.bp.blogspot.com/-8CypHZ33eMU/YCfK8Er_fUI/AAAAAAAADmY/HIj9TfvQzawNWKsCcszoEFcQX4xG1IcnACLcBGAsYHQ/s320/cmdb13.PNG" width="320" /></a></div><div><br /></div><div>Then create the modules directory under site/wwwroot - this is where the module code needs to be stored</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-jnoj_XMQAvY/YCfK8fjjx-I/AAAAAAAADmc/kFcE8u90hoEVf5auu0q8xMBVFls-NNFUQCLcBGAsYHQ/s1174/cmdb14.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="427" data-original-width="1174" src="https://1.bp.blogspot.com/-jnoj_XMQAvY/YCfK8fjjx-I/AAAAAAAADmc/kFcE8u90hoEVf5auu0q8xMBVFls-NNFUQCLcBGAsYHQ/s320/cmdb14.PNG" width="320" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Then on your laptop download the modules locally (this wouldn't work in the console above which is annoying). In the case below I end up with 2 directories containing the module files.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-mWa2dLN74os/YCfK8kFI47I/AAAAAAAADmg/1o6UL63bqUcXqmxXhqgOVtWQP1-tlxEQACLcBGAsYHQ/s699/cmdb15.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="80" data-original-width="699" src="https://1.bp.blogspot.com/-mWa2dLN74os/YCfK8kFI47I/AAAAAAAADmg/1o6UL63bqUcXqmxXhqgOVtWQP1-tlxEQACLcBGAsYHQ/s320/cmdb15.PNG" width="320" /></a></div><div><br /></div><div>Now zip up those directories (screenshot also shows az.accounts but that should be automatic as mentioned above)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-91-Bu5YGsFA/YCfK8zsHKtI/AAAAAAAADmk/wD6DSOrjMjsabwPNrMqaGZwPAv0SEp4PgCLcBGAsYHQ/s896/cmdb16.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="211" data-original-width="896" src="https://1.bp.blogspot.com/-91-Bu5YGsFA/YCfK8zsHKtI/AAAAAAAADmk/wD6DSOrjMjsabwPNrMqaGZwPAv0SEp4PgCLcBGAsYHQ/s320/cmdb16.PNG" width="320" /></a></div><div><br /></div><div>Then drag that zip file into the kudu console (making sure you are set to the modules directory)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-5anlN8EHWWU/YCfK9R1s69I/AAAAAAAADmo/ozv5HmF4JhEDGUpoXckt775rYPTLT7nCACLcBGAsYHQ/s1257/cmdb17.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="466" data-original-width="1257" src="https://1.bp.blogspot.com/-5anlN8EHWWU/YCfK9R1s69I/AAAAAAAADmo/ozv5HmF4JhEDGUpoXckt775rYPTLT7nCACLcBGAsYHQ/s320/cmdb17.PNG" width="320" /></a></div><div><br /></div><div>That will then upload and unzip the files ready to use.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-1GUjqGFFtGQ/YCfK98I0tuI/AAAAAAAADms/-5w9uWiKjX4zNuNCnjYZ0uVl0NmLWCzOgCLcBGAsYHQ/s1188/cmdb18.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="785" data-original-width="1188" src="https://1.bp.blogspot.com/-1GUjqGFFtGQ/YCfK98I0tuI/AAAAAAAADms/-5w9uWiKjX4zNuNCnjYZ0uVl0NmLWCzOgCLcBGAsYHQ/s320/cmdb18.PNG" width="320" /></a></div><div><br /></div><div>Now we need to go back to the function app main window and go into the code + test window</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Tf4NajKOIOE/YCfK-aW1n2I/AAAAAAAADmw/KTRhKTN2FXc5ULNRDf6ws6CMl1QAbfLSgCLcBGAsYHQ/s1307/cmdb19.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="344" data-original-width="1307" src="https://1.bp.blogspot.com/-Tf4NajKOIOE/YCfK-aW1n2I/AAAAAAAADmw/KTRhKTN2FXc5ULNRDf6ws6CMl1QAbfLSgCLcBGAsYHQ/s320/cmdb19.PNG" width="320" /></a></div><br /><div><br /></div><div>Now this is the magic powershell script you've all been waiting for - we then paste this in - quick couple of comments on it:</div><div><br /></div><div>1) I make use of the join-object function - that's from <a href="https://github.com/RamblingCookieMonster/PowerShell/blob/master/Join-Object.ps1">https://github.com/RamblingCookieMonster/PowerShell/blob/master/Join-Object.ps1</a>- this is a nice function that enables you to join 2 powershell arrays in the same way you might join 2 tables in a relational database.</div><div><br /></div><div>2) I run the get-azvm script to loop through all the subscriptions and store the data in an array</div><div><br /></div><div>3) I run a resource graph query to get the results to fill in blanks for shut down machines</div><div><br /></div><div>4) I join the 2 datasets based on a concatenation of vmname/rgname/subscription - to make sure we are not going to get duplicates.</div><div><br /></div><div>5) I output the result to a storage account - this code needs improving to remove the hardcoded account key - that's possible just a little more fiddly - it's annoying that storage account doesn't just support the identity being passed straight in.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>6) note that the resource graph query can be case sensitive in parts hence the use of some conversions - there are also going to be tags you don't have - so remove those</div><div><br /></div><div>7) first few lines vary depending on the type of function you choose - the below is for the timer one - just grab the code below that if you choose a different type.</div><div><br /></div><div><!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #888888;"># Input bindings are passed in via param block.</span>
<span style="color: #008800; font-weight: bold;">param</span>(<span style="color: #996633;">$Timer</span>)
<span style="color: #888888;"># Get the current universal time in the default string format.</span>
<span style="color: #996633;">$currentUTCtime</span> = (<span style="color: #007020;">Get-Date</span>).ToUniversalTime()
<span style="color: #888888;"># The 'IsPastDue' property is 'true' when the current function invocation is later than scheduled.</span>
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$Timer</span>.IsPastDue) {
<span style="color: #007020;">Write-Host</span> <span style="background-color: #fff0f0;">"PowerShell timer is running late!"</span>
}
<span style="color: #888888;"># Write an information log with the current time.</span>
<span style="color: #007020;">Write-Host</span> <span style="background-color: #fff0f0;">"PowerShell timer trigger function ran! TIME: $currentUTCtime"</span>
<span style="color: #008800; font-weight: bold;">function</span> <span style="color: #007020;">Join-Object</span>
{
<span style="color: #888888;"><#</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.SYNOPSIS</span><span style="color: #888888;"></span>
<span style="color: #888888;"> Join data from two sets of objects based on a common value</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.DESCRIPTION</span><span style="color: #888888;"></span>
<span style="color: #888888;"> Join data from two sets of objects based on a common value</span>
<span style="color: #888888;"> For more details, see the accompanying blog post:</span>
<span style="color: #888888;"> http://ramblingcookiemonster.github.io/Join-Object/</span>
<span style="color: #888888;"> For even more details, see the original code and discussions that this borrows from:</span>
<span style="color: #888888;"> Dave Wyatt's Join-Object - http://powershell.org/wp/forums/topic/merging-very-large-collections</span>
<span style="color: #888888;"> Lucio Silveira's Join-Object - http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Left</span>
<span style="color: #888888;"> 'Left' collection of objects to join. You can use the pipeline for Left.</span>
<span style="color: #888888;"> The objects in this collection should be consistent.</span>
<span style="color: #888888;"> We look at the properties on the first object for a baseline.</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Right</span>
<span style="color: #888888;"> 'Right' collection of objects to join.</span>
<span style="color: #888888;"> The objects in this collection should be consistent.</span>
<span style="color: #888888;"> We look at the properties on the first object for a baseline.</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> LeftJoinProperty</span>
<span style="color: #888888;"> Property on Left collection objects that we match up with RightJoinProperty on the Right collection</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> RightJoinProperty</span>
<span style="color: #888888;"> Property on Right collection objects that we match up with LeftJoinProperty on the Left collection</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> LeftProperties</span>
<span style="color: #888888;"> One or more properties to keep from Left. Default is to keep all Left properties (*).</span>
<span style="color: #888888;"> Each property can:</span>
<span style="color: #888888;"> - Be a plain property name like "Name"</span>
<span style="color: #888888;"> - Contain wildcards like "*"</span>
<span style="color: #888888;"> - Be a hashtable like @{Name="Product Name";Expression={$_.Name}}.</span>
<span style="color: #888888;"> Name is the output property name</span>
<span style="color: #888888;"> Expression is the property value ($_ as the current object)</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> Alternatively, use the Suffix or Prefix parameter to avoid collisions</span>
<span style="color: #888888;"> Each property using this hashtable syntax will be excluded from suffixes and prefixes</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> RightProperties</span>
<span style="color: #888888;"> One or more properties to keep from Right. Default is to keep all Right properties (*).</span>
<span style="color: #888888;"> Each property can:</span>
<span style="color: #888888;"> - Be a plain property name like "Name"</span>
<span style="color: #888888;"> - Contain wildcards like "*"</span>
<span style="color: #888888;"> - Be a hashtable like @{Name="Product Name";Expression={$_.Name}}.</span>
<span style="color: #888888;"> Name is the output property name</span>
<span style="color: #888888;"> Expression is the property value ($_ as the current object)</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> Alternatively, use the Suffix or Prefix parameter to avoid collisions</span>
<span style="color: #888888;"> Each property using this hashtable syntax will be excluded from suffixes and prefixes</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Prefix</span>
<span style="color: #888888;"> If specified, prepend Right object property names with this prefix to avoid collisions</span>
<span style="color: #888888;"> Example:</span>
<span style="color: #888888;"> Property Name = 'Name'</span>
<span style="color: #888888;"> Suffix = 'j_'</span>
<span style="color: #888888;"> Resulting Joined Property Name = 'j_Name'</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Suffix</span>
<span style="color: #888888;"> If specified, append Right object property names with this suffix to avoid collisions</span>
<span style="color: #888888;"> Example:</span>
<span style="color: #888888;"> Property Name = 'Name'</span>
<span style="color: #888888;"> Suffix = '_j'</span>
<span style="color: #888888;"> Resulting Joined Property Name = 'Name_j'</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.PARAMETER</span><span style="color: #888888;"> Type</span>
<span style="color: #888888;"> Type of join. Default is AllInLeft.</span>
<span style="color: #888888;"> AllInLeft will have all elements from Left at least once in the output, and might appear more than once</span>
<span style="color: #888888;"> if the where clause is true for more than one element in right, Left elements with matches in Right are</span>
<span style="color: #888888;"> preceded by elements with no matches.</span>
<span style="color: #888888;"> SQL equivalent: outer left join (or simply left join)</span>
<span style="color: #888888;"> AllInRight is similar to AllInLeft.</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> OnlyIfInBoth will cause all elements from Left to be placed in the output, only if there is at least one</span>
<span style="color: #888888;"> match in Right.</span>
<span style="color: #888888;"> SQL equivalent: inner join (or simply join)</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> AllInBoth will have all entries in right and left in the output. Specifically, it will have all entries</span>
<span style="color: #888888;"> in right with at least one match in left, followed by all entries in Right with no matches in left, </span>
<span style="color: #888888;"> followed by all entries in Left with no matches in Right.</span>
<span style="color: #888888;"> SQL equivalent: full join</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"></span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> #Define some input data.</span>
<span style="color: #888888;"> $l = 1..5 | Foreach-Object {</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Name = "jsmith$_"</span>
<span style="color: #888888;"> Birthday = (Get-Date).adddays(-1)</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> $r = 4..7 | Foreach-Object{</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Department = "Department $_"</span>
<span style="color: #888888;"> Name = "Department $_"</span>
<span style="color: #888888;"> Manager = "jsmith$_"</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> #We have a name and Birthday for each manager, how do we find their department, using an inner join?</span>
<span style="color: #888888;"> Join-Object -Left $l -Right $r -LeftJoinProperty Name -RightJoinProperty Manager -Type OnlyIfInBoth -RightProperties Department</span>
<span style="color: #888888;"> # Name Birthday Department </span>
<span style="color: #888888;"> # ---- -------- ---------- </span>
<span style="color: #888888;"> # jsmith4 4/14/2015 3:27:22 PM Department 4</span>
<span style="color: #888888;"> # jsmith5 4/14/2015 3:27:22 PM Department 5</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"> </span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> #Define some input data.</span>
<span style="color: #888888;"> $l = 1..5 | Foreach-Object {</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Name = "jsmith$_"</span>
<span style="color: #888888;"> Birthday = (Get-Date).adddays(-1)</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> $r = 4..7 | Foreach-Object{</span>
<span style="color: #888888;"> [pscustomobject]@{</span>
<span style="color: #888888;"> Department = "Department $_"</span>
<span style="color: #888888;"> Name = "Department $_"</span>
<span style="color: #888888;"> Manager = "jsmith$_"</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> #We have a name and Birthday for each manager, how do we find all related department data, even if there are conflicting properties?</span>
<span style="color: #888888;"> $l | Join-Object -Right $r -LeftJoinProperty Name -RightJoinProperty Manager -Type AllInLeft -Prefix j_</span>
<span style="color: #888888;"> # Name Birthday j_Department j_Name j_Manager</span>
<span style="color: #888888;"> # ---- -------- ------------ ------ ---------</span>
<span style="color: #888888;"> # jsmith1 4/14/2015 3:27:22 PM </span>
<span style="color: #888888;"> # jsmith2 4/14/2015 3:27:22 PM </span>
<span style="color: #888888;"> # jsmith3 4/14/2015 3:27:22 PM </span>
<span style="color: #888888;"> # jsmith4 4/14/2015 3:27:22 PM Department 4 Department 4 jsmith4 </span>
<span style="color: #888888;"> # jsmith5 4/14/2015 3:27:22 PM Department 5 Department 5 jsmith5 </span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"></span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> #Hey! You know how to script right? Can you merge these two CSVs, where Path1's IP is equal to Path2's IP_ADDRESS?</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> #Get CSV data</span>
<span style="color: #888888;"> $s1 = Import-CSV $Path1</span>
<span style="color: #888888;"> $s2 = Import-CSV $Path2</span>
<span style="color: #888888;"> #Merge the data, using a full outer join to avoid omitting anything, and export it</span>
<span style="color: #888888;"> Join-Object -Left $s1 -Right $s2 -LeftJoinProperty IP_ADDRESS -RightJoinProperty IP -Prefix 'j_' -Type AllInBoth |</span>
<span style="color: #888888;"> Export-CSV $MergePath -NoTypeInformation</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.EXAMPLE</span><span style="color: #888888;"></span>
<span style="color: #888888;"> #</span>
<span style="color: #888888;"> # "Hey Warren, we need to match up SSNs to Active Directory users, and check if they are enabled or not.</span>
<span style="color: #888888;"> # I'll e-mail you an unencrypted CSV with all the SSNs from gmail, what could go wrong?"</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> # Import some SSNs. </span>
<span style="color: #888888;"> $SSNs = Import-CSV -Path D:\SSNs.csv</span>
<span style="color: #888888;"> #Get AD users, and match up by a common value, samaccountname in this case:</span>
<span style="color: #888888;"> Get-ADUser -Filter "samaccountname -like 'wframe*'" |</span>
<span style="color: #888888;"> Join-Object -LeftJoinProperty samaccountname -Right $SSNs `</span>
<span style="color: #888888;"> -RightJoinProperty samaccountname -RightProperties ssn `</span>
<span style="color: #888888;"> -LeftProperties samaccountname, enabled, objectclass</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.NOTES</span><span style="color: #888888;"></span>
<span style="color: #888888;"> This borrows from:</span>
<span style="color: #888888;"> Dave Wyatt's Join-Object - http://powershell.org/wp/forums/topic/merging-very-large-collections/</span>
<span style="color: #888888;"> Lucio Silveira's Join-Object - http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx</span>
<span style="color: #888888;"> Changes:</span>
<span style="color: #888888;"> Always display full set of properties</span>
<span style="color: #888888;"> Display properties in order (left first, right second)</span>
<span style="color: #888888;"> If specified, add suffix or prefix to right object property names to avoid collisions</span>
<span style="color: #888888;"> Use a hashtable rather than ordereddictionary (avoid case sensitivity)</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.LINK</span><span style="color: #888888;"></span>
<span style="color: #888888;"> http://ramblingcookiemonster.github.io/Join-Object/</span>
<span style="color: #888888;"> </span><span style="color: #dd4422;">.FUNCTIONALITY</span><span style="color: #888888;"></span>
<span style="color: #888888;"> PowerShell Language</span>
<span style="color: #888888;"> #></span>
[<span style="color: #008800; font-weight: bold;">CmdletBinding</span>()]
<span style="color: #008800; font-weight: bold;">Param</span>
(
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span>=<span style="color: #996633;">$true</span>,
<span style="color: #008800; font-weight: bold;">ValueFromPipeLine</span> = <span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[object[]]</span> <span style="color: #996633;">$Left</span>,
<span style="color: #888888;"># List to join with $Left</span>
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span>=<span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[object[]]</span> <span style="color: #996633;">$Right</span>,
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span> = <span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[string]</span> <span style="color: #996633;">$LeftJoinProperty</span>,
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span> = <span style="color: #996633;">$true</span>)]
<span style="color: #003366; font-weight: bold;">[string]</span> <span style="color: #996633;">$RightJoinProperty</span>,
<span style="color: #003366; font-weight: bold;">[object[]]</span><span style="color: #996633;">$LeftProperties</span> = <span style="background-color: #fff0f0;">'*'</span>,
<span style="color: #888888;"># Properties from $Right we want in the output.</span>
<span style="color: #888888;"># Like LeftProperties, each can be a plain name, wildcard or hashtable. See the LeftProperties comments.</span>
<span style="color: #003366; font-weight: bold;">[object[]]</span><span style="color: #996633;">$RightProperties</span> = <span style="background-color: #fff0f0;">'*'</span>,
[<span style="color: #008800; font-weight: bold;">validateset</span>( <span style="background-color: #fff0f0;">'AllInLeft'</span>, <span style="background-color: #fff0f0;">'OnlyIfInBoth'</span>, <span style="background-color: #fff0f0;">'AllInBoth'</span>, <span style="background-color: #fff0f0;">'AllInRight'</span>)]
[<span style="color: #008800; font-weight: bold;">Parameter</span>(<span style="color: #008800; font-weight: bold;">Mandatory</span>=<span style="color: #996633;">$false</span>)]
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Type</span> = <span style="background-color: #fff0f0;">'AllInLeft'</span>,
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Prefix</span>,
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Suffix</span>
)
<span style="color: #008800; font-weight: bold;">Begin</span>
{
<span style="color: #008800; font-weight: bold;">function</span> AddItemProperties(<span style="color: #996633;">$item</span>, <span style="color: #996633;">$properties</span>, <span style="color: #996633;">$hash</span>)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$item</span>)
{
<span style="color: #008800; font-weight: bold;">return</span>
}
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$property</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$properties</span>)
{
<span style="color: #996633;">$propertyHash</span> = <span style="color: #996633;">$property</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[hashtable]</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$null</span> <span style="color: #333333;">-ne</span> <span style="color: #996633;">$propertyHash</span>)
{
<span style="color: #996633;">$hashName</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"name"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[string]</span>
<span style="color: #996633;">$expression</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"expression"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[scriptblock]</span>
<span style="color: #996633;">$expressionValue</span> = <span style="color: #996633;">$expression</span>.Invoke(<span style="color: #996633;">$item</span>)[0]
<span style="color: #996633;">$hash</span>[<span style="color: #996633;">$hashName</span>] = <span style="color: #996633;">$expressionValue</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$itemProperty</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$item</span>.psobject.Properties)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$itemProperty</span>.Name <span style="color: #333333;">-like</span> <span style="color: #996633;">$property</span>)
{
<span style="color: #996633;">$hash</span>[<span style="color: #996633;">$itemProperty</span>.Name] = <span style="color: #996633;">$itemProperty</span>.Value
}
}
}
}
}
<span style="color: #008800; font-weight: bold;">function</span> TranslateProperties
{
[<span style="color: #008800; font-weight: bold;">cmdletbinding</span>()]
<span style="color: #008800; font-weight: bold;">param</span>(
<span style="color: #003366; font-weight: bold;">[object[]]</span><span style="color: #996633;">$Properties</span>,
<span style="color: #003366; font-weight: bold;">[psobject]</span><span style="color: #996633;">$RealObject</span>,
<span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$Side</span>)
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$Prop</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Properties</span>)
{
<span style="color: #996633;">$propertyHash</span> = <span style="color: #996633;">$Prop</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[hashtable]</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$null</span> <span style="color: #333333;">-ne</span> <span style="color: #996633;">$propertyHash</span>)
{
<span style="color: #996633;">$hashName</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"name"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[string]</span>
<span style="color: #996633;">$expression</span> = <span style="color: #996633;">$propertyHash</span>[<span style="background-color: #fff0f0;">"expression"</span>] <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[scriptblock]</span>
<span style="color: #996633;">$ScriptString</span> = <span style="color: #996633;">$expression</span>.tostring()
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$ScriptString</span> <span style="color: #333333;">-notmatch</span> <span style="background-color: #fff0f0;">'param\('</span>)
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Property '$HashName'`: Adding param(`$_) to scriptblock '$ScriptString'"</span>
<span style="color: #996633;">$Expression</span> = <span style="color: #003366; font-weight: bold;">[ScriptBlock]</span><span style="background-color: #ffaaaa; color: red;">::</span>Create(<span style="background-color: #fff0f0;">"param(`$_)`n $ScriptString"</span>)
}
<span style="color: #996633;">$Output</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{Name =<span style="color: #996633;">$HashName</span><span style="background-color: #ffaaaa; color: red;">;</span> Expression = <span style="color: #996633;">$Expression</span> }
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Found $Side property hash with name </span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Output.Name)</span><span style="background-color: #fff0f0;">, expression:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Output.Expression | out-string)</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #996633;">$Output</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$ThisProp</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$RealObject</span>.psobject.Properties)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$ThisProp</span>.Name <span style="color: #333333;">-like</span> <span style="color: #996633;">$Prop</span>)
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Found $Side property '</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">ThisProp.Name)</span><span style="background-color: #fff0f0;">'"</span>
<span style="color: #996633;">$ThisProp</span>.Name
}
}
}
}
}
<span style="color: #008800; font-weight: bold;">function</span> WriteJoinObjectOutput(<span style="color: #996633;">$leftItem</span>, <span style="color: #996633;">$rightItem</span>, <span style="color: #996633;">$leftProperties</span>, <span style="color: #996633;">$rightProperties</span>)
{
<span style="color: #996633;">$properties</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{}
AddItemProperties <span style="color: #996633;">$leftItem</span> <span style="color: #996633;">$leftProperties</span> <span style="color: #996633;">$properties</span>
AddItemProperties <span style="color: #996633;">$rightItem</span> <span style="color: #996633;">$rightProperties</span> <span style="color: #996633;">$properties</span>
<span style="color: #007020;">New-Object</span> psobject -Property <span style="color: #996633;">$properties</span>
}
<span style="color: #888888;">#Translate variations on calculated properties. Doing this once shouldn't affect perf too much.</span>
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$Prop</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="background-color: #ffaaaa; color: red;">@</span>(<span style="color: #996633;">$LeftProperties</span> + <span style="color: #996633;">$RightProperties</span>))
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$Prop</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[hashtable]</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$variation</span> <span style="color: #008800; font-weight: bold;">in</span> (<span style="background-color: #fff0f0;">'n'</span>,<span style="background-color: #fff0f0;">'label'</span>,<span style="background-color: #fff0f0;">'l'</span>))
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Name'</span>) )
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$Prop</span>.ContainsKey(<span style="color: #996633;">$variation</span>) )
{
<span style="color: #996633;">$Prop</span>.Add(<span style="background-color: #fff0f0;">'Name'</span>,<span style="color: #996633;">$Prop</span>[<span style="color: #996633;">$Variation</span>])
}
}
}
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Name'</span>) <span style="color: #333333;">-or</span> <span style="color: #996633;">$Prop</span>[<span style="background-color: #fff0f0;">'Name'</span>] <span style="color: #333333;">-like</span> <span style="color: #996633;">$null</span> )
{
Throw <span style="background-color: #fff0f0;">"Property is missing a name`n. This should be in calculated property format, with a Name and an Expression:`n@{Name='Something';Expression={`$_.Something}}`nAffected property:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Prop | out-string)</span><span style="background-color: #fff0f0;">"</span>
}
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Expression'</span>) )
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'E'</span>) )
{
<span style="color: #996633;">$Prop</span>.Add(<span style="background-color: #fff0f0;">'Expression'</span>,<span style="color: #996633;">$Prop</span>[<span style="background-color: #fff0f0;">'E'</span>])
}
}
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$Prop</span>.ContainsKey(<span style="background-color: #fff0f0;">'Expression'</span>) <span style="color: #333333;">-or</span> <span style="color: #996633;">$Prop</span>[<span style="background-color: #fff0f0;">'Expression'</span>] <span style="color: #333333;">-like</span> <span style="color: #996633;">$null</span> )
{
Throw <span style="background-color: #fff0f0;">"Property is missing an expression`n. This should be in calculated property format, with a Name and an Expression:`n@{Name='Something';Expression={`$_.Something}}`nAffected property:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">Prop | out-string)</span><span style="background-color: #fff0f0;">"</span>
}
}
}
<span style="color: #996633;">$leftHash</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{}
<span style="color: #996633;">$rightHash</span> = <span style="background-color: #ffaaaa; color: red;">@</span>{}
<span style="color: #888888;"># Hashtable keys can't be null; we'll use any old object reference as a placeholder if needed.</span>
<span style="color: #996633;">$nullKey</span> = <span style="color: #007020;">New-Object</span> psobject
<span style="color: #996633;">$bound</span> = <span style="color: #996633;">$PSBoundParameters</span>.keys <span style="color: #333333;">-contains</span> <span style="background-color: #fff0f0;">"InputObject"</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> <span style="color: #996633;">$bound</span>)
{
<span style="color: #003366; font-weight: bold;">[System.Collections.ArrayList]</span><span style="color: #996633;">$LeftData</span> = <span style="background-color: #ffaaaa; color: red;">@</span>()
}
}
<span style="color: #008800; font-weight: bold;">Process</span>
{
<span style="color: #888888;">#We pull all the data for comparison later, no streaming</span>
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #996633;">$bound</span>)
{
<span style="color: #996633;">$LeftData</span> = <span style="color: #996633;">$Left</span>
}
<span style="color: #008800; font-weight: bold;">Else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$Object</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Left</span>)
{
<span style="color: #003366; font-weight: bold;">[void]</span><span style="color: #996633;">$LeftData</span>.add(<span style="color: #996633;">$Object</span>)
}
}
}
<span style="color: #008800; font-weight: bold;">End</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$item</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Right</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$item</span>.<span style="color: #996633;">$RightJoinProperty</span>
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$key</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$nullKey</span>
}
<span style="color: #996633;">$bucket</span> = <span style="color: #996633;">$rightHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$bucket</span>)
{
<span style="color: #996633;">$bucket</span> = <span style="color: #007020;">New-Object</span> System.Collections.ArrayList
<span style="color: #996633;">$rightHash</span>.Add(<span style="color: #996633;">$key</span>, <span style="color: #996633;">$bucket</span>)
}
<span style="color: #996633;">$null</span> = <span style="color: #996633;">$bucket</span>.Add(<span style="color: #996633;">$item</span>)
}
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$item</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$LeftData</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$item</span>.<span style="color: #996633;">$LeftJoinProperty</span>
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$key</span>)
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$nullKey</span>
}
<span style="color: #996633;">$bucket</span> = <span style="color: #996633;">$leftHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$bucket</span>)
{
<span style="color: #996633;">$bucket</span> = <span style="color: #007020;">New-Object</span> System.Collections.ArrayList
<span style="color: #996633;">$leftHash</span>.Add(<span style="color: #996633;">$key</span>, <span style="color: #996633;">$bucket</span>)
}
<span style="color: #996633;">$null</span> = <span style="color: #996633;">$bucket</span>.Add(<span style="color: #996633;">$item</span>)
}
<span style="color: #996633;">$LeftProperties</span> = TranslateProperties -Properties <span style="color: #996633;">$LeftProperties</span> -Side <span style="background-color: #fff0f0;">'Left'</span> -RealObject <span style="color: #996633;">$LeftData</span>[0]
<span style="color: #996633;">$RightProperties</span> = TranslateProperties -Properties <span style="color: #996633;">$RightProperties</span> -Side <span style="background-color: #fff0f0;">'Right'</span> -RealObject <span style="color: #996633;">$Right</span>[0]
<span style="color: #888888;">#I prefer ordered output. Left properties first.</span>
<span style="color: #003366; font-weight: bold;">[string[]]</span><span style="color: #996633;">$AllProps</span> = <span style="color: #996633;">$LeftProperties</span>
<span style="color: #888888;">#Handle prefixes, suffixes, and building AllProps with Name only</span>
<span style="color: #996633;">$RightProperties</span> = <span style="color: #008800; font-weight: bold;">foreach</span>(<span style="color: #996633;">$RightProp</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$RightProperties</span>)
{
<span style="color: #008800; font-weight: bold;">if</span>(<span style="color: #333333;">-not</span> (<span style="color: #996633;">$RightProp</span> <span style="color: #333333;">-as</span> <span style="color: #003366; font-weight: bold;">[Hashtable]</span>))
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Transforming property $RightProp to $Prefix$RightProp$Suffix"</span>
<span style="background-color: #ffaaaa; color: red;">@</span>{
Name=<span style="background-color: #fff0f0;">"$Prefix$RightProp$Suffix"</span>
Expression=<span style="color: #003366; font-weight: bold;">[scriptblock]</span><span style="background-color: #ffaaaa; color: red;">::</span>create(<span style="background-color: #fff0f0;">"param(`$_) `$_.'$RightProp'"</span>)
}
<span style="color: #996633;">$AllProps</span> += <span style="background-color: #fff0f0;">"$Prefix$RightProp$Suffix"</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Skipping transformation of calculated property with name </span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">RightProp.Name)</span><span style="background-color: #fff0f0;">, expression:`n</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">RightProp.Expression | out-string)</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #996633;">$AllProps</span> += <span style="color: #003366; font-weight: bold;">[string]</span><span style="color: #996633;">$RightProp</span>[<span style="background-color: #fff0f0;">"Name"</span>]
<span style="color: #996633;">$RightProp</span>
}
}
<span style="color: #996633;">$AllProps</span> = <span style="color: #996633;">$AllProps</span> | Select -Unique
<span style="color: #007020;">Write-Verbose</span> <span style="background-color: #fff0f0;">"Combined set of properties: </span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">AllProps -join ', ')</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #008800; font-weight: bold;">foreach</span> ( <span style="color: #996633;">$entry</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$leftHash</span>.GetEnumerator() )
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$entry</span>.Key
<span style="color: #996633;">$leftBucket</span> = <span style="color: #996633;">$entry</span>.Value
<span style="color: #996633;">$rightBucket</span> = <span style="color: #996633;">$rightHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$rightBucket</span>)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInLeft'</span> <span style="color: #333333;">-or</span> <span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInBoth'</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$leftItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$leftBucket</span>)
{
WriteJoinObjectOutput <span style="color: #996633;">$leftItem</span> <span style="color: #996633;">$null</span> <span style="color: #996633;">$LeftProperties</span> <span style="color: #996633;">$RightProperties</span> | Select <span style="color: #996633;">$AllProps</span>
}
}
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$leftItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$leftBucket</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$rightItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$rightBucket</span>)
{
WriteJoinObjectOutput <span style="color: #996633;">$leftItem</span> <span style="color: #996633;">$rightItem</span> <span style="color: #996633;">$LeftProperties</span> <span style="color: #996633;">$RightProperties</span> | Select <span style="color: #996633;">$AllProps</span>
}
}
}
}
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInRight'</span> <span style="color: #333333;">-or</span> <span style="color: #996633;">$Type</span> <span style="color: #333333;">-eq</span> <span style="background-color: #fff0f0;">'AllInBoth'</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$entry</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$rightHash</span>.GetEnumerator())
{
<span style="color: #996633;">$key</span> = <span style="color: #996633;">$entry</span>.Key
<span style="color: #996633;">$rightBucket</span> = <span style="color: #996633;">$entry</span>.Value
<span style="color: #996633;">$leftBucket</span> = <span style="color: #996633;">$leftHash</span>[<span style="color: #996633;">$key</span>]
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #996633;">$null</span> <span style="color: #333333;">-eq</span> <span style="color: #996633;">$leftBucket</span>)
{
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$rightItem</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$rightBucket</span>)
{
WriteJoinObjectOutput <span style="color: #996633;">$null</span> <span style="color: #996633;">$rightItem</span> <span style="color: #996633;">$LeftProperties</span> <span style="color: #996633;">$RightProperties</span> | Select <span style="color: #996633;">$AllProps</span>
}
}
}
}
}
}
<span style="color: #996633;">$RESULT</span> = <span style="background-color: #ffaaaa; color: red;">@</span>()
<span style="color: #996633;">$Subscriptions</span> = <span style="color: #007020;">Get-AzSubscription</span>
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$sub</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$Subscriptions</span>) {
<span style="color: #007020;">Get-AzSubscription</span> -SubscriptionName <span style="color: #996633;">$sub</span>.Name | <span style="color: #007020;">Set-AzContext</span>
<span style="color: #996633;">$vms</span> = <span style="color: #007020;">get-azvm</span>
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #996633;">$vm</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$vms</span>) {
<span style="color: #996633;">$info</span> = <span style="background-color: #fff0f0;">""</span> | <span style="color: #007020;">Select-Object</span> v_Name,v_ComputerName,v_OsName,v_OsVersion,v_HyperVGeneration,v_subname ,v_ResourceGroupName,v_joiner
<span style="color: #996633;">$VMINFO</span> = <span style="color: #007020;">get-azvm</span> -resourcegroupname <span style="color: #996633;">$vm</span>.ResourceGroupName -name <span style="color: #996633;">$vm</span>.Name -status
<span style="color: #996633;">$info</span>.v_Name = <span style="color: #996633;">$VMINFO</span>.Name.ToLower()
<span style="color: #996633;">$info</span>.v_ComputerName = <span style="color: #996633;">$VMINFO</span>.ComputerName
<span style="color: #996633;">$info</span>.v_OsName = <span style="color: #996633;">$VMINFO</span>.OsName
<span style="color: #996633;">$info</span>.v_OsVersion = <span style="color: #996633;">$VMINFO</span>.OsVersion
<span style="color: #996633;">$info</span>.v_HyperVGeneration = <span style="color: #996633;">$VMINFO</span>.HyperVGeneration
<span style="color: #996633;">$info</span>.v_subname = <span style="color: #996633;">$sub</span>.Id.ToLower()
<span style="color: #996633;">$info</span>.v_ResourceGroupName = <span style="color: #996633;">$VMINFO</span>.ResourceGroupName.ToLower()
<span style="color: #996633;">$info</span>.v_joiner = <span style="background-color: #fff0f0;">"</span><span style="background-color: #eeeeee;">$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">info.v_Name)$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">info.v_ResourceGroupName)$(</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="background-color: #eeeeee;">info.v_subname)</span><span style="background-color: #fff0f0;">"</span>
<span style="color: #996633;">$RESULT</span> +=<span style="color: #996633;">$info</span> }
}
<span style="color: #996633;">$RESULT</span>.v_joiner | Ft
<span style="color: #996633;">$Subscriptions</span>
<span style="color: #996633;">$ARG</span> = Search-AzGraph -Query <span style="background-color: #fff0f0;">'(Resources |where type == "microsoft.compute/virtualmachines" </span>
<span style="background-color: #fff0f0;">| project vmname=name,resourceGroup=toupper(resourceGroup),vmsize=properties.hardwareProfile.vmSize,sub=toupper(subscriptionId),os=properties.storageProfile.osDisk.osType,shut=tags.AutoShutdownSchedule,vmid=id,offer=properties.storageProfile.imageReference.offer,sku=properties.storageProfile.imageReference.sku</span>
<span style="background-color: #fff0f0;">| join kind=inner (ResourceContainers |where type=="microsoft.resources/subscriptions/resourcegroups" |project name=toupper(name),itera=tags.IteraplanID,own=tags.Owner,email=tags.Owner_email,sub2=toupper(subscriptionId) ) on $left.resourceGroup==$right.name and $left.sub==$right.sub2) </span>
<span style="background-color: #fff0f0;">| join kind=leftouter (ResourceContainers | where type=="microsoft.resources/subscriptions" | project SubName=name,sub=toupper(subscriptionId)) on $left.sub==$right.sub |</span>
<span style="background-color: #fff0f0;">project vmname=tolower(vmname),resourceGroup=tolower(resourceGroup),sub=tolower(sub),SubName,vmsize,os,itera,shut,own,email,vmid,offer,sku,joiner=strcat(vmname,resourceGroup,sub)'</span> -First 5000 -subscription <span style="color: #996633;">$Subscriptions</span>.id
<span style="color: #007020;">Join-Object</span> -Left <span style="color: #996633;">$RESULT</span> -Right <span style="color: #996633;">$ARG</span> -LeftJoinProperty v_joiner -RightJoinProperty joiner -Type OnlyIfInBoth | <span style="color: #007020;">Export-Csv</span> <span style="background-color: #fff0f0;">"temp.csv"</span>
<span style="color: #996633;">$date</span> = <span style="color: #007020;">Get-Date</span> -format dd-MM-yyyy
<span style="color: #996633;">$Context</span> = <span style="color: #007020;">New-AzStorageContext</span> -StorageAccountName <span style="background-color: #fff0f0;">"storageaccounthere"</span> -StorageAccountKey <span style="background-color: #fff0f0;">"secretkethere"</span>
<span style="color: #007020;">Set-AzStorageBlobContent</span> -Context <span style="color: #996633;">$Context</span> -Container <span style="background-color: #fff0f0;">"azure"</span> <span style="color: #333333;">-File</span> <span style="background-color: #fff0f0;">"temp.csv"</span> -Blob <span style="background-color: #fff0f0;">"azurevm$date.csv"</span> -Force
</pre></div>
</div><div><br /></div><div><br /></div><div>Now when you execute that code it should chug away and spew out a csv file to your storage account on the time schedule you specified. You can always test manually by clicking the test/run option from the menu to trigger it. I scheduled it midday Monday to Friday - that is the time when most VM's would be 'on'.</div><div><br /></div><div>To refer back to the timeout issue I hinted at earlier - this is how that gets reported in the debug output.</div><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-h6PriUN06ns/YCfLQXQzgmI/AAAAAAAADnI/BGJC50qnHcMVZJJaenMeN6hisUzK78ZIgCLcBGAsYHQ/s1022/cmdb22.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="117" data-original-width="1022" src="https://1.bp.blogspot.com/-h6PriUN06ns/YCfLQXQzgmI/AAAAAAAADnI/BGJC50qnHcMVZJJaenMeN6hisUzK78ZIgCLcBGAsYHQ/s320/cmdb22.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>This timeout can be increased up the max allows for the tier you are on - easiest way to do that in the portal is to make use of the app service editor - so click on that as shown below<br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-BWJbHtrAV_o/YCfLRbGTKPI/AAAAAAAADnY/hjpRkg9679Q3UfLH5PZgY-dcrIOw0RncwCLcBGAsYHQ/s1059/cmdb25.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="715" data-original-width="1059" src="https://1.bp.blogspot.com/-BWJbHtrAV_o/YCfLRbGTKPI/AAAAAAAADnY/hjpRkg9679Q3UfLH5PZgY-dcrIOw0RncwCLcBGAsYHQ/s320/cmdb25.PNG" width="320" /></a></div><div><br /></div><div>If you then navigate to the host.json file you can set the max timeout value as shown below to the max value allowed for your tier (sensible to just make it an hour or so to avoid a code mistake with an infinite loop running forever)</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-ntM8PjJWC2E/YCfLRuzlH0I/AAAAAAAADnc/Cmkkeo32piAQpUOI8b3NFegZmJjlzv_kACLcBGAsYHQ/s1130/cmdb26.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="497" data-original-width="1130" src="https://1.bp.blogspot.com/-ntM8PjJWC2E/YCfLRuzlH0I/AAAAAAAADnc/Cmkkeo32piAQpUOI8b3NFegZmJjlzv_kACLcBGAsYHQ/s320/cmdb26.PNG" width="320" /></a></div><br /><div><br /></div><div>And there you have it - I was quite pleased - this pulled a lot of techniques together to create what I think is quite a neat solution.</div><div><br /></div><div>Output looks like the below when you open the csv</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://lh3.googleusercontent.com/-LNeYYwrgFls/YCfatcTTL8I/AAAAAAAADn4/jEvTM0ogrb83qBxirk6QqFt158HtGhkIwCLcBGAsYHQ/image.png" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="307" data-original-width="1628" height="60" src="https://lh3.googleusercontent.com/-LNeYYwrgFls/YCfatcTTL8I/AAAAAAAADn4/jEvTM0ogrb83qBxirk6QqFt158HtGhkIwCLcBGAsYHQ/image.png" width="320" /></a></div><br /><br /></div><div>Hope you find this useful - any feedback or any other ways of getting complete dataset I'd be interested to hear.</div><div><br /></div></div><div>Picture at the top is nothing to do with anything in the article - i just liked it...</div>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-87342436802540579682020-10-23T13:29:00.000-07:002020-10-23T13:29:28.527-07:00A bit of powershell and Azure magic<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-3buwWoXDX54/X5M8vP8LxiI/AAAAAAAADhM/R8qvzyhE4LcVWqYAcDtYHIHATZQNZTu-QCLcBGAsYHQ/s640/wizard-1662948_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="425" data-original-width="640" src="https://1.bp.blogspot.com/-3buwWoXDX54/X5M8vP8LxiI/AAAAAAAADhM/R8qvzyhE4LcVWqYAcDtYHIHATZQNZTu-QCLcBGAsYHQ/s320/wizard-1662948_640.jpg" width="320" /></a></div><br /> After a long hiatus where volume of work has juts left me with no desire to spend extra effort writing things up on here I've finally put pen to paper (well technically hands to keyboard).<p></p><p>There have been so many things I could have written about recently and In some ways I wish I had (mainly selfishly as I often refer back to my own notes.....) but this Is a good one and covers a few things that others will find useful - even those outside of Azure for part of this one.</p><p>So this is the problem statement (and there are likely many ways to solve this - but this method has some useful tricks worth sharing).</p><p><b>I want to make a file appear on the desktop of a remote windows 10 device in Azure, this file needs to contain some PowerShell code to do 'something' on the device (actual content not important) - but it should be as simple as possible for the end user - i.e. they are non technical</b></p><p>I have some limitations at the point - the main one being is that I'm running the script against a machine that is not AD joined (it's AAD joined) and I don't know Intune or have any access to mess around with it anyway. So that rules out a couple of traditional approaches. It also (for reasons I won't go into) has to be run explicitly by the end user and can't just be fully automated.</p><p>In the end I decided there were two distinct technical things to resolve</p><p>1. How to make it easy for the end user?</p><p>2. How to get the solution to the end user?</p><p>I'll start (as is normal) with the first point - giving this some thought the only way to make this really simple for an end user is just to make it something you can double click. Powershell can sometimes just run OK when double clicked but is often subject to what has been allowed by policy and is also likely to open in the wrong program if file extension mappings are wrong. I wanted to avoid anything that would result in confusion and then questions back to the support team.</p><p>So how to deal with that? Well usefully on a PowerShell course I attended a couple of years back I was told about a simple utility that can convert PowerShell scripts into executables - there is even a simple GUI utility for that</p><p><a href="https://gallery.technet.microsoft.com/scriptcenter/PS2EXE-GUI-Convert-e7cb69d5" style="font-family: Calibri, sans-serif; font-size: 11pt;">https://gallery.technet.microsoft.com/scriptcenter/PS2EXE-GUI-Convert-e7cb69d5</a></p><p>Opening that up you see this simple interface where you give it your input PowerShell file and define some things for output - see pic below</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-2DoRU9Mz_yQ/X5La038riNI/AAAAAAAADhA/I7obg1d9j9U_KFPxarrPNwyQUniL7LLgwCLcBGAsYHQ/s487/powershell.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="365" data-original-width="487" src="https://1.bp.blogspot.com/-2DoRU9Mz_yQ/X5La038riNI/AAAAAAAADhA/I7obg1d9j9U_KFPxarrPNwyQUniL7LLgwCLcBGAsYHQ/s320/powershell.PNG" width="320" /></a></div><br /><p>So in my case c:\temp.ps1 is the input file and I want to generate an executable called adduser.exe and also add some metadata to that file of version number, product name and description.</p><p>So clicking compile and now I have adduser.exe that I can just double click and it performs the same job as the PowerShell script without the PowerShell - so easy for end users.</p><p>So job number 1 ticked off - nice and easy</p><p>Now on to the second point - how to get the file transferred over to the virtual machine?</p><p>First thing we need to do is put the file somewhere it can be reached - easiest way is to just put it into a storage account where it can pulled from.</p><p>I'll assume you know how to do that as it's a very common thing to want to do - the end result is the file ends up in a location like this </p><p>https://somestorageaccount.blob.core.windows.net/adduser/adduser.exe</p><p>Now i just have to execute some code to remotely pull the file down to the machine - again I can do this with some simple PowerShell making use of the relevant Azure cmdlets to run things on remote machines</p><p>The end result being a short bit of code (shown below) that connects to the rights subscription, generates a file to execute locally (as you cant just type the powershell commands in direct here) then runs that on the remote machine.</p><p>
<!--HTML generated using hilite.me--></p><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #007020;">set-azcontext</span> -Subscription SUBSCRIPTION-NAME
<span style="color: #996633;">$remoteCommand</span> =
<span style="background-color: #fff0f0;">@"</span>
<span style="background-color: #fff0f0;">Invoke-WebRequest -Uri "https://somestorageaccount.blob.core.windows.net/adduser/adduser.exe" -OutFile "C:\Users\username\Desktop\adduser.exe"</span>
<span style="background-color: #fff0f0;">"@</span>
<span style="color: #007020;">Set-Content</span> -Path .\downloadfile.ps1 -Value <span style="color: #996633;">$remoteCommand</span>
<span style="color: #007020;">Invoke-AzVMRunCommand</span> -ResourceGroupName <span style="background-color: #fff0f0;">'RGNAME'</span> -Name <span style="background-color: #fff0f0;">'VMNAME'</span> -CommandId <span style="background-color: #fff0f0;">'RunPowerShellScript'</span> -ScriptPath <span style="background-color: #fff0f0;">'.\downloadfile.ps1'</span>
</pre></div>
<p></p><p><br /></p><p>The end result being that a VM called VMNAME in the RGNAME resourcegroup in a subscription called SUBSCRIPTION-NAME runs a script that downloads my pre-created file to the desktop of the remote user.</p><p>I executed this from Azure cloud shell.</p><p>Now they can simply double click the file to complete the process we need from them.</p><p>This process would need to be done for a lot of users and the part I didn't cover was the mass rollout (because I didn't write it yet) - but all that needs to happen is to build an array of machines (maybe from a resource graph query) and then just perform the steps above in a loop.</p><p><br /></p><p>So there you have it - a couple of useful techniques in there. Even if you are unlikely to want to do something just like my use case the tips above can still be useful I think.</p><p><br /></p><p><br /></p>DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com1tag:blogger.com,1999:blog-302298286928742422.post-47475300238595637742020-03-22T14:56:00.002-07:002020-03-22T14:56:58.742-07:00Application proxy with AADDS and kerberos delegation<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-oJ_jfCBGTOQ/XnaN-4zkgxI/AAAAAAAADe8/HBnTt9twhykqPsMusL4U6Y3RkO4wx_0BACEwYBhgL/s1600/easter-2149637_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://1.bp.blogspot.com/-oJ_jfCBGTOQ/XnaN-4zkgxI/AAAAAAAADe8/HBnTt9twhykqPsMusL4U6Y3RkO4wx_0BACEwYBhgL/s320/easter-2149637_640.jpg" width="320" /></a></div>
<br />
I know what you're thinking - this is the worst name for a blog post ever...... (or possibly what has that picture got to do with any of this).<br />
<br />
This post is describing what the new kerberos constrained delegation (KCD) feature for Azure Active directory domain services (AADDS) makes possible. Now before you stop reading at this point let me tell you what this enables (and actually the same thing was always possible with 'normal' AD the new stuff is just making it possible in AADDS).<br />
<br />
Say you have a windows authenticated IIS site (these are incredibly common in most enterprises) if you are logged on to the domain and have your kerberos ticket you can then transparently access the website as yourself. All lovely stuff.<br />
<br />
What happens though if you want to modernize that login to use AAD instead (this is surprisingly not that easy to achieve by just changing IIS), or expose that internal website securely to the internet with minimal effort, or indeed allow transparent login to a domain inside Azure that is different to the domain you have a kerberos ticket for on your workstation/laptop?<br />
<br />
In fact what if you want to do all three of those in one hit?<br />
<br />
Enter <a href="https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/application-proxy">App proxy</a> into the arena......<br />
<br />
App proxy enables all of these things for you in a surprisingly easy way - so lets walk through those few simple steps.<br />
<br />
In my example case i have a windows authenticated IIS site (using kerberos) and i want to be able to transparently log into that using the AAD token i already have from my windows 10 login (assuming you have AAD joined machines rather than AD ones) - otherwise any access to most Microsoft sites would trigger an AAD login and you are likely to have a token from that process.<br />
<br />
So lets go through how this is configured.<br />
<br />
First up we need to install the app proxy connector - if you navigate to the AAD pages of the Azure portal you'll see this link:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-8SRQmPE-ixI/Xnfe8jaDgXI/AAAAAAAADfU/VpxXUI0DKbc-KJUcB33-U4mfdgHSrUzFQCLcBGAsYHQ/s1600/kcd14.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="798" data-original-width="291" height="320" src="https://1.bp.blogspot.com/-8SRQmPE-ixI/Xnfe8jaDgXI/AAAAAAAADfU/VpxXUI0DKbc-KJUcB33-U4mfdgHSrUzFQCLcBGAsYHQ/s320/kcd14.PNG" width="116" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Click on that and you'll see this screen<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-nKd7MY_dTkw/XnaIeH7thxI/AAAAAAAADd4/S7u85qcrbvAIP03UqgWzrUxs7P6ikzxzwCLcBGAsYHQ/s1600/kcd1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="553" data-original-width="1270" height="139" src="https://1.bp.blogspot.com/-nKd7MY_dTkw/XnaIeH7thxI/AAAAAAAADd4/S7u85qcrbvAIP03UqgWzrUxs7P6ikzxzwCLcBGAsYHQ/s320/kcd1.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
You'll then need to download the app proxy agent software by clicking on the link in the portal</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-EKMI2sp1mRg/XnaIdwt49BI/AAAAAAAADd0/OhX_8Hj1SyoY5qVo607_rfRmR4h1puYtQCLcBGAsYHQ/s1600/kcd2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="677" data-original-width="1571" height="137" src="https://1.bp.blogspot.com/-EKMI2sp1mRg/XnaIdwt49BI/AAAAAAAADd0/OhX_8Hj1SyoY5qVo607_rfRmR4h1puYtQCLcBGAsYHQ/s320/kcd2.PNG" width="320" /></a></div>
<br />
Save it away somewhere then copy it to the server you want to install it on (in my case I'm installing it on the same server as the IIS site - but that doesn't have to be the case)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-XEmX03F41n4/XnaId04Ms7I/AAAAAAAADdw/vggpKQ3g91YZjObQ7y39BTftI0bUlxcbACLcBGAsYHQ/s1600/kcd3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="85" data-original-width="1238" height="21" src="https://1.bp.blogspot.com/-XEmX03F41n4/XnaId04Ms7I/AAAAAAAADdw/vggpKQ3g91YZjObQ7y39BTftI0bUlxcbACLcBGAsYHQ/s320/kcd3.PNG" width="320" /></a></div>
<br />
Go through the installer (somehow the installer seems a little retro)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-O8577jfpD7o/XnaIeq4ksII/AAAAAAAADd8/hU-EDqIk6OcP61cs5XWzIZJGhXQ9R7kwwCLcBGAsYHQ/s1600/kcd4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="434" data-original-width="613" height="226" src="https://1.bp.blogspot.com/-O8577jfpD7o/XnaIeq4ksII/AAAAAAAADd8/hU-EDqIk6OcP61cs5XWzIZJGhXQ9R7kwwCLcBGAsYHQ/s320/kcd4.PNG" width="320" /></a></div>
<br />
Login to azure AD as part of the process - you will need certain elevated rights (as you might expect) to be able to do this - i cheated and activated GA via PIM but there will be a lesser role that can do this I'm pretty sure (probably mentioned in the docs.....)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9yaey8NNlKk/XnaIex5UtoI/AAAAAAAADeA/ZqhpSozm_94nGqC6L0T4e9F4N_mUzmfUQCLcBGAsYHQ/s1600/kcd5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="749" data-original-width="708" height="320" src="https://1.bp.blogspot.com/-9yaey8NNlKk/XnaIex5UtoI/AAAAAAAADeA/ZqhpSozm_94nGqC6L0T4e9F4N_mUzmfUQCLcBGAsYHQ/s320/kcd5.PNG" width="302" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
And we're installed - you might need to mess with the proxy setting depending on your network setup.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-kS49OotN6jA/XnaIfELpFiI/AAAAAAAADeE/9dWHxUVwB98npA7AXl31TEUq7rMsy1FCgCLcBGAsYHQ/s1600/kcd6.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="435" data-original-width="620" height="224" src="https://1.bp.blogspot.com/-kS49OotN6jA/XnaIfELpFiI/AAAAAAAADeE/9dWHxUVwB98npA7AXl31TEUq7rMsy1FCgCLcBGAsYHQ/s320/kcd6.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Now the connector is there and has punched an outward hole to the internet (one way) . Now we need to assign that connector to a group to then make reference too when we link to an 'enterprise application' in AAD.</div>
<br />
To do this we choose create connector group and give it a name - can be anything but has to be at least 4 characters<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-NXTRMZmAffM/XnaIfW1bDzI/AAAAAAAADeI/ASgoU2A3x_kw5t8HNQX_vH25aYQJDOY4ACLcBGAsYHQ/s1600/kcd7.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="374" data-original-width="1572" height="76" src="https://1.bp.blogspot.com/-NXTRMZmAffM/XnaIfW1bDzI/AAAAAAAADeI/ASgoU2A3x_kw5t8HNQX_vH25aYQJDOY4ACLcBGAsYHQ/s320/kcd7.PNG" width="320" /></a></div>
<br />
Once that is in place we choose configure the app (this is where it creates an enterprise app to tie this stuff together)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-GG6jvfULfCI/XnaIfjTm6iI/AAAAAAAADeM/0eh9nERIpusBuRYZtDZy-bdj8wTokTclACLcBGAsYHQ/s1600/kcd8.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="230" data-original-width="929" height="78" src="https://1.bp.blogspot.com/-GG6jvfULfCI/XnaIfjTm6iI/AAAAAAAADeM/0eh9nERIpusBuRYZtDZy-bdj8wTokTclACLcBGAsYHQ/s320/kcd8.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
We give it a name and configure appropriate urls etc - one is the internal website address that this will ultimately access and the other is the external name we'll be able to reach the website at. We then mention the connector group we created earlier to link the app to the connector agent we installed. There are some other properties of the bottom of the pic but i left all those to default.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/--WQnnpLcMCo/XnaIf_PVpoI/AAAAAAAADeQ/Riac5vRONQAN25dqpgBG6F189cVZGi7UQCLcBGAsYHQ/s1600/kcd9.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="694" data-original-width="1169" height="189" src="https://1.bp.blogspot.com/--WQnnpLcMCo/XnaIf_PVpoI/AAAAAAAADeQ/Riac5vRONQAN25dqpgBG6F189cVZGi7UQCLcBGAsYHQ/s320/kcd9.PNG" width="320" /></a></div>
<br />
Once that is created we then need to find the enterprise app in AAD<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-aZLrz3RiXH4/XnaIolDCvfI/AAAAAAAADec/hteugCvTmCcJohh9--BrV4l_6hYagbC4ACLcBGAsYHQ/s1600/kcd10.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="409" data-original-width="1177" height="111" src="https://1.bp.blogspot.com/-aZLrz3RiXH4/XnaIolDCvfI/AAAAAAAADec/hteugCvTmCcJohh9--BrV4l_6hYagbC4ACLcBGAsYHQ/s320/kcd10.PNG" width="320" /></a></div>
<br />
We then click into that and we need to do a couple of things here - the first thing (as with any AAD app) is to assign users/groups from the directory we want to grant access to - so just go to the add user button and pick the appropriate identity. Once that is in place they'll be able to access the external url and get to the site (if they have a token) - or they'll be forced to pass AAD authentication at that point<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-3oXRBdsEHZM/XnaIobe2_vI/AAAAAAAADeY/BXZtEm3zAuQhaf0blgn1otHhWmMeW6khACLcBGAsYHQ/s1600/kcd11.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="430" data-original-width="478" height="287" src="https://1.bp.blogspot.com/-3oXRBdsEHZM/XnaIobe2_vI/AAAAAAAADeY/BXZtEm3zAuQhaf0blgn1otHhWmMeW6khACLcBGAsYHQ/s320/kcd11.PNG" width="320" /></a></div>
<br />
The next bit is the clever part as at this point the website is reachable but users will be forced to authenticate again using the username/password for the AD domain that the IIS site is configured against. To prevent that we enable windows authenticated pass through (using kerberos delegation)<br />
<br />
To do that we choose the 'integrated windows authenticated' option from the single sign on blade within the enterprise application<br />
<br />
In that we need to enter 2 bits of information -first is the spn - this (in my case) was in the following format (http/servername - take care as this is not a url it is the protocol followed by / followed by servername - at first glance the tendency is to read it like a url and enter the wrong thing)<br />
<br />
The second part if the identity we want to pass through - there are various options here - you need to choose the one that contains the username format that is known by your AD controllers - the list of options should enable any use case i would thing<br />
<br />
In my case that looked like the following<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-NTbD6vDQ2-w/XnaIoes1UuI/AAAAAAAADeU/kWW_-JR9aOsHiTDWhY6j3bN_rjpXrDxKACLcBGAsYHQ/s1600/kcd12.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="508" data-original-width="1240" height="131" src="https://1.bp.blogspot.com/-NTbD6vDQ2-w/XnaIoes1UuI/AAAAAAAADeU/kWW_-JR9aOsHiTDWhY6j3bN_rjpXrDxKACLcBGAsYHQ/s320/kcd12.PNG" width="320" /></a></div>
<br />
<br />
The next bit is the new feature in AADDS to enable the delegation (note this is only possible in powershell and not through the AD GUI). For normal AD though it is possible in the GUI if you are going that route - the AADDS limitation is because it is a 'PaaS' service and you don't get full enterprise admin rights against the domain and the change is blocked.<br />
<br />
This is the simple bit of powershell (you'll need the AD modules for powershell installed to have the cmdlets it wants to use)<br />
<br />
<b>$rich = Get-ADComputer -identity servername</b><br />
<b>set-adcomputer servername -PrincipalsAllowedToDelegateToAccount $rich</b><br />
<br />
Once that is done and all the AAD/app proxy things have had chance to fully deploy and setup (seemed to take about 5 mins for me).<br />
<br />
You'll then be able to do a fully modernized AAD login to an external site which will then transparently log you on to your 'legacy' AD authenticated IIS site<br />
<br />
So in my test case - i could visit the site below, i already had an AAD token so i completely transparently logged on the the IIS site and you can see (though i redacted the import part) that I've logged on using a kerberos ticket as the server was allowed to delegate and use my AD identity to log me in.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-w1xgGA1RhGc/XnaJLifxoXI/AAAAAAAADes/ca5m4JjfCtgK1c4QSJWaSszHCJBsSorbgCLcBGAsYHQ/s1600/kcd13.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="395" data-original-width="782" height="161" src="https://1.bp.blogspot.com/-w1xgGA1RhGc/XnaJLifxoXI/AAAAAAAADes/ca5m4JjfCtgK1c4QSJWaSszHCJBsSorbgCLcBGAsYHQ/s320/kcd13.PNG" width="320" /></a></div>
<br />
I think this is an amazing feature - it's very simple and secure and if you've ever looked into AAD enabling IIS a damn site easier.<br />
<br />
App proxy really is one of the killer features of Azure - i just don't think it's that widely understood/publicized so people haven't realized how good it is.<br />
<br />
The picture by the way is mean to illustrate the delegating of the authority (the egg....) to someone else - tenuous at best i think......DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com1tag:blogger.com,1999:blog-302298286928742422.post-86278411873490592762020-03-13T08:52:00.000-07:002020-03-13T08:52:31.327-07:00Squid meets log analytics<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-EXIYGM2n0tg/XmuDa_HZ5nI/AAAAAAAADb4/q45wo1840w0nlRszI6cYV9oljBnmZiBrwCLcBGAsYHQ/s1600/kawaii-1980460_640.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="456" height="320" src="https://1.bp.blogspot.com/-EXIYGM2n0tg/XmuDa_HZ5nI/AAAAAAAADb4/q45wo1840w0nlRszI6cYV9oljBnmZiBrwCLcBGAsYHQ/s320/kawaii-1980460_640.png" width="228" /></a></div>
<br />
<br />
We've a simple squid forward proxy in our environment and we wanted to be able to do some analysis on the logfiles from it. There are lots of ways we could do this but we wanted to be able to leverage what we already had in place with Azure rather than use yet another tool.<br />
<br />
Let's me explain how we configured this and then how we could do some analysis from it.<br />
<br />
The first thing you need to do is have the extension 'OMSAgentForLinux' deployed - this can be done manually following Microsoft notes but in most cases can just be directly deployed from the portal from the insights section under monitoring where it will create the agent deployment for you - here is an example of that screen (in a lot of cases if you are invested in Azure you will likely already have this)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-_HfZJO3FsEU/XmuFLVLrHMI/AAAAAAAADcE/4OMETr8cWN4F-90oukzqI3kaoCDRNu_9ACLcBGAsYHQ/s1600/squid1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="680" data-original-width="1137" height="191" src="https://1.bp.blogspot.com/-_HfZJO3FsEU/XmuFLVLrHMI/AAAAAAAADcE/4OMETr8cWN4F-90oukzqI3kaoCDRNu_9ACLcBGAsYHQ/s320/squid1.png" width="320" /></a></div>
<br />
<br />
Once that agent is deployed you will see a whole load of metric data is being collected and fed back to the log analytics workspace<br />
<br />
The additional bit to collect custom log data is simple - we just need to navigate to the screen below - this is just reached from the advanced settings part of the log analytics workspace<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-ETvSoln62qg/XmuJdaSPIYI/AAAAAAAADcQ/_DBZyi6Lj4EXvh4U6Gp91keBrL8IWFOQQCLcBGAsYHQ/s1600/squid2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="513" data-original-width="1408" height="116" src="https://1.bp.blogspot.com/-ETvSoln62qg/XmuJdaSPIYI/AAAAAAAADcQ/_DBZyi6Lj4EXvh4U6Gp91keBrL8IWFOQQCLcBGAsYHQ/s320/squid2.PNG" width="320" /></a></div>
<br />
The screenshot already has my squid config there - but yours would be initially blank<br />
<br />
To add the squid rule click the Add button and go through the steps below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-cfOB_Fe5gmE/XmuKbhN815I/AAAAAAAADcY/H4YMDPJvbGIp4DxiNg46Ejy1JOFp7hxEwCLcBGAsYHQ/s1600/squid3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="655" data-original-width="1138" height="184" src="https://1.bp.blogspot.com/-cfOB_Fe5gmE/XmuKbhN815I/AAAAAAAADcY/H4YMDPJvbGIp4DxiNg46Ejy1JOFp7hxEwCLcBGAsYHQ/s320/squid3.PNG" width="320" /></a></div>
<br />
To enable the first screen to work you need to copy a few lines from the actual squid proxy log onto your local computer so the wizard can discover the file format.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-uoajEHwE7Gc/XmuKb_mgluI/AAAAAAAADcc/z2DGmYWcAUwDtqRVGRiGoRHQrSn3OMligCLcBGAsYHQ/s1600/squid4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="614" data-original-width="610" height="320" src="https://1.bp.blogspot.com/-uoajEHwE7Gc/XmuKb_mgluI/AAAAAAAADcc/z2DGmYWcAUwDtqRVGRiGoRHQrSn3OMligCLcBGAsYHQ/s320/squid4.PNG" width="317" /></a></div>
<br />
Choose the new line option<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-F0T2gHakwZY/XmuKb6TfZtI/AAAAAAAADcg/EyIdkIbFPywzuC4sx2-LLNzFTf4Hh9_OQCLcBGAsYHQ/s1600/squid5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="423" data-original-width="756" height="179" src="https://1.bp.blogspot.com/-F0T2gHakwZY/XmuKb6TfZtI/AAAAAAAADcg/EyIdkIbFPywzuC4sx2-LLNzFTf4Hh9_OQCLcBGAsYHQ/s320/squid5.PNG" width="320" /></a></div>
<br />
Tell it where the file can be found<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-TQG_YXnGAMM/XmuKckknj2I/AAAAAAAADck/AYbSoRLJD48nG1urNxbeAiBYH1NcO738gCLcBGAsYHQ/s1600/squid6.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="628" data-original-width="808" height="248" src="https://1.bp.blogspot.com/-TQG_YXnGAMM/XmuKckknj2I/AAAAAAAADck/AYbSoRLJD48nG1urNxbeAiBYH1NcO738gCLcBGAsYHQ/s320/squid6.PNG" width="320" /></a></div>
<br />
Give the 'bucket' where the log data will be put in log analytics a name<br />
<br />
When you get back to the main screen make sure the 'apply below configuration to my linux machines' is ticked and you're done.<br />
<br />
Now the agents will also collect data from the squid access logs (note it won't go back historically - it starts collected from the point of activation)<br />
<br />
If you now go to the main log analytics screen you can query the bucket with the squid data in and you'll see something like this<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-not_6lldAHs/XmuMpbmhENI/AAAAAAAADc8/MzXAXRR9STAHxGm-fj6EVWcj3QFpgYIbgCLcBGAsYHQ/s1600/squid7.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="427" data-original-width="745" height="183" src="https://1.bp.blogspot.com/-not_6lldAHs/XmuMpbmhENI/AAAAAAAADc8/MzXAXRR9STAHxGm-fj6EVWcj3QFpgYIbgCLcBGAsYHQ/s320/squid7.PNG" width="320" /></a></div>
<br />
Each line in the logfile becomes a record in log analytics and the data from each line becomes the 'RawData' column<br />
<br />
So now we're most of the way there - now we just need to write a query to make the data into information<br />
<br />
Here is how i converted the rawdata column into something more useful:<br />
<br />
<div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 16px; white-space: pre;">
<div>
SQUID_PROXY_CL</div>
<div>
|<span style="color: #569cd6;">extend</span> nospace = replace<span style="color: gainsboro;">(</span> @<span style="color: #ce9178;">' +'</span><span style="color: gainsboro;">,</span>@<span style="color: #ce9178;">' '</span> <span style="color: gainsboro;">,</span>RawData<span style="color: gainsboro;">)</span></div>
<div>
|<span style="color: #569cd6;">extend</span> proxdata=split<span style="color: gainsboro;">(</span>nospace<span style="color: gainsboro;">,</span> <span style="color: #ce9178;">' '</span><span style="color: gainsboro;">)</span></div>
<div>
|<span style="color: #569cd6;">extend</span> sourceip = proxdata<span style="color: gainsboro;">[</span><span style="color: #b5cea8;">2</span><span style="color: gainsboro;">]</span></div>
<div>
|<span style="color: #569cd6;">extend</span> dest = proxdata<span style="color: gainsboro;">[</span><span style="color: #b5cea8;">6</span><span style="color: gainsboro;">]</span></div>
<div>
|<span style="color: #569cd6;">extend</span> when = unixtime_milliseconds_todatetime<span style="color: gainsboro;">(</span>toreal<span style="color: gainsboro;">((</span>replace<span style="color: gainsboro;">(</span>@<span style="color: #ce9178;">'</span><span style="color: #f44747;">\</span><span style="color: #ce9178;">.'</span><span style="color: gainsboro;">,</span>@<span style="color: #ce9178;">''</span><span style="color: gainsboro;">,</span>tostring<span style="color: gainsboro;">(</span>proxdata<span style="color: gainsboro;">[</span><span style="color: #b5cea8;">0</span><span style="color: gainsboro;">])))))</span></div>
</div>
<br />
Now to explain whats going on here - the extend keyword allows you to extend the data (and add new columns) - here is what my conversion steps are doing.....<br />
<br />
nospace = make sure we only have a single space between fields - otherwise the following split command gets confused<br />
proxdata = split the rawdata into columns using space as the separator<br />
sourceip = take column 2 from the proxdata array we created in previous step<br />
dest = same as sourceip but using column 6<br />
when = convert that stupid squid epoch time format into something a human can understand (this actually took a while to figure out.....<br />
<br />
Now when i run that query i see the data like this<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-akDQAtZBMCw/XmuOYvuMx5I/AAAAAAAADdI/ZHr_z994w8gyvOhh5I92Cljwx-yj8r3zQCLcBGAsYHQ/s1600/squid8.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="320" data-original-width="1352" height="75" src="https://1.bp.blogspot.com/-akDQAtZBMCw/XmuOYvuMx5I/AAAAAAAADdI/ZHr_z994w8gyvOhh5I92Cljwx-yj8r3zQCLcBGAsYHQ/s320/squid8.PNG" width="320" /></a></div>
<br />
Now it's in a usable format i can do what i like with it - for example i can create a pie chart of the top 10 destinations that we are proxying to<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-xf5hqfi_umk/XmuPBLstsWI/AAAAAAAADdQ/rQibKHhSy6s483jLRNswTCM_VFEC76wXACLcBGAsYHQ/s1600/squid9.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="585" data-original-width="1125" height="166" src="https://1.bp.blogspot.com/-xf5hqfi_umk/XmuPBLstsWI/AAAAAAAADdQ/rQibKHhSy6s483jLRNswTCM_VFEC76wXACLcBGAsYHQ/s320/squid9.PNG" width="320" /></a></div>
<br />
<br />
I can show the usage profile per hour like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-ZtdnKOwejWM/XmuokCNg5NI/AAAAAAAADdc/svCYh5U9QkIDKdE8N9m-LSMivTB8KxbjACLcBGAsYHQ/s1600/squid10.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="648" data-original-width="1177" height="176" src="https://1.bp.blogspot.com/-ZtdnKOwejWM/XmuokCNg5NI/AAAAAAAADdc/svCYh5U9QkIDKdE8N9m-LSMivTB8KxbjACLcBGAsYHQ/s320/squid10.PNG" width="320" /></a></div>
<br />
<br />
You can also create alert rules based on some criteria too - so if you see any access to a certain site, or requests coming from an ip they shouldn't be you can build a query that identifies them and get this to trigger an alert - email/SMS or whatever - the system is very powerful.<br />
<br />
You can also take a load of the defined queries and build yourself a 'proxy' dashboard showing what's relevant for you.<br />
<br />
You could even use this on premises as the log analytics agent can be installed their too - the server just needs a way to be able to get to the public azure endpoint.<br />
<br />
The more i use log analytics and Kusto - the more i like it - it's really powerful.With this method you can read any logfile and then perform graphing/reporting/alerting based on its content.DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-85008898879654948832020-02-19T14:34:00.001-08:002020-02-19T14:34:32.957-08:00Zone redundant linux cluster in Azure<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-G_LnMaYixEE/Xk23a-WlbDI/AAAAAAAADbo/YhfMqLT4b8MsA2nfUCcokJVBtjfEtXDmQCLcBGAsYHQ/s1600/twins-1012243_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="640" height="266" src="https://1.bp.blogspot.com/-G_LnMaYixEE/Xk23a-WlbDI/AAAAAAAADbo/YhfMqLT4b8MsA2nfUCcokJVBtjfEtXDmQCLcBGAsYHQ/s320/twins-1012243_640.jpg" width="320" /></a></div>
<br />
<br />
So I'm back to clustering again - as mentioned in previous blogs I've done this a lot on the past with HPUX/AIX/redhat/windows/oracle in various different forms but never with SLES and never with Linux in Azure until this past week.<br />
<br />
So here is my write up - hopefully this is useful to others - I couldn't find a good bit of documentation that really covered a complete cluster case of what I would call a 'typical' traditional cluster - i.e. and ip address and a filesystem move from one node to another in the event of machine failure (there are still some use cases where this is relevant).<br />
<br />
The following Microsoft note got me a lot of the way there (<a href="https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-suse-nfs">https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-suse-nfs</a>) but there were some things that didn't work.<br />
<br />
So to help me in future (and to hopefully help others too) here is how I built a SLES zone redundant cluster in Azure.<br />
<br />
First up a basic picture of what it will build (don't judge my drawing skills)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-AxnWpthqux0/Xk22jkR8sVI/AAAAAAAADbc/msBlg5p2Fh0sNUeHLCkoQwWSZ8j1G6bBgCLcBGAsYHQ/s1600/cluster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="618" data-original-width="1010" height="195" src="https://1.bp.blogspot.com/-AxnWpthqux0/Xk22jkR8sVI/AAAAAAAADbc/msBlg5p2Fh0sNUeHLCkoQwWSZ8j1G6bBgCLcBGAsYHQ/s320/cluster.png" width="320" /></a></div>
<br />
<br />
So we end up with 3 machines here - the 2 'cluster' nodes - which can host the 'application' service and a 3rd note to act as a 'split brain device' (or quorum disk or cluster lock disk - whatever you want to call that). In windows clustering this device has been replaced with azure blob storage (the so called cloud witness) but that's not directly possible here (well I say that - there is an option of using a fencing agent but that has some limitations so was discounted)<br />
<br />
I split the machines so that the 2 main cluster nodes were in azure zone 1 and 2 (in west Europe) with the split brain device being located in Zone 3. This is then increasing our availability to even better than a normal availability set.<br />
<br />
First up we'll just deal with node 3 as that is very simple (it just needs to be an iscsi target reachable from the other 2 servers) and we can just get it out of the way. All this needs to be is a very small server (B2ms was fine for me) - it probably could be any os that supports creating iSCSI devices but to keep things SuSE I provisioned a SLES15sp1 image from the marketplace<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-LpSlJKePQp0/Xk2eUByAbfI/AAAAAAAADaU/x-9Pc2LysKM3cSTzXJFoLTdnOkboDh_ZwCLcBGAsYHQ/s1600/Untitled%2Bpicture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="91" data-original-width="922" height="31" src="https://1.bp.blogspot.com/-LpSlJKePQp0/Xk2eUByAbfI/AAAAAAAADaU/x-9Pc2LysKM3cSTzXJFoLTdnOkboDh_ZwCLcBGAsYHQ/s320/Untitled%2Bpicture.png" width="320" /></a></div>
<span id="goog_247930069"></span><span id="goog_247930070"></span><br />
<br />
I provision that on a private address only set with a static reservation (no public ip's here - though of course you could choose to do that should you wish....) It doesn't need any data disk and pretty much everything is just default settings.<br />
<br />
Once that is up and running we ssh on to the server and then run the following steps:<br />
<br />
1) Sort out the software packages and enable some iscsi stuff<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">zypper update</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">zypper remove lio-utils
python-rtslib python-configshell targetcli</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">zypper install targetcli-fb
dbus-1-python</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">systemctl enable targetcli</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">systemctl start targetcli</span></span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: lime;"></span><br />
2) Create the iscsi device (i just do this on the os disk) - now my system I'm installing is called 'hub' - so any instances of that name you could replace with whatever you want to call it. My two cluster nodes are called simply node01 and node02 - so again replace those names with the name of your choosing<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">mkdir /sbd</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">targetcli backstores/fileio
create sbdhub /sbd/sbdhub 50M write_back=false</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">targetcli iscsi/ create
iqn.2006-04.hub.local:hub</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">targetcli
iscsi/iqn.2006-04.hub.local:hub/tpg1/luns/ create /backstores/fileio/sbdhub</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">targetcli
iscsi/iqn.2006-04.hub.local:hub/tpg1/acls/ create
iqn.2006-04.node01.local:node01</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">targetcli
iscsi/iqn.2006-04.hub.local:hub/tpg1/acls/ create
iqn.2006-04.node02.local:node02</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: lime;">targetcli saveconfig</span></span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: lime;"></span><br />
And that's it - nothing else to do on this server at all - we just created an iscsi device for later use for split brain detection in the cluster.<br />
<br />
Right now on to the more complicated cluster nodes.<br />
<br />
We start off in pretty much the same way - I just order 2 VM's from the portal - this time however making sure to choose the SLES 15sp1 image for SAP (as this includes the HA packages required for clustering even though I have no intention of using SAP)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-pYrH-bLau-w/Xk2gpaJyRRI/AAAAAAAADag/ppzKgh8tTcYGRjFCNqgi2n_aId7tuXHdQCLcBGAsYHQ/s1600/Untitled%2Bpicture2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="109" data-original-width="863" height="40" src="https://1.bp.blogspot.com/-pYrH-bLau-w/Xk2gpaJyRRI/AAAAAAAADag/ppzKgh8tTcYGRjFCNqgi2n_aId7tuXHdQCLcBGAsYHQ/s320/Untitled%2Bpicture2.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
node01 i order in zone 1 and node02 in zone 2 (original huh). Each machine I add 1 additional data disk to - in my case 1TB each - you may require more or less than that - but just make sure they are the same size.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Once the azure platform has done it's thing i then log on to node01 and run the following steps</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
1) attach to the iscsi device we created on the 3rd node in zone 3</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">systemctl enable iscsid</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">systemctl enable iscsi</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">systemctl enable sbd</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">2) in /etc/iscsi/initiatorname.iscsi file - set name to
match name from sbd node for node1</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">## may reject this initiator.<span style="mso-spacerun: yes;"> </span>The InitiatorName must be unique</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">## for each iSCSI initiator.<span style="mso-spacerun: yes;"> </span>Do NOT duplicate iSCSI InitiatorNames.</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">InitiatorName=iqn.2006-04.node01.local:node01</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">3) restart iscsi to pick that up</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">systemctl restart iscsid</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">systemctl restart iscsi</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
4) discover that device from the 3rd node</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">iscsiadm -m discovery --type=st
--portal=10.10.10.76:3260<span style="mso-spacerun: yes;"> </span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">iscsiadm -m node -T iqn.2006-04.hub.local:hub
--login --portal=10.10.10.76:3260</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">iscsiadm -m node -p 10.10.10.76:3260 --op=update
--name=node.startup --value=automatic</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
5) Now when we run lsscsi we will see an iscsi device (/dev/sdc in the output below) </div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"># lsscsi</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">[1:0:0:0]<span style="mso-spacerun: yes;">
</span>cd/dvd<span style="mso-spacerun: yes;"> </span>Msft<span style="mso-spacerun: yes;"> </span>Virtual CD/ROM<span style="mso-spacerun: yes;"> </span>1.0<span style="mso-spacerun: yes;">
</span>/dev/sr0</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">[2:0:0:0]<span style="mso-spacerun: yes;">
</span>disk<span style="mso-spacerun: yes;"> </span>Msft<span style="mso-spacerun: yes;"> </span>Virtual Disk<span style="mso-spacerun: yes;"> </span>1.0<span style="mso-spacerun: yes;">
</span>/dev/sda</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">[3:0:1:0]<span style="mso-spacerun: yes;">
</span>disk<span style="mso-spacerun: yes;"> </span>Msft<span style="mso-spacerun: yes;"> </span>Virtual Disk<span style="mso-spacerun: yes;"> </span>1.0<span style="mso-spacerun: yes;">
</span>/dev/sdb</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">[6:0:0:0]<span style="mso-spacerun: yes;">
</span>disk<span style="mso-spacerun: yes;"> </span>LIO-ORG<span style="mso-spacerun: yes;"> </span>sbdhub<span style="mso-spacerun: yes;"> </span>4.0<span style="mso-spacerun: yes;">
</span>/dev/sdc</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">6) If we then find the full path to that using this command <span style="color: black; display: inline; float: none; font-family: "calibri"; font-size: 14.66px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">(pick the
one starting scsi-3)</span></span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">ls -l /dev/disk/by-id/scsi-* | grep sdc </span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">/dev/disk/by-id/scsi-3600140591120716e08e4ad4b3b5318c0</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
7) We can then create the sbd device</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">sbd -d
/dev/disk/by-id/scsi-3600140591120716e08e4ad4b3b5318c0 -1 60 -4 120 create</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">8) And add that device to the config file</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">vi /etc/sysconfig/sbd</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">grep scsi /etc/sysconfig/sbd</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">SBD_DEVICE="/dev/disk/by-id/scsi-3600140522db15c1728d4b01a2e2204ed"</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
9) now we enable the 'softdog' module - honestly no clue what this is for but the MS note said to do it...…..</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">echo softdog | sudo tee
/etc/modules-load.d/softdog.conf</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">modprobe -v softdog</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">10) now get all the packages up to date</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">zypper update (then reboot)</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">11) Now a couple of config changes</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>vi
/etc/systemd/system.conf</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
look for this value and set to 4096</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">defaulttasksmax=4096</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">then reload the daemon</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">systemctl daemon-reload</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">add</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">vm.dirty_bytes = 629145600</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">vm.dirty_background_bytes = 314572800</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">to /etc/sysctl.conf</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">Then stop the azure cloud networking thing kicking in as it messes stuff up</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">vi /etc/sysconfig/network/ifcfg-eth0 (set
cloud_netconfig_manage to no)</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">12) Now install some python stuff</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">zypper ar </span><a href="https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/"><span style="background: #00B0F0;">https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/</span></a><span style="background: #00B0F0;"> SLE15-PackageHub</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">zypper in python3-azure-sdk</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
14) Now make sure names resolution is working by adding entries to /etc/hosts on the server</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: #00b0f0;"></span><br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">10.10.10.69 node01</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">10.10.10.70 node02</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: #00b0f0;"></span><br /></div>
<div class="separator" style="clear: both; text-align: left;">
15) Now we can initialize the cluster (albeit with one node at the moment) - pretty much defaults for everything - output below is fairly self explanatory.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">ha-cluster-init -u</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">WARNING: chronyd.service is not configured to start
at system boot.</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">Do you want to continue anyway (y/n)? y</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Generating
SSH key</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Configuring
csync2</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Generating
csync2 shared key (this may take a while)...done</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>csync2
checking files...done</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">Configure Corosync (unicast):</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>This will
configure the cluster messaging layer.<span style="mso-spacerun: yes;">
</span>You will need</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>to specify a
network address over which to communicate (default</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>is eth0's
network, but you can use the network address of any</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>active
interface).</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Address for
ring0 [10.10.10.69]</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Port for
ring0 [5405]</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">Configure SBD:</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>If you have
shared storage, for example a SAN or iSCSI target,</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>you can use
it avoid split-brain scenarios by configuring SBD.</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>This
requires a 1 MB partition, accessible to all nodes in the</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;">
</span>cluster.<span style="mso-spacerun: yes;"> </span>The device path must be
persistent and consistent</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>across all
nodes in the cluster, so /dev/disk/by-id/* devices</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>are a good
choice.<span style="mso-spacerun: yes;"> </span>Note that all data on the
partition you</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>specify here
will be destroyed.</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">Do you wish to use SBD (y/n)? y</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">SBD is already configured to use
/dev/disk/by-id/scsi-3600140522db15c1728d4b01a2e2204ed - overwrite (y/n)? n</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Hawk cluster
interface is now running. To see cluster status, open:</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span></span><a href="https://10.55.172.30:7630/"><span style="background: #00B0F0;">https://10.10.10.69:7630/</span></a></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Log in with
username 'hacluster', password 'linux'</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">WARNING: You should change the hacluster password to
something more secure!</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Waiting for
cluster........done</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Loading
initial cluster configuration</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">Configure Administration IP Address:</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Optionally
configure an administration virtual IP</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>address. The
purpose of this IP address is to</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>provide a
single IP that can be used to interact</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>with the
cluster, rather than using the IP address</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>of any
specific cluster node.</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;">Do you wish to configure a virtual IP address (y/n)?
n</span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #00B0F0;"><span style="mso-spacerun: yes;"> </span>Done (log
saved to /var/log/ha-cluster-bootstrap.log)</span></div>
<div class="separator" style="clear: both; text-align: left;">
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike></div>
Right no on to zone 2 node (I'll switch to orangy colour for this - by the way the colour is not some 80's throwback - it's meant to help illustrate which server stuff is being done on - see pic at top of the post.......)<br />
<br />
Most of this is a repeat so I'll skip a lot of the explanation<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">systemctl enable iscsid</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">systemctl enable iscsi</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">systemctl enable sbd</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>vi
/etc/iscsi/initiatorname.iscsi</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">grep Init
/etc/iscsi/initiatorname.iscsi</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">## Default iSCSI Initiatorname.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">## If you change the InitiatorName, existing access
control lists</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">## may reject this initiator.<span style="mso-spacerun: yes;"> </span>The InitiatorName must be unique</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">## for each iSCSI initiator.<span style="mso-spacerun: yes;"> </span>Do NOT duplicate iSCSI InitiatorNames.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">InitiatorName=iqn.2006-04.node02.local:node02</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">systemctl restart iscsid</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">systemctl restart iscsi</span></div>
<span style="font-family: "calibri";"></span><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">iscsiadm -m discovery --type=st
--portal=10.10.10.76:3260<span style="mso-spacerun: yes;"> </span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">iscsiadm -m node -T iqn.2006-04.hub.local:hub
--login --portal=10.10.10.76:3260</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">iscsiadm -m node -p 10.10.10.76:3260 --op=update
--name=node.startup --value=automatic</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"># lsscsi</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">[1:0:0:0]<span style="mso-spacerun: yes;">
</span>cd/dvd<span style="mso-spacerun: yes;"> </span>Msft<span style="mso-spacerun: yes;"> </span>Virtual CD/ROM<span style="mso-spacerun: yes;"> </span>1.0<span style="mso-spacerun: yes;">
</span>/dev/sr0</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">[2:0:0:0]<span style="mso-spacerun: yes;">
</span>disk<span style="mso-spacerun: yes;"> </span>Msft<span style="mso-spacerun: yes;"> </span>Virtual Disk<span style="mso-spacerun: yes;"> </span>1.0<span style="mso-spacerun: yes;">
</span>/dev/sda</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">[3:0:1:0]<span style="mso-spacerun: yes;">
</span>disk<span style="mso-spacerun: yes;"> </span>Msft<span style="mso-spacerun: yes;"> </span>Virtual Disk<span style="mso-spacerun: yes;"> </span>1.0<span style="mso-spacerun: yes;">
</span>/dev/sdb</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">[6:0:0:0]<span style="mso-spacerun: yes;">
</span>disk<span style="mso-spacerun: yes;"> </span>LIO-ORG<span style="mso-spacerun: yes;"> </span>sbdhub<span style="mso-spacerun: yes;"> </span>4.0<span style="mso-spacerun: yes;">
</span>/dev/sdc</span></div>
<span style="font-family: "calibri";"></span><br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">/dev/disk/by-id/scsi-3600140591120716e08e4ad4b3b5318c0
(same as on node 1)</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-family: Times New Roman;"></span><br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">dont recreate sbd device again - just enter into
config</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">vi /etc/sysconfig/sbd</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">hubclus-1:~ # grep scsi /etc/sysconfig/sbd</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">SBD_DEVICE="/dev/disk/by-id/scsi-3600140522db15c1728d4b01a2e2204ed"</span></div>
<span style="font-family: "calibri";"></span><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">echo softdog | sudo tee
/etc/modules-load.d/softdog.conf</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">modprobe -v softdog</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">zypper update</span></div>
<span style="font-family: "calibri";"></span><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>vi
/etc/systemd/system.conf</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">defaulttasksmax=4096</span></div>
<span style="font-family: "calibri";"></span><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">systemctl daemon-reload</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">add</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">vm.dirty_bytes = 629145600</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">vm.dirty_background_bytes = 314572800</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">to /etc/sysctl.conf</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">vi /etc/sysconfig/network/ifcfg-eth0 (set
cloud_netconfig_manage to no)</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">zypper ar </span><a href="https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/"><span style="background: #FFC000;">https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/</span></a><span style="background: #FFC000;"> SLE15-PackageHub</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">zypper in python3-azure-sdk</span></div>
<span style="font-family: "calibri";"></span><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">update hosts</span></div>
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">10.10.10.69 node01</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">10.10.10.70 node02</span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />
Now on both nodes we need to generate ssh keys to allow transparent login<br />
<br />
ssh-keygen (default everything when prompted)<br />
then copy the content of the public key into the authorized_keys file on the opposite node - then you can ssh with no password to the opposite node (I'm assuming if you are contemplating building a cluster you'll know the basics of how to set this up to be honest.......)<br />
<br />
Now we join the second node to the cluster<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">ha-cluster-join</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">WARNING: chronyd.service is not configured to start
at system boot.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">Do you want to continue anyway (y/n)? y</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Join This
Node to Cluster:</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>You will be
asked for the IP address of an existing node, from which</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;">
</span>configuration will be copied.<span style="mso-spacerun: yes;"> </span>If
you have not already configured</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>passwordless
ssh between nodes, you will be prompted for the root</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>password of
the existing node.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>IP address
or hostname of existing node (e.g.: 192.168.1.1) []node01</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Retrieving
SSH keys - This may prompt for root@node01:</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">/root/.ssh/id_rsa already exists - overwrite (y/n)?
n</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>No new SSH
keys installed</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Configuring
csync2...done</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Merging
known_hosts</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Probing for
new partitions...done</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Address for
ring0 [10.10.10.70]</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Hawk cluster
interface is now running. To see cluster status, open:</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span></span><a href="https://10.55.13.70:7630/"><span style="background: #FFC000;">https://10.10.10.70:7630/</span></a></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Log in with
username 'hacluster', password 'linux'</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">WARNING: You should change the hacluster password to
something more secure!</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Waiting for
cluster....done</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Reloading
cluster configuration...Password:</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">Password:</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;">done</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: #FFC000;"><span style="mso-spacerun: yes;"> </span>Done (log
saved to /var/log/ha-cluster-bootstrap.log)</span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />
And at this point we have a basic cluster in place - it's not really got anything useful in it though at all - so now lets add in that.<br />
<br />
The following steps need to be done on both nodes (showing this in red......)<br />
<br />
1) update corosync config - just these 2 values need updating and the service restarting<br />
<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">update /etc/corosync/corosync.conf
file</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>token:<span style="mso-spacerun: yes;"> </span>30000</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>consensus:<span style="mso-spacerun: yes;"> </span>36000</span></div>
<span style="font-family: "calibri";"></span><br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">service corosync restart</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
2) Then we install socat - this we'll use to host a 'dummy' service running on a port that we can use for the loadbalancer probe we'll create later on</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">zypper in socat </span></div>
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
3) Now we prep that 1TB disk I added to be used by drbd (I find this for some reason the most impossible sequence of 4 letters to type - I get it wrong at least 50% of the time - too long working with db's I think)</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Make sure you run this against the 1TB device and not the iscsi device...…. (sdd in my case where iscsi was sdc)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"> # fdisk /dev/sdd</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Welcome to fdisk (util-linux 2.33.1).</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Changes will remain in memory only,
until you decide to write them.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Be careful before using the write
command.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Device does not contain a recognized
partition table.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Created a new DOS disklabel with disk
identifier 0x207d70aa.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Command (m for help): n</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Partition type</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>p<span style="mso-spacerun: yes;"> </span>primary (0 primary, 0
extended, 4 free)</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>e<span style="mso-spacerun: yes;"> </span>extended (container for
logical partitions)</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Select (default p): p</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Partition number (1-4, default 1):</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">First sector (2048-2147483647, default
2048):</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Last sector, +/-sectors or
+/-size{K,M,G,T,P} (2048-2147483647, default 2147483647):</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Created a new partition 1 of type
'Linux' and of size 1024 GiB.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Command (m for help): w</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">The partition table has been altered.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Calling ioctl() to re-read partition
table.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">Syncing disks.</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Now we want to initialize this disk with lvm (I don't think you need to use LVM to be honest - but I just prefer to work using it </div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">pvcreate /dev/sdd1</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">vgcreate /dev/netahub /dev/sdd1</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">lvcreate --name esis --size 128G
/dev/netahub</span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">now we have a 128GB block device that we will use with drbd</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">(As a side note i had never even heard of drbd before i started building this - it essentially does block replication - doesnt have to be used in a cluster but i guess thats the main use case)</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">So lets set that part up</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: white;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;">1) set content of vi
/etc/drbd.d/global_common.conf</span><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"> to</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">global {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>usage-count no;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">common {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>handlers {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>fence-peer
"/usr/lib/drbd/crm-fence-peer.sh";</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>after-resync-target
"/usr/lib/drbd/crm-unfence-peer.sh";</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>split-brain
"/usr/lib/drbd/notify-split-brain.sh root";</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>pri-lost-after-sb
"/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>startup {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>wfc-timeout 0;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>options {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>disk {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>md-flushes yes;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>disk-flushes yes;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>c-plan-ahead 1;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>c-min-rate 100M;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>c-fill-target 20M;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>c-max-rate 4G;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>net {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>after-sb-0pri
discard-younger-primary;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>after-sb-1pri discard-secondary;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>after-sb-2pri call-pri-lost-after-sb;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>protocol<span style="mso-spacerun: yes;"> </span>C;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>tcp-cork yes;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>max-buffers 20000;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>max-epoch-size 20000;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>sndbuf-size 0;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>rcvbuf-size 0;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
2) Now we create a drbd config file for our lvm device - this file is created in /etc/drbd.d and in my case is named hub-esis.res (the name of the file has to match the name in the first line of the file). Note that we name the virtual device that will be created here as /dev/drbd0</div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">resource hub-esis {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>protocol<span style="mso-spacerun: yes;"> </span>C;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>disk {</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>on-io-error<span style="mso-spacerun: yes;"> </span>detach;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>on node01{</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>address<span style="mso-spacerun: yes;"> </span>10.10.10.69:7790;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>device<span style="mso-spacerun: yes;"> </span>/dev/drbd0;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>disk<span style="mso-spacerun: yes;"> </span>/dev/netahub/esis;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>meta-disk internal;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>on node02{</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>address<span style="mso-spacerun: yes;"> </span>10.10.10.70:7790;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>device<span style="mso-spacerun: yes;"> </span>/dev/drbd0;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>disk<span style="mso-spacerun: yes;"> </span>/dev/netahub/esis;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;"> </span>meta-disk internal;</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="mso-spacerun: yes;">
</span>}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;">}</span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />
3) Now we create the actual device<br />
<br /></span><span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;"> drbdadm create-md hub-esis</span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">drbdadm up hub-esis</span></span></div>
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">drbdadm new-current-uuid
--clear-bitmap hub-esis</span></span><br />
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">drbdadm primary --force hub-esis</span></span><br />
<br />
Now we bring it up on the second node also<br />
<br />
<span style="background-color: #00b0f0;">drbdadm create-md hub-esis</span><br />
<span style="background-color: #00b0f0;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;">drbdadm up hub-esis</span></span><br />
<span style="background-color: #00b0f0;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="-webkit-text-stroke-width: 0px; background-color: #00b0f0; color: black; display: inline !important; float: none; font-family: "calibri"; font-size: 14.66px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">drbdadm down hub-esis</span></span></span><br />
<span style="background-color: #00b0f0;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="-webkit-text-stroke-width: 0px; background-color: #00b0f0; color: black; display: inline !important; float: none; font-family: "calibri"; font-size: 14.66px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">drbdadm up hub-esis</span></span></span><br />
<span style="background-color: #00b0f0;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br /></span></span>
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: white;">Now we go back to primary and force the 2 devices to sync</span></span><br />
<span style="font-family: Calibri;"></span><span style="background-color: white;"></span><br /></span><span style="background-color: orange; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;">drbdsetup wait-sync-resource hub-esis</span></span><br />
<span style="color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: #3d85c6;"></span><span style="background-color: #6fa8dc;"></span><span style="background-color: #6fa8dc;"></span><span style="background-color: black;"></span><span style="background-color: #6fa8dc;"></span><span style="background-color: #6fa8dc;"></span><span style="background-color: orange;"></span></span><span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">So now that block device is syncing between the 2 servers - now it's active lets create a filesystem of the meta device that got added</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: orange; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">mkfs.xfs /dev/drbd0</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span style="background-color: orange;"></span><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Right at this point we can now start adding stuff into the actual cluster config and get something actually useful running</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">1) we set a global setting for the cluster and go into maintenance mode (so no failovers start happening)</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure property
maintenance-mode=true</span></span></span><br />
<span style="color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure rsc_defaults
resource-stickiness="200"</span></span><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike></span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: orange;"></span><br /></span>
2<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">) Lets add the drbd device to the cluster config - the first command defines the device and the second defines the master slave relationship</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure primitive drbd_hub-esis
\</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
ocf:linbit:drbd \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
params drbd_resource=hub-esis \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
op monitor interval="15" role="Master" \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
op monitor interval="30" role="Slave"
notify="true"</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure ms ms-drbd_hub-esis
drbd_hub-esis \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
meta master-max="1" master-node-max="1"
clone-max="2" \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
clone-node-max="1" notify="true"
interleave="true"</span></span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: orange;"></span></span><span style="background-color: white; font-family: "calibri";">3) Now we add the filesystem definition to mount the filesystem (remember that you have to actually create the mount point on both nodes - in this case /esis)</span><br />
<span style="background-color: white; font-family: "calibri";"><br /></span>
<span style="background-color: white; font-family: "calibri";">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure primitive fs_esis \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
ocf:heartbeat:Filesystem \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
params device=/dev/drbd0 \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
directory=/esis \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
fstype=xfs \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
op monitor interval="10s"</span></span></div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: orange;"></span></span><span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">4) Now we add the virtual cluster ip (make sure the netmask and nic device are set correctly)</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure primitive vip_hub \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
IPaddr2 \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
params ip=10.10.10.68 cidr_netmask=28 nic=eth0 op monitor interval=10
timeout=20</span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-color: orange;"></span><br /></div>
</span><span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">5) Now we add the probeport service (i'll link back to this on the loadbalancer in a bit) - this is just creating a dummy 'thing' running on port 55555</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-variant: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure primitive probeport
anything \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
params binfile="/usr/bin/socat" cmdline_options="-U
TCP-LISTEN:55555,backlog=10,fork,reuseaddr /dev/null" \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
op monitor timeout=20s interval=10 depth=0</span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="background-color: orange;"></span><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
6) Now we group some of the objecs together and define some dependencies between them<br />
<br />
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure group hub \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
fs_esis probeport vip_hub</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure order o-drbd_before_hub
mandatory: \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
ms-drbd_hub-esis:promote hub:start</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure colocation
col-hub_on_drbd mandatory: \</span></span></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">
hub ms-drbd_hub-esis:Master</span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="background: red; mso-highlight: red;"><span style="background-color: orange;"></span><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
7) Now we come out of maintenance mode<br />
<br />
<span style="background-attachment: scroll; background-clip: border-box; background-image: none; background-origin: padding-box; background-position: 0% 0%; background-repeat: repeat; background-size: auto;"><span style="background-color: orange;">crm configure property
maintenance-mode=false</span></span><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><span style="background-color: orange;"></span><br />
At this point the cluster should be up and running and all healthy - (if you have any warning messages from crm status you can clean them up with "crm resource cleanup" - I found this was sometimes essential as if it gets in a sort of funny state it will refuse to run on the other node - this command clears up and previous error messages and sets the state as OK)<br />
<br />
You'll find it you just shutdown a node the cluster package of drbd/vip/probeport/filesystem will very rapidly (just few secs for me) switch to the other node and everything works great.</span><span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">However there is one more thing to do in the portal and that is to create a load balancer - this is needed as the vip we created via the cluster is not known to azure - so you can't actually get to that ip outside of the servers. The load balancer we will create has the same vip as the cluster ip (very confusing i know - but it works this way for windows cluster too). The loadbalancer will be looking for something running on port 55555 (the dummy service thing we created) - if it finds it it sends all the traffic there. So when that service moves so does all the traffic the load balancer is sending.</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">So in the load balancer screen be sure to choose a standard load balancer (required to use across zones) and set it something like this (ignore error as its just beause i didnt choose vnet etc first)</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-p3lNNrz4oHo/Xk2w3iLguRI/AAAAAAAADas/e0dz2bGNk38pwL6AhY_YG3icU6WskH7MQCLcBGAsYHQ/s1600/Capture.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="763" data-original-width="947" height="257" src="https://1.bp.blogspot.com/-p3lNNrz4oHo/Xk2w3iLguRI/AAAAAAAADas/e0dz2bGNk38pwL6AhY_YG3icU6WskH7MQCLcBGAsYHQ/s320/Capture.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">backend pools is just the 2 cluster nodes - for the health probe it looks like this:</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-gFjoJByxjaU/Xk2xW_eqoSI/AAAAAAAADa0/8bclrNLwU6wNuFzQJ-nZTlShd4QEcBjDQCLcBGAsYHQ/s1600/Capture2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="448" data-original-width="1486" height="96" src="https://1.bp.blogspot.com/-gFjoJByxjaU/Xk2xW_eqoSI/AAAAAAAADa0/8bclrNLwU6wNuFzQJ-nZTlShd4QEcBjDQCLcBGAsYHQ/s320/Capture2.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">And finally the rules screen - be sure to check the ha ports box so all ports are load balanced (unless you know specifically which ones you need and are going to create a rule for each one)</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-xvhBFmPYmOc/Xk2x022FIHI/AAAAAAAADa8/9j_gqIUAq9IZQwYLmQR3EDVrQwPw5hEIwCLcBGAsYHQ/s1600/capture3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="819" data-original-width="855" height="306" src="https://1.bp.blogspot.com/-xvhBFmPYmOc/Xk2x022FIHI/AAAAAAAADa8/9j_gqIUAq9IZQwYLmQR3EDVrQwPw5hEIwCLcBGAsYHQ/s320/capture3.PNG" width="320" /></a></div>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">And that's it done - as a simple test you should now be able to ssh to the cluster ip from your laptop/desktop and you'll be routed to the host where the app currently is. If you kill the first node and then try and connect again with ssh you'll see you end up on the other node like magic</span><br />
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span>
<span style="background-color: white; color: black; display: inline; float: none; font-family: "calibri"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">There you have it linux cluster in Azure across availability zones - enjoy........</span><br />
<br />
As a small aside - the website that is mentioned during cluster creation is actually quite nice as a visual overview but also as a management console - see example screenshot from another test cluster below<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-D-R3cRC_CLU/Xk2ztjv3aUI/AAAAAAAADbI/tvlfXxvYc8kG_h4WbHSeE9bl22chuo7AwCLcBGAsYHQ/s1600/Capture4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="855" data-original-width="1600" height="170" src="https://1.bp.blogspot.com/-D-R3cRC_CLU/Xk2ztjv3aUI/AAAAAAAADbI/tvlfXxvYc8kG_h4WbHSeE9bl22chuo7AwCLcBGAsYHQ/s320/Capture4.PNG" width="320" /></a></div>
<br />
<br />DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-10663487457647987792020-01-16T12:36:00.000-08:002020-01-16T12:36:36.335-08:00Getting an excel list of all Azure Virtual machines<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-meiX34ustb8/XiBvj-7mZyI/AAAAAAAADZ8/ZM9Enk1g0E4TNxAE6ElaG2tAF0xqiV28QCLcBGAsYHQ/s1600/noel-1827278_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="527" height="320" src="https://1.bp.blogspot.com/-meiX34ustb8/XiBvj-7mZyI/AAAAAAAADZ8/ZM9Enk1g0E4TNxAE6ElaG2tAF0xqiV28QCLcBGAsYHQ/s320/noel-1827278_640.jpg" width="263" /></a></div>
<br />
<br />
Now i don't think i'm unique in getting frustrated that from some of the Azure portal screens there is no download as csv option or a more effective filter to show exactly what i want.<br />
<div>
<br /></div>
<div>
A prime example which has annoyed me a few times is just getting a list of all my virtual machines - this is all there on the VM blade but how do i get that in excel format?</div>
<div>
<br /></div>
<div>
There are a few ways of going about getting this - but i think one of the easiest is to take the approach i describe below....</div>
<div>
<br /></div>
<div>
First up we need to create a resource graph query - this is a relatively new feature that allows us to query the Azure metadata using Kusto ( i talked a little more about that <a href="http://dbaharrison.blogspot.com/2019/11/kql-is-my-new-sql.html">here</a>) We can then make use of the Kusto functionality to join to lots of data sources and pull together a report of exactly what we want.</div>
<div>
<br /></div>
<div>
The primary data i want to pull out is the machine name,resource group, subscription, vmsize, os and the iteraplan identifier (this is present as a tag on our resource groups).</div>
<div>
<br /></div>
<div>
Using kusto i can do this relatively easily as:</div>
<div>
<br /></div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">(Resources |where type == <span style="background-color: #fff0f0;">"microsoft.compute/virtualmachines"</span>
| project vmname=name,resourceGroup=toupper(resourceGroup),vmsize=properties.hardwareProfile.vmSize,sub=toupper(subscriptionId),os=properties.storageProfile.osDisk.osType
| join kind=inner (ResourceContainers
|where type==<span style="background-color: #fff0f0;">'microsoft.resources/subscriptions/resourcegroups'</span>
|project name=toupper(name),itera=tags.IteraplanID) on <span style="color: #996633;">$left</span>.resourceGroup==<span style="color: #996633;">$right</span>.name)
| join kind=leftouter (ResourceContainers
| where type==<span style="background-color: #fff0f0;">'microsoft.resources/subscriptions'</span>
| project SubName=name,sub=toupper(subscriptionId)) on <span style="color: #996633;">$left</span>.sub==<span style="color: #996633;">$right</span>.sub
|project vmname,resourceGroup,SubName,vmsize,os,itera
</pre>
</div>
<br /></div>
<div>
This gives an output like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-zld5ZrrRseQ/XiBkiyNXsbI/AAAAAAAADZc/Njge4tjMC-UtcmlQcdiDAS7BZ79mvpIUgCLcBGAsYHQ/s1600/kusto.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="1600" height="96" src="https://1.bp.blogspot.com/-zld5ZrrRseQ/XiBkiyNXsbI/AAAAAAAADZc/Njge4tjMC-UtcmlQcdiDAS7BZ79mvpIUgCLcBGAsYHQ/s320/kusto.PNG" width="320" /></a></div>
<br /></div>
<div>
That screen helpfully has a download as csv button so i can then extract it in the format i want - i can even save it as a resource graph query (see the save as button in the screenshot) - then i can directly navigate to that - and also share that with other people.<br />
<br />
To make things even simpler i can do the following:<br />
<br />
1) create a new dashboard name 'VM CMDB' - choose the dashboard option and then click New dashboard and give it a name (leave the dashboard itself empty for now)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-JeNjEhrEMug/XiBlhtrPXlI/AAAAAAAADZk/N40gFy9QFfwbizcFWffIW_Uev4lula9TwCLcBGAsYHQ/s1600/kusto2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="180" data-original-width="1041" height="55" src="https://1.bp.blogspot.com/-JeNjEhrEMug/XiBlhtrPXlI/AAAAAAAADZk/N40gFy9QFfwbizcFWffIW_Uev4lula9TwCLcBGAsYHQ/s320/kusto2.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
2) Now that dashboard is set as the current one we can go back to resource graph explorer screen and click the 'pin to dashboard' option. This then create a tile automatically on the current dashboard.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
A little bit of resizing and then i get this:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9vckh1T_xM0/XiBsBTPEDXI/AAAAAAAADZw/yV_PXdOCBVIijBsmSNH56FTzH08FPB67gCLcBGAsYHQ/s1600/kusto3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="353" data-original-width="1552" height="72" src="https://1.bp.blogspot.com/-9vckh1T_xM0/XiBsBTPEDXI/AAAAAAAADZw/yV_PXdOCBVIijBsmSNH56FTzH08FPB67gCLcBGAsYHQ/s320/kusto3.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
I can now just choose that dashboard to go direct to that view and extract the data in excel.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
3) i can even then share the dashboard (those are objects too) - i just click the share button from the top of the dashboard - set privileges as appropriate - then share the direct link with people (cut and paste from browser address bar) - this makes it dead easy to give everyone access.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br /></div>
DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-55127567795238891202020-01-13T08:49:00.000-08:002020-01-13T08:49:16.890-08:00Getting a grip on what is done in Azure<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-jzSZTnryig0/XhyfbuC1XKI/AAAAAAAADZQ/7gyRTWoI1N4uofSMce2RhnQgzlTzpc6EQCLcBGAsYHQ/s1600/hold-461234_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="480" height="320" src="https://1.bp.blogspot.com/-jzSZTnryig0/XhyfbuC1XKI/AAAAAAAADZQ/7gyRTWoI1N4uofSMce2RhnQgzlTzpc6EQCLcBGAsYHQ/s320/hold-461234_640.jpg" width="240" /></a></div>
<br />
<br />
Tim (you know who you are) has kicked me back into life to start writing some stuff up again. The volume of work of late has just been so much that i had no inclination to write, maybe this new year can get me back into the saddle again.<br />
<br />
So what am i going to talk about?<br />
<br />
Azure Policy - perhaps the least sexy part of what Azure has to offer.... :-)<br />
<br />
From a governance and control point of view though it's actually very good - you can read more of the general spiel about it here : <a href="https://docs.microsoft.com/en-us/azure/governance/policy/overview">https://docs.microsoft.com/en-us/azure/governance/policy/overview</a><br />
<br />
In short it allows you to block/audit/change what is being deployed into Azure - it's much more than RBAC (which is essentially a yes/no permission on a task) - Policy enables much more granularity - for example you could have rules that say:<br />
<br />
- you can't deploy a VM into North Europe unless it's called XXXXX<br />
- you can deploy a PaaS storage account but it must only allow https comms to it.<br />
etc<br />
etc<br />
<br />
The possibilities are huge - made even more so by it being extended to actual VM configuration inside the VM - to so called guest configuration <a href="https://docs.microsoft.com/en-us/azure/governance/policy/concepts/guest-configuration">https://docs.microsoft.com/en-us/azure/governance/policy/concepts/guest-configuration</a><br />
<br />
So that's a bit of background - now on to a concrete example - validating that all virtual machines have backup enabled - how do i do that (and how do i get the data out in a format that's more easily usable than the summary screens!)<br />
<br />
I'm not going over how to define/assign the policy (there are other posts covering that sort of thing) -in my case the backup one is an inbuilt one and i just assigned it to all my subscriptions. Once that has deployed (which can take a while - and actually doesn't refresh that often).<br />
<br />
What i wanted to show was the results and how annoying the normal format is (though it does look quite nice at first glance)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Sl0HsydsqK4/XhyYAmC5GaI/AAAAAAAADYs/SN4pzForGkk2CfjxiCVXV9x2BgX6r1BSQCLcBGAsYHQ/s1600/policy.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="863" data-original-width="1600" height="172" src="https://1.bp.blogspot.com/-Sl0HsydsqK4/XhyYAmC5GaI/AAAAAAAADYs/SN4pzForGkk2CfjxiCVXV9x2BgX6r1BSQCLcBGAsYHQ/s320/policy.PNG" width="320" /></a></div>
<br />
So here we can see that I'm 69% compliant - or 171 VM's that have no backup in place - the majority of these are because they are VM's that are implicitly deployed from PaaS services (AKS, Databricks primarily for us) so in that case no backups are required.<br />
<br />
We can adjust the scope to exclude certain subscriptions/resource groups/resources to exclude these and get to our 100% compliance. Then any new resources that are flagged up as having no backups are genuine problems that we need to do something about.<br />
<br />
To then extract the list we need to do in a useful format is when i then got very frustrated - there is no export option from the portal for this - you don't even seem to be able to do select all and do a basic copy/paste of the data.<br />
<br />
Googling round when i first was looking at this i couldn't see an easy way to extract this in a different format - the only possibility seemed to be to use the REST interface to Azure policy and write some code around that - that was just more effort than i was prepared to spend.<br />
<br />
The log anlytics plugin seemed to do what i wanted until i realized that policy compliance is not part of the data that it records - so that's essentially useless - What you think is that field is just whether the policy has deployed or not - not whether it is compliant!<br />
<br />
What has been released though more recently (not sure exactly when) is a powershell module for Azure policy which we can then use to extract the data (with a little bit of extra script round it). There is a link to this here on github <a href="https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/governance/policy/assign-policy-powershell.md">https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/governance/policy/assign-policy-powershell.md</a><br />
<br />
It can be installed using the standard powershell install/import commands.<br />
<br />
Once it is installed you can do something like this basic script below to extract the data<br />
<br />
<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #007020;">Remove-Item</span> testing.csv
<span style="color: #996633;">$subscriptions</span>=<span style="color: #007020;">Get-AzureRMSubscription</span>
<span style="color: #008800; font-weight: bold;">ForEach</span> (<span style="color: #996633;">$sub</span> <span style="color: #008800; font-weight: bold;">in</span> <span style="color: #996633;">$subscriptions</span>){
<span style="color: #007020;">Select-AzureRmSubscription</span> <span style="color: #996633;">$sub</span>.SubscriptionID
<span style="color: #007020;">Get-AzPolicyState</span> -Filter <span style="background-color: #fff0f0;">"(PolicyDefinitionName eq '013e242c-8828-4970-87b3-ab247555486d')"</span> |select {<span style="color: #996633;">$_</span>.ResourceId <span style="color: #333333;">-replace</span> <span style="background-color: #fff0f0;">'.*/'</span>},<span style="background-color: #ffaaaa; color: red;">@</span>{Name=<span style="background-color: #fff0f0;">"subscription"</span><span style="background-color: #ffaaaa; color: red;">;</span>Expression={<span style="color: #996633;">$sub</span>.Name}},ResourceId,COmplianceState|<span style="color: #007020;">Export-Csv</span> testing.csv -NoTypeInformation -Append
}
</pre>
</div>
<br />
<br />
(Apologies for the basic level of powershell here - i was battling with '/"/{ and what was actually needed)<br />
<br />
This will extract the result data for the specific policy in question - you can find the id needed for this from the definition screen of the policy - see example below<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-n4InY8aoMXA/XhybOYcfRAI/AAAAAAAADY4/Ie0nQgtbgfUugGhCbyQlJ_WOHESSEinywCLcBGAsYHQ/s1600/policy2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="254" data-original-width="1600" height="50" src="https://1.bp.blogspot.com/-n4InY8aoMXA/XhybOYcfRAI/AAAAAAAADY4/Ie0nQgtbgfUugGhCbyQlJ_WOHESSEinywCLcBGAsYHQ/s320/policy2.PNG" width="320" /></a></div>
<br />
There doesn't seem to just extract all the results for all the subscriptions in one go so i do a small loop to set the context for each subscription in turn and append that data to the csv extract.<br />
<br />
The extract is fetching the vmname, subscription name, full resource id and the compliance status (those are the 4 things that follow the select in the command above).<br />
<br />
The finished csv then looks like this (heavily redacted :-))<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-OE8HMLKDlHE/XhycnOQ9KwI/AAAAAAAADZE/_Lz0H51MBMEAUzMnCcYSTt2yCRbhmUNqQCLcBGAsYHQ/s1600/policy3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="125" data-original-width="1080" height="37" src="https://1.bp.blogspot.com/-OE8HMLKDlHE/XhycnOQ9KwI/AAAAAAAADZE/_Lz0H51MBMEAUzMnCcYSTt2yCRbhmUNqQCLcBGAsYHQ/s320/policy3.PNG" width="320" /></a></div>
<br />
You can then work through the results and fix the non compliant ones or filter them out if they are not required.<br />
<br />
I've just scratched the surface here - it's capable of much more than this - including auto fixing or non compliant resources!<br />
<br />
I'm sure the portal team will enhance the screens further to make them more friendly - the other missing thing is a 'deep link' to be able to go straight to policy results - for example i would like to be able to send some a direct link to review a list of non compliant resources - i have no way of doing that other than saying click here, type that then click here etc to find the results.<br />
<br />
Overall though it's a really good feature and one I'm sure to be making more and more use of.DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-43775172752301674282019-11-14T11:19:00.000-08:002019-11-14T11:19:05.379-08:00KQL is my new SQL<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-z5MTgpN0EL4/Xc2ocOx2arI/AAAAAAAADYI/3A57drybROQaOYbvZmF0v639EEoIjWpigCLcBGAsYHQ/s1600/figure-367946_640.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="394" data-original-width="640" height="197" src="https://1.bp.blogspot.com/-z5MTgpN0EL4/Xc2ocOx2arI/AAAAAAAADYI/3A57drybROQaOYbvZmF0v639EEoIjWpigCLcBGAsYHQ/s320/figure-367946_640.png" width="320" /></a></div>
<br />
<br />
As i continue on my cloud journey i find the odd occasion where something i'm working with throws me right back to my days as a DBA working with Oracle (other relational database are available.....).<br />
<br />
This has never been more true than when i talk about KQL (yes KQL , that's not a typo)<br />
<br />
KQL or "Kusto Query Language" to give it it's full name is the language used to work with a lot of the data sources Azure provides - everything from log analytics, to Azure Sentinel to Azure resource explorer - all of these data sources are in the same underlying format which we can then access using KQL.<br />
<br />
Now I'm not going to give a full overview here of everything that can be done - you can read the Microsoft docs for that - instead what I'm going to do is just give you a simple example and then explain what that is doing translating the KQL into what that would look like in SQL.<br />
<br />
So here is the 'raw' KQL code<br />
<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"> (Resources <span style="color: #333333;">|</span><span style="color: #008800; font-weight: bold;">where</span> <span style="color: #008800; font-weight: bold;">type</span> <span style="color: #333333;">==</span> <span style="color: #aa6600;">"microsoft.web/sites"</span> <span style="color: #333333;">|</span> project name,resourceGroup<span style="color: #333333;">=</span>toupper(resourceGroup),sub<span style="color: #333333;">=</span>toupper(subscriptionId)
<span style="color: #333333;">|</span> <span style="color: #008800; font-weight: bold;">join</span> kind<span style="color: #333333;">=</span><span style="color: #008800; font-weight: bold;">inner</span> (ResourceContainers <span style="color: #333333;">|</span><span style="color: #008800; font-weight: bold;">where</span> <span style="color: #008800; font-weight: bold;">type</span><span style="color: #333333;">==</span><span style="background-color: #fff0f0;">'microsoft.resources/subscriptions/resourcegroups'</span> <span style="color: #333333;">|</span>project name<span style="color: #333333;">=</span>toupper(name),tags.IteraplanID,coalesce(tags.<span style="color: #008800; font-weight: bold;">Owner</span>,tags.<span style="color: #008800; font-weight: bold;">owner</span>)) <span style="color: #008800; font-weight: bold;">on</span> <span style="background-color: #ffaaaa; color: red;">$</span><span style="color: #008800; font-weight: bold;">left</span>.resourceGroup<span style="color: #333333;">==</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="color: #008800; font-weight: bold;">right</span>.name)
<span style="color: #333333;">|</span> <span style="color: #008800; font-weight: bold;">join</span> kind<span style="color: #333333;">=</span>leftouter (ResourceContainers <span style="color: #333333;">|</span> <span style="color: #008800; font-weight: bold;">where</span> <span style="color: #008800; font-weight: bold;">type</span><span style="color: #333333;">==</span><span style="background-color: #fff0f0;">'microsoft.resources/subscriptions'</span> <span style="color: #333333;">|</span> project SubName<span style="color: #333333;">=</span>name,sub<span style="color: #333333;">=</span>toupper(subscriptionId)) <span style="color: #008800; font-weight: bold;">on</span> <span style="background-color: #ffaaaa; color: red;">$</span><span style="color: #008800; font-weight: bold;">left</span>.sub<span style="color: #333333;">==</span><span style="background-color: #ffaaaa; color: red;">$</span><span style="color: #008800; font-weight: bold;">right</span>.sub
</pre>
</div>
<br />
or as a nicer picture this<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-BH9AdPBzNkQ/Xc2kImMZxzI/AAAAAAAADX0/s3SVcyxe87wvR23h4C3s0phBQRY14RHRQCLcBGAsYHQ/s1600/kql1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="498" data-original-width="1600" height="99" src="https://1.bp.blogspot.com/-BH9AdPBzNkQ/Xc2kImMZxzI/AAAAAAAADX0/s3SVcyxe87wvR23h4C3s0phBQRY14RHRQCLcBGAsYHQ/s320/kql1.PNG" width="320" /></a></div>
<br />
When run it outputs this dataset (with lots more rows of course):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-aqZWRh_W-WU/Xc2ktOXKDcI/AAAAAAAADX8/VVwJJiTZ81s8_OXyAJtlPy2scVQnrKtFQCLcBGAsYHQ/s1600/kql2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="182" data-original-width="1571" height="37" src="https://1.bp.blogspot.com/-aqZWRh_W-WU/Xc2ktOXKDcI/AAAAAAAADX8/VVwJJiTZ81s8_OXyAJtlPy2scVQnrKtFQCLcBGAsYHQ/s320/kql2.PNG" width="320" /></a></div>
<br />
Which i do appreciate when redacted is actually pretty useless as a picture.<br />
<br />
What it's actually showing is a 'CMDB' type query where i take information from 3 sources - subscriptions/resource groups/websites and join then together so i can see all the detail in one place.<br />
<br />
In our case we are tagging everything at the resource group level (and not on every resource) - so to know which person this belongs to we need information from the parent resource group to supplement what we have a just the resource group level. We further want to identify the subscription (by name) that this belongs to.<br />
<br />
So to breakdown the query a little more<br />
<br />
This first section just fetches data from the 'Resources' datasource - which i'm just considering as a table really, we then only want the 'rows' that are about websites and we only want to display the name,resourcegroup and subscription id columns from that<br />
<br />
<b> (Resources |where type == "microsoft.web/sites" | project name,resourceGroup=toupper(resourceGroup),sub=toupper(subscriptionId)</b><br />
<br />
The next line then joins to the resourcegroups table and we want just the rg name, the iteraplan tag and the owner tag columns (note my use of the coalesce function as the column names are case sensitive and our tags are not in consistent case and have 2 varieties- we join to that on the resource group name<br />
<br />
<b>| join kind=inner (ResourceContainers |where type=='microsoft.resources/subscriptions/resourcegroups' |project name=toupper(name),tags.IteraplanID,coalesce(tags.Owner,tags.owner)) on $left.resourceGroup==$right.name)</b><br />
<br />
We finally join to the subscriptions table where we just want the name and id of the subscription - we do that join on the subscription id<br />
<br />
<b>| join kind=leftouter (ResourceContainers | where type=='microsoft.resources/subscriptions' | project SubName=name,sub=toupper(subscriptionId)) on $left.sub==$right.sub</b><br />
<br />
<br />
In other words this query would look something like this in Oracle style SQL (I'm not translating into ANSI as i can never remember the join syntax after 20 years doing it the oracle way :-))<br />
<br />
<b>Select w.website_name,r.resourcegroup,s.subscription,coalesce(r."owner",r."Owner") as owner, iteraplanid</b><br />
<b>from</b><br />
<b>subscriptions a, resourcegroup r, websites w</b><br />
<b>where w.resourcegroup=r.resourcegroup</b><br />
<b>and s.subid=r.subid</b><br />
<br />
Which i think will be much more familiar to my DBA colleagues<br />
<br />
There is a while load of stuff to read up here and this is quite a nice doc on some of the KQL vs SQL comparisons<br />
<br />
<a href="https://docs.microsoft.com/en-us/azure/kusto/query/sqlcheatsheet">https://docs.microsoft.com/en-us/azure/kusto/query/sqlcheatsheet</a><br />
<br />
The use of this stuff seems to be getting more and more in the Azure space and there are whole lot of visuals it can do also. This really looks like an interesting area to get involved in and prior SQL knowledge is very useful - it's just learning a new syntax.....<br />
<br />
<br />
<br />
<br />
<br />
<br />DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com6tag:blogger.com,1999:blog-302298286928742422.post-19646986330259230622019-08-31T01:15:00.001-07:002019-08-31T01:15:35.867-07:00Adventures in dns - conditional forwarding reverse dns requests<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-vN_bZwC_RpU/XWosmC4feYI/AAAAAAAADXY/Bv8boydYYs857-ThnVRo7-lFKvXtXEtcgCLcBGAs/s1600/frog-949601_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="415" data-original-width="640" height="207" src="https://1.bp.blogspot.com/-vN_bZwC_RpU/XWosmC4feYI/AAAAAAAADXY/Bv8boydYYs857-ThnVRo7-lFKvXtXEtcgCLcBGAs/s320/frog-949601_640.jpg" width="320" /></a></div>
<br />
<br />
Well that title is a bit of a mouthful but what i actually want to do is quite simple - take the case in the screenshot below - i can quite happily do the forward lookup (i.e. resolve a name to an ip address), however if i then try and lookup the ip address in dns to see what name that's linked to it doesn't work......<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-xrk3SY9ZeXA/XWomwVTEsSI/AAAAAAAADXA/OfoW8Cw8KPYaM99bTPcC1gM0SRNphQHlQCLcBGAs/s1600/dns1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="182" data-original-width="782" height="74" src="https://1.bp.blogspot.com/-xrk3SY9ZeXA/XWomwVTEsSI/AAAAAAAADXA/OfoW8Cw8KPYaM99bTPcC1gM0SRNphQHlQCLcBGAs/s320/dns1.png" width="320" /></a></div>
<br />
<br />
So in the case above i quite happily resolve portal.azure.com to an ip (this is kind of how the internet works so it's pretty fundamental that it does work), However if i take that ip and look it up it can't be resolved back to a name.<br />
<br />
This is generally not a problem as you don't often need to do this, in our specific case though we have a use case where we need to be able to resolve an on premises ip address back to a name - so how can we deal with that?<br />
<br />
After a bit of research it seems it's possible by adding a conditional forwarder - now normally these are used for the forward lookups I mentioned just previously resolving names to ip addresses. A classic case is for an environment where you have multiple dns servers all managing their own domain. Instead of replicating the information in all dns servers we just set a conditional forwarder - so for example if i send a request to a dns server that looks after internaldomain.com for a name test.otherinternaldomain.com by default it won't be able to do anything with it and will just fail to resolve.<br />
<br />
My adding a conditional forwarder we are just 'passing the buck' - i.e. getting someone else who knows what to do look it up for us. In the case above we just add a record in internaldomain.com that says if you get any requests for otherinternaldomain.com send them to x.x.x.x (the ip address of the other dns server) - that then resolves it sends us back the result which we then pass back to the client.<br />
<br />
This is all pretty standard stuff.<br />
<br />
I personally had never had to do this for a reverse lookup and wasn't sure it was possible - there is no option to directly add a conditional reverse lookup.<br />
<br />
It turns out it is possible though - you just need a specially formatted conditional forwarder.<br />
<br />
Take the case that the range of ip's i want to be able to resolve back to name all exist in the 10.20.30.x subnet. I need to add a forwarder for that specific subnet to the dns server that has those reverse records in it - to do that is quite simple.<br />
<br />
The screenshot below shows how to add that in Microsoft dns manager<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-D6a6jA1UcyA/XWoqWuQrDGI/AAAAAAAADXM/7jXSwLs3PyE_nxxwib8OKI1TF2QhL-IFwCLcBGAs/s1600/dns2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="708" data-original-width="467" height="320" src="https://1.bp.blogspot.com/-D6a6jA1UcyA/XWoqWuQrDGI/AAAAAAAADXM/7jXSwLs3PyE_nxxwib8OKI1TF2QhL-IFwCLcBGAs/s320/dns2.png" width="211" /></a></div>
<br />
The key part being the format of the 'domain' we wan't to forward - in this case it's not a domain really as it's an ip - but we have to add it here.<br />
<br />
The name we add (as you can see in the pic) is 30.20.10.in-addr.arpa - this specially formatted name is in the format that dns understands as a 'reverse conditional forwarder' - note the ip numbers have to be reversed so 10.20.30 becomes 30.20.10 in the record (that is not a mistake :-)). The other element we need is the ip address of the server that has these reverse records in it - in my case i just picked 10.10.10.10 as an example.<br />
<br />
And that's it - now it should work.<br />
<br />
If i now do nslookup 10.20.30.1, a request is sent to my normal dns server, this server hasn't got the info but it now has a forwarder to say send it on to 10.10.10.10. 10.10.10.10 then checks its records finds the name and sends that back to my dns server which then sends back the name to me.<br />
<br />
Conditional reverse lookups done......<br />
<br />
<br />
<br />DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com6tag:blogger.com,1999:blog-302298286928742422.post-53245539652119610902019-07-28T12:59:00.002-07:002019-07-28T12:59:43.259-07:00When the cloud dies - getting into your windows VM<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-802QWb4uSPM/XT3-fHep9eI/AAAAAAAADWs/c3YBhRcwGbwBWNjbsiBlWg6dcvVESE_uwCLcBGAs/s1600/shark-47634_640.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="547" data-original-width="640" height="273" src="https://1.bp.blogspot.com/-802QWb4uSPM/XT3-fHep9eI/AAAAAAAADWs/c3YBhRcwGbwBWNjbsiBlWg6dcvVESE_uwCLcBGAs/s320/shark-47634_640.png" width="320" /></a></div>
<br />
<br />
Now the cloud never fails right....?<br />
<br />
Well just assume for a minute that's it's not this magical thing that never breaks and is fallible, and then take in one step further and imagine some kind of really nasty scenario where the managed domain controller is toasted and you see a horrible message like the screenshot below:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-5BswH4kpr9c/XT341gJO5CI/AAAAAAAADVg/KW0Tny40qcM44Om1OBHuhluM0v2d1kguACLcBGAs/s1600/nla0.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="399" data-original-width="701" height="182" src="https://1.bp.blogspot.com/-5BswH4kpr9c/XT341gJO5CI/AAAAAAAADVg/KW0Tny40qcM44Om1OBHuhluM0v2d1kguACLcBGAs/s320/nla0.PNG" width="320" /></a></div>
<br />
Game over right?<br />
<br />
How do you ever get on to the machine to fix that - even if we restore from backup we still have the NLA issue and we can never talk to the domain again (even when the domain is fixed this machine was somehow linked to the previous life of the domain in some mysterious way).<br />
<br />
Well it turns out there is a way - you just have to be a bit creative and persistent to sort this out - so this is how i did it....<br />
<br />
First up navigate to the serial console of the VM - see the picture below - in most cases though this is not enabled and you initially can't use it.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-h5bsnAFvqaE/XT341mbV3vI/AAAAAAAADVY/Cpm3GZpFiNI4C8is3Yp2Dgp0wHPtXWjrACLcBGAs/s1600/nla1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="398" data-original-width="1166" height="109" src="https://1.bp.blogspot.com/-h5bsnAFvqaE/XT341mbV3vI/AAAAAAAADVY/Cpm3GZpFiNI4C8is3Yp2Dgp0wHPtXWjrACLcBGAs/s320/nla1.PNG" width="320" /></a></div>
<br />
The first stage to enable that is then to switch on boot diagnostics - by going to this screen:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-XtY2BnBZeK0/XT343E-dB1I/AAAAAAAADVo/f3O0H_spw_U3PeU0Lyxr4x1omdovdZ4EgCLcBGAs/s1600/nla2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="488" data-original-width="1014" height="154" src="https://1.bp.blogspot.com/-XtY2BnBZeK0/XT343E-dB1I/AAAAAAAADVo/f3O0H_spw_U3PeU0Lyxr4x1omdovdZ4EgCLcBGAs/s320/nla2.PNG" width="320" /></a></div>
<br />
Then switch to on:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-xZmMnApJkcQ/XT344KSV0PI/AAAAAAAADV0/QFVEMYT6xygEqKm7E8AJs9YusgOgW49DQCLcBGAs/s1600/nla3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="275" data-original-width="318" src="https://1.bp.blogspot.com/-xZmMnApJkcQ/XT344KSV0PI/AAAAAAAADV0/QFVEMYT6xygEqKm7E8AJs9YusgOgW49DQCLcBGAs/s1600/nla3.PNG" /></a></div>
<br />
After that we have to make sure the EMC is enabled - to do that navigate to the run command screen of the VM and choose that option - then click run:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Zhx4Hd0kpqA/XT344cPNI3I/AAAAAAAADV4/LMob398X4ZMisT1eJ9U-hWq-4U7bj5rfwCLcBGAs/s1600/nla4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="418" data-original-width="930" height="143" src="https://1.bp.blogspot.com/-Zhx4Hd0kpqA/XT344cPNI3I/AAAAAAAADV4/LMob398X4ZMisT1eJ9U-hWq-4U7bj5rfwCLcBGAs/s320/nla4.PNG" width="320" /></a></div>
<br />
<br />
After a couple of minutes you should see this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-LiePVRCh0QY/XT344d42NQI/AAAAAAAADV8/NY37Phf5JiwanV79iaBRMjjrbV1LpOeogCLcBGAs/s1600/nla5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="416" data-original-width="377" height="320" src="https://1.bp.blogspot.com/-LiePVRCh0QY/XT344d42NQI/AAAAAAAADV8/NY37Phf5JiwanV79iaBRMjjrbV1LpOeogCLcBGAs/s320/nla5.PNG" width="290" /></a></div>
<br />
Now for me i had to reboot the server before moving on - this may or may not be necessary for you.<br />
<br />
Now when i go back to the console screen i see this output - and it's as if we are sat in a freezing computer room in front of a slide out screen and KVM setup....<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-G_F7Hjxbezk/XT344qbMjYI/AAAAAAAADWA/OrQfxCJZr3Ye4yZfgmXWWqcuI1j0_HIrACLcBGAs/s1600/nla6.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="324" data-original-width="685" height="151" src="https://1.bp.blogspot.com/-G_F7Hjxbezk/XT344qbMjYI/AAAAAAAADWA/OrQfxCJZr3Ye4yZfgmXWWqcuI1j0_HIrACLcBGAs/s320/nla6.PNG" width="320" /></a></div>
<br />
At this prompt we then need to launch a dos command window - to do that we run the following set of commands:<br />
<br />
<b>cmd</b><br />
<br />
(that creates a channel that we then need to connect to)<br />
<br />
<b>ch -sn Cmd0001</b><br />
<br />
That attached to the named channel i just created (this is likely to be Cmd0001 in most cases - but the previous cmd command will output the name to you)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-J3y52FHkX8s/XT345LNSxdI/AAAAAAAADWE/9KlR03MbutYdLgrDBGfSG68mw6twCEMTQCLcBGAs/s1600/nla7.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="598" data-original-width="691" height="276" src="https://1.bp.blogspot.com/-J3y52FHkX8s/XT345LNSxdI/AAAAAAAADWE/9KlR03MbutYdLgrDBGfSG68mw6twCEMTQCLcBGAs/s320/nla7.PNG" width="320" /></a></div>
<br />
Once i do that i get a login prompt<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-_oBSxeDMIkU/XT345iOtdcI/AAAAAAAADWI/PXacMcEE9UA11JCiS8scw6Yz8z9Pihx6QCLcBGAs/s1600/nla8.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="193" data-original-width="457" height="135" src="https://1.bp.blogspot.com/-_oBSxeDMIkU/XT345iOtdcI/AAAAAAAADWI/PXacMcEE9UA11JCiS8scw6Yz8z9Pihx6QCLcBGAs/s320/nla8.PNG" width="320" /></a></div>
<br />
Now i just need to login as a local account (now if you don;t have a local account password saved away - you may be able to reset it using the VM agent - which is generally always on and working for windows VM's).<br />
<br />
Just type the username followed by . for domain (i.e. local account) followed by the password.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-jwtsZRoDKng/XT3454QFneI/AAAAAAAADWM/bQl8Cg7d9Kgrtp3E1geVEwEVckwYVV8XACLcBGAs/s1600/nla9.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="130" data-original-width="416" height="100" src="https://1.bp.blogspot.com/-jwtsZRoDKng/XT3454QFneI/AAAAAAAADWM/bQl8Cg7d9Kgrtp3E1geVEwEVckwYVV8XACLcBGAs/s320/nla9.PNG" width="320" /></a></div>
<br />
Then you should magically be at a dos prompt!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-J8fcfCL7y-8/XT341vsm2zI/AAAAAAAADVc/FcfX55c3T34z0xLzsFJjaIUK6NsGaqzdACLcBGAs/s1600/nla10.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="105" data-original-width="571" height="58" src="https://1.bp.blogspot.com/-J8fcfCL7y-8/XT341vsm2zI/AAAAAAAADVc/FcfX55c3T34z0xLzsFJjaIUK6NsGaqzdACLcBGAs/s320/nla10.PNG" width="320" /></a></div>
<br />
Now we can type powershell to get to a powershell prompt and run a nifty bit of powershell to disable NLA<br />
<br />
<br />
<b>(Get-WmiObject -class Win32_TSGeneralSetting -Namespace root\cimv2\terminalservices -ComputerName localhost -Filter "TerminalName='RDP-tcp'").SetUserAuthenticationRequired(0)</b><br />
<br />
(note here is that cut and paste into the serial console seems to add rnanom extra characters so you may need to edit the command line after pasting)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-I-TNuBrKx3U/XT342dI2yeI/AAAAAAAADVk/99qHXz0S6G4AO7YALZMoxmEYzF7Xyz2EgCLcBGAs/s1600/nla11.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="443" data-original-width="744" height="190" src="https://1.bp.blogspot.com/-I-TNuBrKx3U/XT342dI2yeI/AAAAAAAADVk/99qHXz0S6G4AO7YALZMoxmEYzF7Xyz2EgCLcBGAs/s320/nla11.PNG" width="320" /></a></div>
<br />
After that if we go back to remote desktop connection we now get past the NLA warning and instead see this.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-qbqJfoHerzY/XT343J59eEI/AAAAAAAADVs/Oas-n9XAds0vELI8jzuPv1RAY3z-bwnzQCLcBGAs/s1600/nla12.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="703" data-original-width="1250" height="179" src="https://1.bp.blogspot.com/-qbqJfoHerzY/XT343J59eEI/AAAAAAAADVs/Oas-n9XAds0vELI8jzuPv1RAY3z-bwnzQCLcBGAs/s320/nla12.PNG" width="320" /></a></div>
<br />
Now if i explicitly type .\username and then password at the login prompt we can get in - job done (note in my case the keyboard was confused as @ was typing as " - so click on the little icon to confirm what you are typing is actually what the machine is receiving!)<br />
<br />
<br />
Neat huh - snatched victory from the jaws of defeat!DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0tag:blogger.com,1999:blog-302298286928742422.post-53128953370133100892019-07-16T13:24:00.001-07:002019-07-16T13:24:28.884-07:00Azure scalesets - a useful technique?<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Cc-ulpSmwE0/XS4yb2OS-xI/AAAAAAAADVI/MBys4xmnxRAabMBRDaTQqVeNG-KGnVuLgCLcBGAs/s1600/green-tree-python-8343_640.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://1.bp.blogspot.com/-Cc-ulpSmwE0/XS4yb2OS-xI/AAAAAAAADVI/MBys4xmnxRAabMBRDaTQqVeNG-KGnVuLgCLcBGAs/s320/green-tree-python-8343_640.jpg" width="320" /></a></div>
<br />
<br />
This post is describing how to take an existing VMSS running instance and then using that as a baseline to create a new base image for all other instances going forward. Now ordinarily you would think why on earth are you doing this - surely you already have a defined base image and all instances are a copy of that?<br />
<br />
Well in this case the instances had been edited directly and quite a lot of work had gone into that and we didn't want to lose the work that had gone on (i know this situation should never have happened, but it did - we'll say no more than that).<br />
<br />
So how can we deal with that - VMSS instances are not 'normal' machines - i.e. they don't show up in virtual machines for a start - how can we even access details of the hard drive to be able to progress this?<br />
<br />
Well lets start off with how we find the base os disk that it's using:<br />
<br />
This az command shows details of just one of the running instances (make sure to set correct subscription first)<br />
<br />
<b>az account set -s yoursubname</b><br />
<b>az vmss show -g yourrgname --name yourvmssname--instance yourinstanceid</b><br />
<br />
The instance id can be found via other command line means or it can be seen in the instances screen of the VMSS itself - in the picture below you can see each instance is suffixed with the instance id<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-dIXZz66CHgo/XS33J3CrabI/AAAAAAAADUQ/tu5Ujg3tj8whChBTcM-oJHa4xywZBZe8QCLcBGAs/s1600/vmss1.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="455" data-original-width="1206" height="120" src="https://1.bp.blogspot.com/-dIXZz66CHgo/XS33J3CrabI/AAAAAAAADUQ/tu5Ujg3tj8whChBTcM-oJHa4xywZBZe8QCLcBGAs/s320/vmss1.PNG" width="320" /></a></div>
<br />
<br />
So in my case i chose instance id 14 as this was the one the developers told me was the one that was in the best state to take as the base image.<br />
<br />
Within the output from that command will be a section like this<br />
<br />
<b> "image": null,</b><br />
<b> "managedDisk": {</b><br />
<b> "id": <span style="color: orange;">"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxxx/providers/Microsoft.Compute/disks/xxxxxxx_disk1_d7d1b3b73b754a6684f540ee1ebe4a4c",</span></b><br />
<b> "resourceGroup": "xxxxxxx",</b><br />
<b> "storageAccountType": "Standard_LRS"</b><br />
<b> },</b><br />
<br />
<br />
The key part being the bit in orange - this is where the actual disk is held. Once we have this bit of information we need to snapshot that disk.<br />
<br />
az snapshot create -g yourrgname -n yoursnapshotname--source <b><span style="color: orange;">"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxxx/providers/Microsoft.Compute/disks/xxxxxxx_disk1_d7d1b3b73b754a6684f540ee1ebe4a4c"</span></b><br />
<br />
So i now have a snapshot of that running vmss instance - now i need to create a disk based on that snapshot.<br />
<br />
The main inputs you need are the snaphot location (which is in the output from the previous command) and the size of the disk - again in the info from the last command.<br />
<br />
<b>az disk create --resource-group yourrgname --name giveyourdiskaname --sku Standard_LRS --size-gb 127 --source "the path returned from the last command for the snapshot location"</b><br />
<br />
Now we have a complete copy of that os disk and we can move on to the next steps.<br />
<br />
Now for me this gets a little more complicated as we have domain joined windows machines - so i have to copy the disk into an isolated area so it doesn't try and talk to the domain and mess everything up with the running image as at the moment they have the exact same SID. If you have a standalone machine (or a linux one then you can maybe simplify the next section).<br />
<br />
So to copy the disk to an isolated area i now switch to powershell (this is likely also possible in az command line but i found a simple example i could cut and paste from so that's what i used......)<br />
<br />
The short script below just copies the disk to another subscription - honestly this feels overly long and I'm sure there is a shorter version than this.<br />
<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">#Provide the subscription Id of the subscription where managed disk exists</span>
<span style="color: #996633;">$sourceSubscriptionId</span>=<span style="background-color: #fff0f0;">"xxxxxx"</span>
<span style="color: #888888;">#Provide the name of your resource group where managed disk exists</span>
<span style="color: #996633;">$sourceResourceGroupName</span>=<span style="background-color: #fff0f0;">'yourrgname'</span>
<span style="color: #888888;">#Provide the name of the managed disk</span>
<span style="color: #996633;">$managedDiskName</span>=<span style="background-color: #fff0f0;">'richvmssdisk'</span>
<span style="color: #888888;">#Set the context to the subscription Id where Managed Disk exists</span>
<span style="color: #007020;">Select-AzSubscription</span> -SubscriptionId <span style="color: #996633;">$sourceSubscriptionId</span>
<span style="color: #888888;">#Get the source managed disk</span>
<span style="color: #996633;">$managedDisk</span>= <span style="color: #007020;">Get-AzDisk</span> -ResourceGroupName <span style="color: #996633;">$sourceResourceGroupName</span> -DiskName <span style="color: #996633;">$managedDiskName</span>
<span style="color: #888888;">#Provide the subscription Id of the subscription where managed disk will be copied to</span>
<span style="color: #888888;">#If managed disk is copied to the same subscription then you can skip this step</span>
<span style="color: #996633;">$targetSubscriptionId</span>=<span style="background-color: #fff0f0;">"yyyyyyyy"</span>
<span style="color: #888888;">#Name of the resource group where disk will be copied to</span>
<span style="color: #996633;">$targetResourceGroupName</span>=<span style="background-color: #fff0f0;">'newrgname'</span>
<span style="color: #888888;">#Set the context to the subscription Id where managed disk will be copied to</span>
<span style="color: #888888;">#If snapshot is copied to the same subscription then you can skip this step</span>
<span style="color: #007020;">Select-AzSubscription</span> -SubscriptionId <span style="color: #996633;">$targetSubscriptionId</span>
<span style="color: #996633;">$diskConfig</span> = <span style="color: #007020;">New-AzDiskConfig</span> -SourceResourceId <span style="color: #996633;">$managedDisk</span>.Id -Location <span style="color: #996633;">$managedDisk</span>.Location -CreateOption Copy
<span style="color: #888888;">#Create a new managed disk in the target subscription and resource group</span>
<span style="color: #007020;">New-AzDisk</span> -Disk <span style="color: #996633;">$diskConfig</span> -DiskName <span style="color: #996633;">$managedDiskName</span> -ResourceGroupName <span style="color: #996633;">$targetResourceGroupName</span>
</pre>
</div>
<br />
<br />
The end result of that is just copying the disk to another subscription.....<br />
<br />
Now we have the disk where i want it i now need to reattach it to a VM to create a 'temporary' machine.<br />
<br />
Again many ways to do this - this time I'm using the portal - so navigate to the disk you just copied and then click the create VM button<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-vQ0TdxPeo7o/XS4sV0YcqZI/AAAAAAAADUc/EyoLEBeFvS8Jobu07d6goX1tLH5myTv0QCLcBGAs/s1600/vmss2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="411" data-original-width="1402" height="93" src="https://1.bp.blogspot.com/-vQ0TdxPeo7o/XS4sV0YcqZI/AAAAAAAADUc/EyoLEBeFvS8Jobu07d6goX1tLH5myTv0QCLcBGAs/s320/vmss2.PNG" width="320" /></a></div>
<br />
<br />
Fill in all the usual stuff and wait couple of minutes for that machine to start up.<br />
<br />
At this point we have an exact copy of the source VMSS instance.<br />
<br />
Now i log on to that machine using the local administrator account, and again in my specific case i do an additional step which is to take the machine out of the domain using the standard process. A quick reboot to complete that step and we're ready to start turning this VM into an image.<br />
<br />
Again i log in with the local admin account and run the sysprep tool to 'generalize' the machine - i.e. put it in a state where it can be used as the base of many new machines (so remove the machine specifics and have it in a 'new' machine state)<br />
<br />
To trigger sysprep i run this<br />
<br />
<b>%WINDIR%\system32\sysprep\sysprep.exe /generalize /shutdown /oobe</b><br />
<br />
That takes a few minutes and the machine is then shut down.<br />
<br />
Now we create the image - to do this i use the portal (again an be done with other tooling but this was the first way i found to do it....)<br />
<br />
Navigate to the VM page and then click capture<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-ZDzVq-bwgPo/XS4ukXrDcvI/AAAAAAAADUo/l5QoHDsS8kMFs-z77K5yyVIyxJRiYGtqQCLcBGAs/s1600/vmss3.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="188" data-original-width="1109" height="54" src="https://1.bp.blogspot.com/-ZDzVq-bwgPo/XS4ukXrDcvI/AAAAAAAADUo/l5QoHDsS8kMFs-z77K5yyVIyxJRiYGtqQCLcBGAs/s320/vmss3.PNG" width="320" /></a></div>
<br />
<br />
On the next screen just give it a name and click create<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-dFEhL7R9JIU/XS4upuDffEI/AAAAAAAADUs/tqblqZFVAGYaJzqulpZmDZTzaifp95cCQCLcBGAs/s1600/vmss4.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="797" data-original-width="733" height="320" src="https://1.bp.blogspot.com/-dFEhL7R9JIU/XS4upuDffEI/AAAAAAAADUs/tqblqZFVAGYaJzqulpZmDZTzaifp95cCQCLcBGAs/s320/vmss4.PNG" width="294" /></a></div>
<br />
Now we have an image created of our machine - we can now set that to be the base image for our scaleset and complete the circle....<br />
<br />
Just one quick step though (because of my domain joining 'issue') i just need to copy that image to the original subscription so it's visible to the scaleset. Again sure there is more than one way to do this but the simplest way i found is to navigate to the resource group containing the image and do a resource move from there - everything else i found seemed to be tens of lines of code for what should be a simple operation<br />
<br />
So fro the resource group screen containing the image click on the following option<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-_0OzvaR_0E0/XS4wA4kzSqI/AAAAAAAADU8/AT5EoD1WIRwPNjiaqOmoGHZCsl7wwH1uQCLcBGAs/s1600/vmss5.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="495" data-original-width="1466" height="108" src="https://1.bp.blogspot.com/-_0OzvaR_0E0/XS4wA4kzSqI/AAAAAAAADU8/AT5EoD1WIRwPNjiaqOmoGHZCsl7wwH1uQCLcBGAs/s320/vmss5.PNG" width="320" /></a></div>
<br />
<br />
That will then be checked (but the image will have no dependencies and can easily be moved) - and it can go back to the original resource group where everything started out.<br />
<br />
The final step is to now set this 'new' image to be the base image for the scaleset - i switch this time back to az command line as it's simple here and run the following<br />
<br />
<b>az vmss update --resource-group yourrgname --name yourscalesetname --set virtualMachineProfile.storageProfile.imageReference.id=/subscriptions/xxxxxxx/resourceGroups/yourrgname/providers/Microsoft.Compute/images/yourimagename</b><br />
<br />
Now all of the instances in the scaleset are replaced with the 'new' image and any new instances added will also use that one.<br />
<br />
And there you have it a 'simple' way to take a VMSS instance as a baseline and turn that back into a usable image for all the other instances.<br />
<br />
Hopefully that's useful - even if it only highlights multiple different ways of interacting with the azure platform and using functionality that is maybe not an everyday thing to want to do.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />DBAHarrisonhttp://www.blogger.com/profile/16901239165682958859noreply@blogger.com0