NLog Target for Azure ServiceBus Event Hub

NLog is a popular open source logging framework for .Net applications. It writes to various destinations via Target. It has a large number of Targets available available. I created a NLog Target that can send message to Azure ServiceBus EventHub. You can get the source code and documentation here: https://github.com/rajinders/nlog-targets-azureeventhub

I also created a NuGet package which you can download from here: https://www.nuget.org/packages/NLog.Targets.AzureEventHub/

If you already know how to use NLog it will take you a few minutes to start using the target.

Feel free to use it and let me know if you have any suggestions for improvements.

You may be wondering why would anyone would to send logs to Azure Event Hub. Most applications use logging frameworks to write application logs. These logs are not only helpful in debugging issues they are also a source for business intelligence. There are already successful companies like Splunk, Logentries and Loggly who provide cloud based log aggregation services. If you wanted to create your own log aggregation service without write a lot of code you can do so in Azure platform. You can send you log messages to EventHub with NLog or Serilog targets for EventHub. You can leverage Azure stream analytics service to process your log streams. You can even send these logs to Power BI to create dashboards. Both Azure Event Hub and stream analytics are highly scalable. Scaling up can be achieved by simple configuration changes.

Posted in Azure, EventHub, NLog, ServiceBus | Tagged , , , | 2 Comments

Bloggers Guide to Azure Event Hub

I love Integration/Middleware space. In the spring of 2004 I was working on a large implementation for a client. We had to integrate externally, internally with a large number of systems and we also had a need for long running processes. We were already using BizTalk 2002. We came to know about a radical new version of BizTalk Server called BizTalk 2004. It was based on .Net and was re-written from scratch. As soon as we learned about its capabilities we knew that it was a far better product for what we were implementing. We made a decision to use BizTalk Server 2004 Beta during our development. Since the product we were building was releasing in fall/winter we knew that it will become generally available before we go live. Making the decision to switch to BizTalk 2004 was an easy decision. Hard part came when I had to design 30 plus long running processes using BizTalk. There wasn’t any documentation. There were no BizTalk experts we could reach out it. At that time somebody began publishing a guide called “Bloggers Guide to BizTalk”. It was a compiled help file which included blog posts from authors all over the world. Without this guide we would have failed to implement our solution using BizTalk 2004.

I  still like middleware space but I have added Cloud, IOT, DevOps to my list of technologies I use every day. Azure Event Hub is a relatively new PAAS Service that was announced in last Build conference. It became Generally Available at TechEd Barcelona in October 2014. I will use this blog post to document various resources about Azure ServiceBus EventHub service. I named it “Bloggers Guide to Azure Event Hub” as an Ode to “Bloggers Guide to BizTalk”. I want to make it easier for anybody learning about Azure Event Hub to find helpful resources that will quickly get them started. I will make weekly updates to keep it current.

 

Videos

Introduction to EventHub from TechEd Barcelona: http://channel9.msdn.com/Events/TechEd/Europe/2014/CDP-B307

Cloud Cover Show about Event Hub: http://search.channel9.msdn.com/content/result?sid=b8411351-e4b2-4fff-bb3c-a64b566c7d99&rid=85437dcd-37ee-4965-ab09-a3d4013c30d7

 

MSDN Documentation

Event Hub Overview: https://msdn.microsoft.com/en-us/library/azure/dn836025.aspx

Eventh Hubs Programming Guide: https://msdn.microsoft.com/en-us/library/azure/dn789972.aspx

Event Hub API Overview: https://msdn.microsoft.com/en-us/library/azure/dn790190.aspx

 

Event Processor Host

EventProcessorHost class: https://msdn.microsoft.com/en-us/library/azure/microsoft.servicebus.messaging.eventprocessorhost.aspx

EventProcessor Host is covered in the API overview but I want to call this out once again as it is the easiest way to process messages out of Event Hub. It may meet the needs of more 90-95% of scenarios. To get an in depth understanding of EventProcessorHost you should read this series of blog posts from Dan Rosanova.

Event Processor Host Best Practices Part I : http://blogs.msdn.com/b/servicebus/archive/2015/01/16/event-processor-host-best-practices-part-1.aspx

Event Process Host Best Practices Part II: http://blogs.msdn.com/b/servicebus/archive/2015/01/21/event-processor-host-best-practices-part-2.aspx

 

Code Samples

ServiceBus Event Hubs Getting Started : https://code.msdn.microsoft.com/windowsapps/Service-Bus-Event-Hub-286fd097

Scale Out Event Processing with Event Hub: https://code.msdn.microsoft.com/windowsapps/Service-Bus-Event-Hub-45f43fc3

ServiceBus Event Hub Direct Receiver: https://code.msdn.microsoft.com/windowsapps/Event-Hub-Direct-Receivers-13fa95c6

 

Reference Architecture

data-pipeline

https://github.com/mspnp/data-pipeline

If you are looking for reference architecture and code sample for how to build a scalable real world application data-pipeline will be helpful to you.

Real-Time Event Processing with Microsoft Azure Stream Analytics

http://azure.microsoft.com/en-us/documentation/articles/stream-analytics-real-time-event-processing-reference-architecture/

This reference architecture is about Stream Analytics but it shows how Event Hub is a core part of the real-time event processing architecture.

 

Tools

ServiceBus Explorer

https://code.msdn.microsoft.com/windowsapps/Service-Bus-Explorer-f2abca5a

Anybody developing ServiceBus application should be using this tool. It has Queues, Topics and EventHub support as well.

 

Provisioning

If you want to provision EventHub in Azure you options are:

1. Use the Azure Management Portal

2. Use the SDK to provision it in code

3. Use the REST API

4. Paolo Salvatori created a PowerShell Script that invokes the REST API to create Service Bus namespace and EventHub. This is the script I am using my current project. http://blogs.msdn.com/b/paolos/archive/2014/12/01/how-to-create-a-service-bus-namespace-and-an-event-hub-using-a-powershell-script.aspx

 

Logging Framework

EventHub makes an excellent target for ingesting logs at scale.

Serilog

Serilog is a easy to use .Net structured logging framework. It already has an EventHub Appender. You can check it out here:

https://github.com/serilog/serilog/tree/dev/src/Serilog.Sinks.AzureEventHub

 

Miscellaneous Blog Posts

Azure Event Hubs – All my thoughts by Nino Crudele: http://ninocrudele.me/2014/12/12/azure-event-hub-all-my-thoughts/

Getting Started with Azure Event Hub by Fabric Controller: http://fabriccontroller.net/blog/posts/getting-started-azure-service-bus-event-hubs-building-a-real-time-log-stream/

Azure’s New Event Hub – Brent’s Notepad: https://brentdacodemonkey.wordpress.com/2014/11/18/azures-new-event-hub/

Sending Raspberry Pi data to Event Hub and many blog posts about Azure Event Hub on Faister’s blog: http://blog.faister.com/

Sending Kinect data to Azure Event Hub at Alejandro’s blog: http://blogs.southworks.net/ajezierski/2014/11/10/azure-event-hubs-the-thing-and-the-internet/

Azure Stream Analytics, Scenarios and Introduction by Sam Vanhoutte. This blog post is about Azure Stream Analytics but both of these services will work together in many scenarios. http://www.codit.eu/blog/2015/01/azure-stream-analytics-getting-started/

Azure Event Hub Updates from a NetMF Device on Dev Mobiles blog: http://blog.devmobile.co.nz/2014/11/30/azure-event-hub-updates-from-a-netmf-device/

Posted in Uncategorized | Leave a comment

Azure Usage monitoring with Azure Automation

When you purchase an Azure subscription it comes with usage caps for various resources. As an example the usage cap for number of cores is 20. You can call use Azure Support and open a free billing support case to increase this core limit.

In the past few years I have had many clients ask for basic alerting capability when they are about to exceed their resource limits. They have Azure subscriptions that are being used by various teams and they want to know if they are reaching their Azure usage limit. They can install Azure PowerShell cmdlet and easily find the answer to this question. However they are looking for automated alerting service. I heard this request last week so I thought I will use Azure Automation to implement this solution.

There are two use case scenarios for this script:

1. It can be used by Azure  Subscription Owner to understand if they are about to exceed the resource(compute cores) quota for an Azure subscription.

2. There have been times when you keep Azure services running longer than you need them. This script will run on a schedule and inform you about the compute cores you are currently using. This could have helped me last year when I left  HD Insight cluster with 32 cores running for a month.

Azure Automation recently became generally available and it can be used to automate error prone, time consume, cloud management tasks. It leverages PowerShell based workflow scripts to automate tasks. You can learn more about it here:

http://azure.microsoft.com/en-us/services/automation/

I also highly recommend this course in virtual academy.

http://www.microsoftvirtualacademy.com/training-courses/automating-the-cloud-with-azure-automation

Here are the high level steps to implement this script.

  1. Create Azure automation account
  2. Create Credential Asset for Azure Administration
  3. Create Credential Asset for Office 365 user that will be used to send emails
  4. Create the runbook
  5. Test the runbook
  6. Publish the runbook
  7. Link it to a schedule
  8. View Job history

Create Azure Automation Account using Azure Portal

You can do so by selecting Automation and “+ Create” button.

image

Right now you can create Azure Automation account in “East US”, “Southeast Asia” and “West Europe” only.

When you create an account in a region it stores its assets in that region. However this account can automate tasks in any other region.

image

Creating Credentials

Azure Active Directory for Azure Credentials

Create a new user in Azure Active Directory

Use Azure Portal and select “Active Directory”

Select your Active Directory instance and navigate to “User” section and use “Add User” button in the bottom toolbar.

image

Select “new user in your organization”

Enter the user name.

image

Enter user information in the User Profile section.

image

Press “Create” button and it will show you the temporary password.

Sign into the Azure Active Directory as this newly created user and change the temporary password.

Sign in to Windows Azure Active Directory

CoAdmin Access

Make this new user a Co-administrator for the Azure subscription you want to monitor.

You do this by select “Settings” –>Administrators and press “Add” button in the bottom toolbar

image

 

On the “Add A CO-Administrator” screen specify the Azure AD user you just created and select the appropriate subscription from the list below.

image 

Create an asset of type Credentials in your automation account

Automation accounts has assets that can be used by runbooks. These are convenient place to securely store user names, passwords and connection strings.

We need to create Credentials to get access to the Azure subscription. Select your newly created Azure automation account and select “Assets”. Press in the “Add Setting” button

image

Select “add credential”

image

There are two options for credentials:

1. Windows PowerShell Credential

2. Certificate

You need to select “Windows Azure PowerShell”

image

Enter the name and password of the Azure AD user that is also a Co-Administrator to the Azure subscription you are monitoring.

image

Create Office 365 Credentials to send out emails

I have Office 365 small business account. I have a separate Azure subscription. Until now I never had a need to use Active Directory associated with my Office 365 account. Here are the steps to setup credentials for Office 365 as assets in Azure Automation.

Use Azure Management Portal   New->App Services ->Active Directory-Directory-Custom Create

On the Add Directory popup you need to select “Use Existing directory”

image

You will be asked to sign in as administrator for your Office 365 account.

Once Office 365 Directory has been added to the Portal you can see the list of existing users or add a new user that will be used to send out emails about Azure  resource usage.

You need to create an asset of type Credentials in your Azure Automation account next.

The steps the create the Credentials are identical to steps to create Azure administration account. I name the credential object O365Cred.

Create Runbook

Select your Azure Automation Account and select “New->Automation->RunBook->QuickCreate to create you new RunBook.

You can use the Author Tab to create the run book. Authoring in the portal worked OK for me but I had trouble navigating through the script as it grew longer. I tried IE and Chrome and got the same results. In future I may first create the runbook in PowerShell ISC first and unit test it in the Azure portal.

Here is the script for the runbook. It looks like a normal PowerShell script with a few differences

You declare Parameters for the runbook in lines 3 through 9.

You retrieve the credentials for the Azure administration account in line 12.

You determine the current resources consumed in line 16

I want you to look at line 21 carefully as this is where I get list of services that are not in “StoppedDeallocated” status. These are the services that are incurring compute charges. Automation runbooks do not support positional parameters. I had to add –FilterScript after the Where-Object to make this expression work. Without the –FilerScript I was getting the following error:

azure automation parameter set cannot be resolved using the specified named parameters.

You retrieve the office 365 credentials in line 30

You send email with Send-MailMessage cmdlet in line 36

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
workflow Get-CurrentAzureResourceUsage
{
    param (
       [Parameter(Mandatory=$False)]
       [string] $AzureAdmin = “autoadmin@xxxxxxxxxx.onmicrosoft.com”,   
       [Parameter(Mandatory=$False)]
       [string] $SubName = “Your sub name”, 
       [Parameter(Mandatory=$False)]
       [string] $MessageTo = “Your email address” 
    ) 
   
    $cred = Get-AutomationPSCredential -Name $AzureAdmin
   
    Add-AzureAccount -Credential $cred
   
    $details = Get-AzureSubscription -Name $SubName -ExtendedDetails
   
    $MaxCoreCount = $details.MaxCoreCount
    $CurrentCoreCount = $details.CurrentCoreCount
   
    $VMSNotDeallocated = get-azurevm | Where-Object -FilterScript { $_.Status -ne ‘StoppedDeallocated’ } | Select-Object ServiceName

    $MessageBody =  [string]::Format(“You are using {0:N0} of {1:N0} cores.”,$CurrentCoreCount, $MaxCoreCount)

    if($VMSNotDeallocated)
    {
        $MessageBody =  $MessageBody + [string]::Format(“The following services are still incurring compute charges:{0}”, $VMSNotDeallocated)       
    }
          
    $AzureO365Credential = Get-AutomationPSCredential -Name “O365Cred”
   
    if ($AzureO365Credential) 
    { 
        $MessageFrom = $AzureO365Credential.Username 
        $MessageSubject = “Azure Subscription Resource Usage”
        Send-MailMessage -To $MessageTo -Subject $MessageSubject -Body $MessageBody -UseSsl -Port 587 -SmtpServer ‘smtp.office365.com’ -From $MessageFrom -BodyAsHtml -Credential $AzureO365Credential  
    } 
    else 
    { 
      throw “AzureO365Credential not found” 
    } 
   
    Write-Output “Finished running script”
}

Testing

You can test the runbook in the portal by pressing the “Test” button in the bottom toolbar. When you run your tests you will see a window to enter the parameters. If the script runs successfully you will see the output.

Publishing

Once your testing is complete  you can press the “Publish” button to publish this run.

Here is an email received from the runbook.

You are using 2 of 20 cores.The following services are still incurring compute charges:@{ServiceName=sansoroprovtest; PSComputerName=localhost; PSShowComputerName=True; PSSourceJobInstanceId=5d402195-f0a1-4a72-8b72-c27f0633ab58}

You can schedule this run book to run on daily or hourly basis.

You can create a new schedule by selecting “Schedule” and “Link to New Schedule”

Adding a Schedule

image

image

image

You can view the Job History by looking at the Job section of the runbook.

 

image

You can drill down and view the details of the last run.

Summary section of the history shows job summary, input parameters and script output.

image

image

There is also a history section that shows information about previous executions of the runbook.

image

With this simple example I hoped to demonstrate how you can automate cloud management tasks using Azure automation runbooks. Here are a few things about Azure automation worth mentioning:

  • Runbooks can call other runbooks inline or invoke them asynchronously.
  • You can leverage integration modules as well. As an example I wanted to use Azure Resource Manager with Azure Automation but it is currently not supported. All I had to do was zip the Azure Resource Manager directory upload it and start using it. It is still not officially supported.
  • I was surprised to learn that we can call Runbooks from on premise PowerShell cmdlets.
  • You can run parallel activities in these runbooks
  • Since they are based on workflow you can save the state of a running runbook and rollback if needed.
  • Runbooks don’t support positional parameters
  • Certain cmdlets like Write-Host are not supported. I replaced Write-Host with Write-Output

Azure Automation is an easy, secure, flexible, extensible and scalable way to automate cloud management tasks. Most of your existing PowerShell scripts can be easily converted into runbooks. There is already a gallery of runbooks available in the Azure portal. You can import these runbooks and use them to automate tasks. You can find many sample runbook’s here:

https://social.technet.microsoft.com/Search/en-US/scriptcenter?query=azure%20automation&beta=0&ac=5#refinementChanges=&pageNumber=2&showMore=false

I highly recommend reading “Authoring Automation Runbooks” guide

http://technet.microsoft.com/en-us/library/dn469262.aspx

As a future enhancement you can pass in subscriptions via a JSON/XML file stored in blob storage. You can send push notifications in addition to emails.

We have barely scratched the surface of Azure Automation. It can and will play pivotal role in implementing continuous deployment and other tasks related to DevOps. I have started using Azure automation in a few of my projects. I will share my learning in future blog posts. How are you using Azure automation?

Posted in Automation, Azure, DevOps, PowerShell | Tagged , , | Leave a comment

Taking new Azure High Performance Gateway for a Test Drive

Overview

In the past month Azure platform has announced many improvements to their networking services.

One of the improvement that was announced was a release of High Performance network gateway.

You can read about High Performance Network Gateway here: http://azure.microsoft.com/blog/2014/12/02/azure-virtual-network-gateway-improvements/

Until this time the gateway network throughput was limited to 80 Mbps. New high performance gateway has network throughput of 200 Mbps. It also allows up to 30 Site to Site tunnels as compared to 10 tunnels allowed by default network gateway. I will create two virtual networks. I will add high performance network gateway to each of them. I will connect them to each other. I will create a virtual machine in each virtual network. I will test the network throughput of the high performance network gateway. I will provision the entire infrastructure with PowerShell based automation without any manual steps or logging into the Azure management portal.

Instructions to setup a VNet to VNet connection are posted here. There are a few manual steps required by these instructions but I will automate the entire provisioning and setup.

http://msdn.microsoft.com/en-us/library/azure/dn690122.aspx

Setup

Virtual Network configuration can be defined in the portal or a configuration file. In my case I have no virtual networks defined in my Azure subscription. Configuration file schema is documented here:

http://msdn.microsoft.com/en-us/library/azure/jj157100.aspx

As you can see below it has a root element NetworkConfiguration which has one child element called “VirtualNetworkConfiguration”

VirtualNetworkConfiguration has three child elements:

Dns: This is used to define DNS server names and their IP addresses

LocalNetworkSites: This is used to define Local networks which are connected to a virtual network

VirtualNetworkSites: This is where you define the virtual network configuration.

image

If you  are not comfortable with working with this XML file you can use Azure management portal to define the two virtual networks. Here is the definition of Virtaul Network named: ANetwork

This network is located in Location “Central US”. It has a MainSubnet and a GatewaySubnet

image

The second network is called BNetwork. It is also located in “Central US”.

It has a MainSubnet and a GatewaySubnet.

image

If we need to connect these two networks you will need to define them as local networks. These local networks will have the exact same address space as the networks you previously defined.

VPNGatewayAddress element defines the IP address of the gateway. Since the gateway has not been created yet I have inserted a placeholder IP address in the two local networks.

image

If you want to connect ANetwork with BNetwork all you have to do is to insert the Gateway element after Subnets in the definition of the Virtual Network as shown below. In my example ANetwork is connected with BNetworkLocal

image

If you want to connect BNetwork with ANetwork all you have to do is to insert the Gateway element after subnets in the definition of the Virtual Network as shown below. In my example BNetwork is connected with ANetworkLocal

image

If your virtual network was connected to more than one virtual network you will have to define a separate LocalNetworkSiteRef element for each of the virtual networks.

Create the virtual network

We  will use Set-AzureVNetConfig cmdlet to create the virtual network. If there are any errors in creating the virtual network this script will throw and error and abort.

I have no existing virtual networks in my subscription. If you have existing virtual network you have to export your current virtual network configuration for the portal or with Get-AzureVNetConfig and manually add your two new virtual networks. If you don’t do this your virtual network settings may be replaced.

001
002
003
004
005
006
#create/update the network configuration file
Set-AzureVNetConfig  -ConfigurationPath $vNetFilePath -ErrorAction SilentlyContinue -ErrorVariable errorVariable
if (!($?))
{
throw “Unable to set virtual network configuration included in config file: $vNetFilePath. Error detail is: $errorVariable” 
}

 

Create the High Performance Virtual Network Gateway

Previously you had created the network configuration file. You had defined your local networks under element LocalNetworkSite. You had entered place holder IP address for network gateways VPNGatewayAddress. In this step we will create the high performance network gateway’s. We will get the IP address of the gateway and update the Network Configuration file. It will take 20-30 minutes to finish this step.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
Create-GatewayUpdateConfig $Network1Name $vNetFilePath “1.1.1.1” “DynamicRouting” “HighPerformance”
Create-GatewayUpdateConfig $Network2Name $vNetFilePath “2.2.2.2” “DynamicRouting” “HighPerformance”
VPNGatewayAddress
function Create-GatewayUpdateConfig 
{ 
    param 
    ( 
        # Virtual Network name
        [Parameter(Mandatory = $true)] 
        [String] 
        $NetworkName, 
 
        # VNetConfig file that will be updated with the actual gateway IP address
        [Parameter(Mandatory = $true)] 
        [string] 
        $VNetConfigFile,

    # Placeholder IP address that will be replaced by actual gateway IP address
        # If this switch is not specified, then images from all possible publishers are considered.
        [Parameter(Mandatory = $true)] 
        [string] 
        $IPAddressToBeReplaced,

# Gateway Type
[Parameter(Mandatory = $true)] 
[ValidateSet(“StaticRouting”, “DynamicRouting”)]
[string]
$GatewayType,
 
# Gateway Size
[Parameter(Mandatory = $true)] 
[ValidateSet(“Default”, “HighPerformance”)]
[string]
$GatewaySKU
 
    ) 

#Create the gateway for virtual networks
#Gateway sku values are Default or HighPerformance
$gateway1 = Get-AzureVNetGateway -VNetName  $NetworkName -ErrorVariable errorVariable -ErrorAction SilentlyContinue | Out-Null

if($null -eq $gateway1)
{
Write-Host “Creating new gatewary for network: $NetworkName”

New-AzureVNetGateway –VNetName $NetworkName -GatewayType $GatewayType -GatewaySKU $GatewaySKU -ErrorAction SilentlyContinue -ErrorVariable errorVariable
if (!($?)) 
{ 
throw “Unable to create new AzureNetGateway for $NetworkName. Error detail is: $errorVariable” 
} 
}

$gateway1 = Get-AzureVNetGateway -VNetName  $NetworkName
$gateway1IP = $gateway1.VIPAddress
Write-Host “Gateway VIP Address for $NetworkName is $gateway1IP”

if(($null -eq $gateway1IP) -or ($gateway1IP -eq “”))
{
throw “Unable to get VIP for AzureVNetGateway for $NetworkName. Error detail is: $errorVariable” 
} 

#Get the IP address and replace the place holder IP address with actual gateway ip address
$con = Get-Content $VNetConfigFile
$con | % { $_.Replace($IPAddressToBeReplaced, $gateway1IP) } | Set-Content $VNetConfigFile
} # end of function Create-Gateway…

 

 

You can verify that your HighPerformance network gateways were created as shown below. Note the GatewaySKU of “HighPerformance”. It will be “Default” for standard gateway.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
PS C:\projects\PowerShell\HighPerfGateway\HighPerfGateway> Get-AzureVNetGateway -VNetName ANetwork

LastEventData        :
LastEventTimeStamp   : 12/5/2014 9:40:56 AM
LastEventMessage     : Successfully configured the gateway.
LastEventID          : 23005
State                : Provisioned
VIPAddress           : 23.101.120.180
DefaultSite          :
GatewaySKU           : HighPerformance
OperationDescription : Get-AzureVNetGateway
OperationId          : ba3b506b-902a-301b-b791-dcce9a90c530
OperationStatus      : Succeeded

PS C:\projects\PowerShell\HighPerfGateway\HighPerfGateway> Get-AzureVNetGateway -VNetName BNetwork

LastEventData        :
LastEventTimeStamp   : 12/5/2014 9:40:54 AM
LastEventMessage     : Successfully configured the gateway.
LastEventID          : 23005
State                : Provisioned
VIPAddress           : 23.101.116.231
DefaultSite          :
GatewaySKU           : HighPerformance
OperationDescription : Get-AzureVNetGateway
OperationId          : 8e5e15a4-873e-314c-943b-55e6e1e83cc7
OperationStatus      : Succeeded

 

Update Virtual Network Configuration

Update the configuration of Virtual Network with the updated Network Configuration file. This file was updated in the previous step with actual IP addresses of the newly created network gateways.

001
002
003
004
005
006
007
008
009
#update the vnet config with newly updated config file
Set-AzureVNetConfig  -ConfigurationPath $vNetFilePath -ErrorAction SilentlyContinue -ErrorVariable errorVariable
if (!($?))
{
throw “Unable to set virtual network configuration with updated config file: $vNetFilePath. Error detail is: $errorVariable” 
} 

 

Set the preshared keys for the two networks. After gateway key has been set it can take up to 5 minutes to verify that network connectivity has been established.

001
002
003
004
005
006
007
008
#Update the virtual network configuration.
Set-AzureVNetGatewayKey -VNetName $Network1Name -LocalNetworkSiteName BNetworkLocal -SharedKey yoursharedkey
Set-AzureVNetGatewayKey -VNetName $Network2Name -LocalNetworkSiteName ANetworkLocal -SharedKey yoursharedkey

#get the status of the virtual network
Get-AzureVnetConnection -VNetName $Network1Name
Get-AzureVnetConnection -VNetName $Network2Name

 

Here is the output of Get-AzureVnetConnection when connectivity has been established successfully.

You will notice that ConnectivityState now shows “Connected” for both the networks.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
PS C:\projects\PowerShell\HighPerfGateway\HighPerfGateway> get-azurevnetconnection -VNetName ANetwork

ConnectivityState         : Connected
EgressBytesTransferred    : 392
IngressBytesTransferred   : 6976
LastConnectionEstablished : 12/5/2014 9:44:22 AM
LastEventID               : 24401
LastEventMessage          : The connectivity state for the local network site ‘BNetworkLocal’ changed from Not
                            Connected to Connected.
LastEventTimeStamp        : 12/5/2014 9:44:22 AM
LocalNetworkSiteName      : BNetworkLocal
OperationDescription      :
OperationId               :
OperationStatus           :

PS C:\projects\PowerShell\HighPerfGateway\HighPerfGateway> get-azurevnetconnection -VNetName BNetwork

ConnectivityState         : Connected
EgressBytesTransferred    : 9529
IngressBytesTransferred   : 440
LastConnectionEstablished : 12/5/2014 9:45:52 AM
LastEventID               : 24401
LastEventMessage          : The connectivity state for the local network site ‘ANetworkLocal’ changed from Not
                            Connected to Connected.
LastEventTimeStamp        : 12/5/2014 9:45:52 AM
LocalNetworkSiteName      : ANetworkLocal
OperationDescription      :
OperationId               :
OperationStatus           :

 

Until now we have not even logged into the Azure Management Portal and we have successfully created 2 Virtual networks, created 2 high performance gateways and connected these two virtual networks. For those of you who like to view things visually I have attached the following views of the portal that show virtual networks were successfully connected.

image

image

Create Virtual Machines

In this step we will create virtual machines in the two virtual networks. We plan to use these virtual machines to test the network bandwidth of your “HighPerformance” network gateway. The script below will create a VM in ANetwork. You can use the same script to create another VM in BNetwork. I created Medium VM’s instead of Small VM’s because I wanted to make sure these VM’s have enough network bandwidth to run my tests.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
Write-Verbose “Prompt user for admininstrator credentials to use when provisioning the virtual machine(s).” 
$credential = Get-Credential 
Write-Verbose “Administrator credentials captured. Use these credentials to login to the virtual machine(s) when the script is complete.” 

$ImageName = “a699494373c04fc0bc8f2bb1389d6106__Windows-Server-2012-R2-201411.01-en.us-127GB.vhd”

#Configure the virtual machines to be created
$vm11 = New-AzureVMConfig -Name “anetworkvm1” -InstanceSize “Medium” -ImageName $ImageName |  `
                Set-AzureSubnet “MainSubnet” | `
                Add-AzureProvisioningConfig -Windows -AdminUsername $credential.GetNetworkCredential().username -Password $credential.GetNetworkCredential().password 

# Make an array of the virtual machine configuration so we can create them with 1 call
$vms = @($vm11) 

# Create a new cloud service and Deploy Virtual Machines to Virtual Network
New-AzureVM -ServiceName “anetworksvc” -Location “Central US” -VMs $vms -VNetName “ANetwork” -ErrorVariable errorVariable -ErrorAction SilentlyContinue | Out-Null
 
if (!($?)) 
{ 
    throw “Unable to create virtual machine anetowrkvm1. Error detail is: $errorVariable” 
} 
else
{
    Write-Verbose “Successfully created virtual machine anetworkvm1” 
}

 

Test Network Bandwidth

In the final step we will log in to each virtual machine and download psping tool and use it to test network bandwidth.

psping can be downloaded from http://technet.microsoft.com/en-us/sysinternals

It is one of the easiest way to test network bandwidth, latency etc.

On VM anetworkvm1 that was created in ANetwork I run the following command. It opens the firewall ports for the duration of the test and is the server listening on the specified port. Here 172.16.100.4 is the internal IP address of this VM.

On VM bnetworkvm1 that was created in BNetwork I ran the following command. Here we are running a client that will end out 100K requests to the server 10000 times.

001
002
003
004
005
006
007
008
009
010
011
012
C:\pstools>psping -b -l 100k -n 10000 172.16.100.4:5000

PsPing v2.01  PsPing  ping, latency, bandwidth measurement utility
Copyright (C) 2012-2014 Mark Russinovich
Sysinternals  www.sysinternals.com

TCP bandwidth test connecting to 172.16.100.4:5000: Connected
10005 iterations (5 warmup) sending 102400 bytes TCP bandwidth test: 100%

TCP sender bandwidth statistics:
  Sent = 10000, Size = 102400, Total Bytes: 1024307200,
  Minimum = 22.32 MB/s, Maximum = 33.11 MB/s, Average = 29.51 MB/s

 

I ran these tests about 10 times and I was getting similar bandwidths. These results are in MB/s so I was seeing bandwidth close to 236 Mbps. This is higher than 200 Mbps that was mentioned in the specification of High Performance network gateway. Your actual results may vary.

Summary

In this blog post I hoped to demonstrate:

  1. How to provision virtual networks with PowerShell.
  2. How to create recently released high performance network gateway with PowerShell.
  3. How to connect two virtual networks with PowerShell
  4. How to create virtual machines with PowerShell
  5. How much bandwidth you can expect.
  6. You can automate most aspects of virtual network provisioning. I wanted to add a Point 2 site network but I was unable to do so because there is no PowerShell cmdlet that allows me to upload a client certificate. You can do this by invoking the Rest API.
Posted in Azure, DevOps, Virtual Machines, Virtual Networks | Tagged | Leave a comment

DNS issue while setting Active Directory in Azure Virtual Network

I was recently setting up active directory in an Azure Virtual Network and I ran into an issue related to DNS.

MSDN has the following documentation about setting up Active Directory in Azure Virtual Network.

Guidelines for deploying Active Directory on Azure Virtual Machines

http://msdn.microsoft.com/en-us/library/azure/jj156090.aspx#BKMK_HybidExt

Step by step instructions are provided in this article below:

Install a new Active Directory Forest on  an Azure Virtual Network

http://azure.microsoft.com/en-us/documentation/articles/active-directory-new-forest-virtual-machine/

I followed the instructions in this second article carefully. After setting up DNS I added my first virtual machine which was domain joined. My virtual machine was unable to access the internet. I tried search for a solution to this issue and found the following blog post:

http://windowsitpro.com/windows-azure/solve-dc-name-resolution-problems-azure

When I checked my DNS forwarder setting I did have a forwarder listed. I removed this forwarder as suggested the blog post shown above. My internet access started working. However it was not reliable. There were times I was unable to access sites like microsoft.com.

I was fortunate to get help from somebody from Azure DNS team. Here is what they told me.

DCPROMO looks up the root hints by querying the upstream DNS server.  The upstream DNS server is returning all the NS records but only as many A/AAAA records as it can fit into the UDP DNS response.  That seems valid because they’re in the “Additional Section”.  For some reason DCPROMO only looks up the A/AAAA records directly if there are none provided in the response, when some are in the response it uses those and only those.  So DCPROMO picks up a subset of root hints, in my case it did not pick complete set of root hints. This is what was causing reliability issues in connecting to the internet.

Fix for this issue was simple.

Resolution

What I had do was to set the DNS IP on the VNET to 127.0.0.1 before creating the Active Directory and then setting it to the Domain Controller after it’s setup.  This had two effects:

1) it prevents the ADDS from creating a DNS forward rule as we shouldn’t memorize DNS IPs in the guest OS and

2) it forces ADDS to use the default set of root hints and look their IPs up directly.

Following this suggestion helped me get complete set of root hints and my internet access started working reliably.

If you are following these steps below to install a new AD Forest 

http://azure.microsoft.com/en-us/documentation/articles/active-directory-new-forest-virtual-machine/

you should insert the steps listed in the “Resolution” section above after Step 2 and before Step 3(section where you install active directory).

MSDN article will be updated in near future but I thought I will share this information with the Azure community.

Posted in Azure, IAAS, Virtual Machines, Windows | 2 Comments