A new Microsoft tutorial is now available on how to install Microsoft System Center 2012 R2 Virtual Machine Manager (SCVMM)
The tutorial runs through setting up the database, installing Virtual Machine Manager and add in your first Hyper-V host
Microsoft System Center 2012 R2 Virtual Machine Manager Install
This week i’ve been troubleshooting a very bizarre Wireless Station Deauth issue on an Aruba 6000 Controller along with some HP and Dell Laptops running Windows 8 and Windows 8.1
The issue came about after upgrading the Laptops from Windows 7 to Windows 8.1 over the Christmas break. Since these laptops are used in a school, many of them get powered on at once. What was identified was that some of the laptops were connecting and other weren’t. The ones that weren’t connecting were in a state of searching for wireless networks.
This was total intermittent, meaning that the laptop that didn’t work 5 minutes ago, could suddenly associate and then jump onto the network and work perfectly. So with any intermittent issue this was going to take time to troubleshoot.
Firstly the way that a Laptop successfully authenticates via 802.1x onto the network is via the following:
If you are not sure on how 802.1x works between Supplicant, Authenticator and Authentication Server you can click here for an excellent explanation on Wikipedia
The first step was to install the latest Intel Windows 8.1 Wireless card drivers for the adapters within the laptops. These consisted of 2 different types of Intel cards:
Both cards now had the latest drivers and still the same problem persisted. I played around with changing the following settings within the adapter to see if it made any difference:
Still the same problem. After searching on the Intel Communities for known problems with the above cards and Windows 8.1, I found that plenty of people are experiencing very similar issues. They explain that everything was working fine and then as soon as upgrading to Windows 8.1, they experienced very weird connectivity problems. A few Intel mods replied saying that the issue had been fixed in newer drivers however from the responses that followed, it appears this was not the case.
I then installed Microsoft Network Monitor 3.4 on the laptops so I could see from the Laptops point of view what it was trying to do. You can use any packet tracing software for this. Another good alternative is Wireshark
It was the time to turn on debugging in the Aruba 6000 wireless switch so I could see what was going on in this area. The commands I used to turn on the debugging are:
Sometimes during my tests I would also clear the MAC address from the user-table. The command I used to do this was:
If you need to look into the authentication packets being sent and receive via the Switch (Authenticator) you can use this command:
I had the 2 laptops with me, HP and Dell, as I powered off the wireless adapter on both computers and then powered on the HP one so that I can see the debug logs in action within the wireless switch. I issued the following command on the Aruba 6000 wireless switch:
This is what I noticed:
This is all good up to here but then the problem packet comes where the wireless station wants to deauth for some reason and I see 1 of the 2 messages below in the debug log:
The first line I cannot find or figure out what Unspecified Failure means, and Reason 255 according to this post on Aruba Airheads, means that the client sent a Deauth but did not give a reason.
From the Wireless Station point of view with the Packet Capture I see the following:
All this diagnostics points straight back to the client. Intel have Windows 8.1 drivers for the integrated cards so it looks like the operating system is supported.
My final step was to try and find the oldest driver that would work on Windows 8.1 and slowly step up from that, testing each time.
Our tests were pretty simple, turn the wireless switch off, wait, and on again on the laptop. Maybe after 1 time I would see the problem, then again maybe after 20 times I would see that problem.
In the end, we ended up running a Windows 7 driver that was able to install into Windows 8.1 and after a full afternoon of testing and roaming around to different IP’s we did not see the error.
A new VMware tutorial is now available featuring vCenter Operations Manager 5.8. In this tutorial I walk you through how to install and get the vCenter Operations Manager up and running ready to begin monitoring and reporting on your environment.
This tutorial is a video walk through tutorial
I’ve just composed a quick video that I think you guys will find handy, especially if you are always scratching your head around finding the BASE DN in your Microsoft Active Directory Domain.
This tutorial will walk you through using the DSQUERY tool included with Microsoft Windows, that allows you to search and query your Active Directory Domain via the command prompt.
The output is the location of the object and also the BASE DN. In my tutorial I give you examples of searching on: Users, Groups, Computers and Organisational Units.
This week I was working on a customer where their website was being bombarded by random hits coming from a referer out there on the internet.
Enabling the IIS 7 logs, we could see that the referer was www.badreferer.com (URL changed for security reasons) and the hits were in the thousands per hour which actually drove the web servers to 100% CPU, making the main site barely usable.
So how do we block a referer with Microsoft IIS7 ?
Microsoft URL Rewrite is a free add-on to Microsoft IIS7 that enables us to write specific rules to allow or block certain traffic to our website. Similar to a .htaccess file in Linux.
You can download URL Rewrite from here
1. Install Microsoft URL Rewrite. Once it is installed you will see a URL Rewrite button in the right hand side pane
2. Double click on URL Rewrite. On the top right hand corner click on Add Rule(s)…
3. Select Request Blocking and click ok
4. Under Block access based on – select Referer. Under Block request that – select Matches the Pattern. Under Pattern (Referer) – select *badreferer.com. Under Using select wildcards and under how to block, select Abort Request.
5. The URL Rewrite rule is now in place and blocking.
Here is a sample config of what I used to block the HTTP referer from a Cisco ASA
class-map URL-BLOCK-MAP
match access-list web_servers
access-list web_servers extended permit tcp any object WEB-SERVER-1 eq www
access-list web_servers extended permit tcp any object WEB-SERVER-2 eq www
regex BADREFERER “[Bb][Aa][Dd][Rr][Ee][Ff][Ee][Rr][Ee][Rr].[Cc][Oo][Mm]”
policy-map type inspect http INSPECT-HTTP
parameters
match request header referer regex BADREFERER
drop-connection log
policy-map global_policy
class inspection_default
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
inspect icmp
inspect pptp
inspect ip-options
class choice-filter
inspect http INSPECT-HTTP
class global-class
class class-default
user-statistics accounting
service-policy global_policy global
To check that the Cisco ASA is matching the rregular expression for badreferer.com and dropping you can type:
show service-policy global inspect http
Global policy:
Service-policy: global_policy
Class-map: URL-BLOCK-MAP
Inspect: http INSPECT-HTTP, packet 21981960, drop 13465, reset-drop 26
protocol violations
packet 567
match request header referer regex BADREFERER
drop-connection log, packet 13439
If you have any technical questions about this tutorial or any other tutorials on this site, please open a new thread in the forums and the community will be able to help you out.
Disclaimer:
All the tutorials included on this site are performed in a lab environment to simulate a real world production scenario. As everything is done to provide the most accurate steps to date, we take no responsibility if you implement any of these steps in a production environment.
A lot of people have asked me how to do certain tasks within Cisco UCS, however they tell me they don’t have a lab system to test on. What you may of already known or not is that Cisco provide a Cisco UCS Simulator for you to configure and test with.
The Cisco UCS Simulator is a free download from cisco.com and is packaged as a VMware Virtual Machine that will run on either VMware Workstation, Player or within ESXi.
In this new Tutorial I walk you through how to setup the Cisco UCS Simulator and connect into the UCS Manager ready to begin your configuration.
http://www.sysadmintutorials.com/tutorials/cisco/cisco-ucs/cisco-ucs-manager-simulator/
There is an update to the Pink Screen of Death (PSOD) that I have been experiencing on some hosts.
I have had a ticket logged with VMware since last year on this and have had to go through multiple debug builds of ESXi to capture the necessary logs for VMware engineering to search through and find the issue.
If you remember back in this post earlier on last year http://www.sysadmintutorials.com/vmware-esxi-5-1-psod-pf-exception-14-crash/ I mentioned that booting the blades from FCoE caused the issue. Well after changing to booting off local disk the issue did die down but it did reappear again.
Some users have mentioned that ensuring all Virtual Machines are running VMXNET3 Network Adapters fixed a particular Pf Exeception 14 crash for them, however this was not the case for us.
To cut a long story short, VMware got back to us today with a bug in the ESXi code, when running a VDS switch with network throttling turned on. If network throttling is turned on then your host could potentially crash at some point.
VMware will be releasing a patch, however the patch will not make the next release of patches but the patch after that.
An indicative timeframe for the patches to be released are:
ESXi 5.1 Patch 04 – Expected end of April 2014.
ESXi 5.5 Patch 02 – Expected end of June 2014.
The work around for others experiencing the same issue is to disable network throttling within the VDS until the patch is released.
With vSphere 5.5 and the ability to upgrade your virtual machine hardware to version 10, you may of seen this error before if you have tried to use the vSphere Client to edit the settings of a Virtual Machine with Hardware Version 10:
You cannot use the vSphere client to edit the settings of virtual machines of version 10 or higher
VM Hardware Version 10
The best way to change settings is to use VMware PowerCLI
For example say you have lost connection to a virtual machine called VMSQL1 because you have implemented a vSphere Distributed Switch, but something screwed up and the VM didn’t change it’s network adapter over, though the host had moved it’s physical NIC and VMK for management over.
PowerCLI C:> get-vm vmsql1 | Get-NetworkAdapter
Name Type NetworkName MacAddress WakeOnLan Enabled
—- —- ———– ———- ———
Network adapter 2 Vmxnet3 VM Network 00:50:56:98:7d:af True
PowerCLI C:> get-vm vmsql1 | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName HomeLabInternal
Confirm
Are you sure you want to perform this action?
Performing operation “Setting NetworkName: HomeLabInternal” on Target “Network adapter 2″.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “Y”): y
Name Type NetworkName MacAddress WakeOnLan Enabled
—- —- ———– ———- ———
Network adapter 2 Vmxnet3 HomeLabInternal 00:50:56:98:7d:af True
Some of you guys might of come across this tool before to add into your documentation, but for those that don’t know about this neat little piece of software, it is a great tool to add into your tool kit.
The tool connects to Microsoft Active Directory and draws visio diagrams with information on the following:
In the video below I run through how to use the Microsoft Active Directory Topology Diagrammer to connect to my Active Directory environment and draw some nice visio diagrams
A quick post to let you know that I’ve just completed a VMware vShield Manager install tutorial. I will be working in April to bring some more tutorials on vShield and also vCloud Director. VMware vShield Install – http://www.sysadmintutorials.com/tutorials/vmware-vsphere-5-x/vmware-vshield-install/
I’ve created this script to get around the tedious task of reallocating each volume on a Netapp 7-mode system once you have added disks or disk shelves to an aggregate.
Currently as of this date there is no Powershell Command to run reallocates on a Clustered Mode system. I have requested Netapp to include this in their next release of the toolkit. When it is available I will update this page with the Clustered Ontap script.
You must have the Netapp DataOntap Powershell Toolkit installed before running the script. You can download the Netapp Powershell Toolkit here:
http://support.netapp.com/NOW/download/tools/powershell_toolkit
The powershell scripts does the following:
If you have multiple aggregates on the controller and you wish to only reallocate volumes on a particular aggregate you can do this by modifying this line:
$volumes = get-navol
and append the following below, changing AGGREGATENAME for the name of the aggregate you wish to reallocate volumes
$volumes = get-navol | where {$_.Aggregate -eq “AGGREGATENAME”}
There is currently an issue if you are running VMware ESXi 5.5 Update 1 hosts and you are using NFS for your storage whereby you will see multiple and intermittent All Paths Down or APD logs in the events tab for the host.
This causes the datastore to pause momentarily until the datastore has exited the All Paths Down state, usually a few seconds.
VMware have confirmed this as a known issue and the work around is to not use ESXi 5.5 Update 1 but use ESXi 5.5 GA
Please see this official VMware KB article
We ran into a nasty bug last week on a production Netapp Clustered Ontap 8.2P3 system where by a storage admin was performing some routine lun maintenance tasks such as offline lun and delete, which caused the system to be unresponsive on one controller for about 2 hours.
After logging a P1 Netapp support case, the engineer identified that this sytem was hitting Bug ID: 712029
The Bug ID identifies an issue when certain lun operations are performed, a system could be unresponsive and stop serving data due to SAN configurations (bcomd) becoming out of sync among nodes.
The link to the article is provided here, however if you do not have a http://support.netapp.com login I have provided the screen shot of the Bug below.
This bug was reportedly fixed in 8.2P6 and 8.2.1, however I have checked the article again tonight (27th April 2014) and they have removed 8.2P6 and state that it is fixed in Data Ontap 8.2.1GA
http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=712029
Netapp Bug 712029
Last April I posted about VMware’s issue with NFS disconnects within ESXi 5.5 Update 1. As I mentioned in that post VMware notified us that there will be an update coming, update 2, that will resolve this issue and that in the meantime you were to step down or downgrade to ESXi 5.5 GA.
However there are a few ESXi advanced settings that you can implement when using Netapp along with VMware that may either eliminate the issue or reduce the likely hood of it occurring so frequently.
If you are running Netapp VSC within vCenter and you have applied the recommended settings to your hosts, after you apply these custom settings you will get a red mark next to NFS settings. Don’t run the recommended settings again for NFS otherwise the custom settings below will revert.
To set advanced options on your ESXi host, select the host within the vsphere client, click configuration tab, then on left hand side – advanced settings
Advanced Options
Disk.QFullSampleSize 64
NFS.MaxVolumes 256
NFS.MaxQueueDepth 64
Net.TcpipHeapSize 32
Net.TcpipHeapMax 128
Net.TcpipEnableABC 0
After making these changes reboot the ESXi host.
[UPDATE - 27-8-2014] VMware have now released a patch that fixes the APD issue with VMware ESXi 5.5 Update 1 which is resolved in ESXi 5.5 Express Patch 04
Here is the link to the patch:
Sometimes when you try to either delete a lun or NFS based datastore from VMware it tells you that the datastore is in use and cannot be removed.
Usually if you right click on the datastore and select browse you can see a some folders belonging to virtual machines in there, which is easy to identify the culprits.
However, what happens when you don’t see any virtual machine folders within the datastore and you still cannot remove it ?
There are 2 ways to identify which vm’s are using the datastore:
Within vCenter or connected directly to the ESXi client you can browse to Datastore and Datastore Clusters – select the datastore that you are trying to remove and then click on the Virtual Machine tab. The virtual machines listed here are currently using the datastore
Via PowerCLI connect to vCenter or directly to the ESXi host and type the following:
PowerCLI C:> get-datastore vmware_datastore1 | get-vm
Replace vmware_datastore1 with the name of your datastore
In my experience i’ve seen the virtual machine having an ISO image mounted from the datastore that has not dismounted correctly even though the vm says it is disconnected.
What you can do is mount another ISO from a different datastore and then dismount it, this will free up the virtual machine from the datastore you wish to delete.
With vSphere 5.5 there is the option to update the virtual machine hardware to version 10. Unfortunately there is no way via the gui to update only to version 9. Why would you want to do this ? Because after upgrading the virtual machine to version 10 you can not longer edit the settings of the virtual machine in the vSphere Client, you can only edit the settings via the web client.
However if you have PowerCLI installed, you can update the virtual machine hardware to what ever level you like as long as it is greater than the level you are currently on. If you upgrade a virtual machine hardware to version 9, you will still be able to edit the settings via the vSphere Client.
The virtual machine must be powered off before changing the hardware level. I also usally take a VMware snapshot until the machine has booted at least once, after which I remove the snapshot.
PowerCLI C:> Set-VM -VM vmsql1 -version v9
The above command will update the VMware virtual machine called vmsql1 to hardware level 9. You can also append -confirm:$false to the command to apply the settings without prompting you.
We had a customer who experienced the purple screen of death on their ESXi server due to a specific bug regarding the use of vSphere Distributed Switch and traffic shaping.
VMware has addressed this bug and there is a patch available which will prevent your ESXi host from crashing.
This patch addresses a few bugs but in particularly this one:
If you enable the traffic shaping policy on a vSwitch portgroup, the ESXi host might intermittently fail with a purple diagnostic screen with a backtrace similar to the following:
2013-01-10T14:41:19.355Z cpu20:14281)0x41229f25b808:[0x418016968ab3]Vmxnet3VMKDevTxComplete@vmkernel#nover+0x1d2 stack: 0x41229f25b838,
2013-01-10T14:41:19.355Z cpu20:14281)0x41229f25b848:[0x418016968d13]Vmxnet3VMKDevTxCompleteCB@vmkernel#nover+0x116 stack: 0x41229f25b800
2013-01-10T14:41:19.356Z cpu20:14281)0x41229f25b8e8:[0x41801693cf40]IOChain_Resume@vmkernel#nover+0x247 stack: 0x41229f25b900, 0x0, 0x0,
2013-01-10T14:41:19.356Z cpu20:14281)0x41229f25b958:[0x41801692b185]Port_IOCompleteList@vmkernel#nover+0x1c4 stack: 0x41229f25b998, 0x14
2013-01-10T14:41:19.357Z cpu20:14281)0x41229f25bb58:[0x418016e3c139]EtherswitchPortDispatch@ # +0x151c stack: 0x412200000018,
2013-01-10T14:41:19.357Z cpu20:14281)0x41229f25bbc8:[0x41801692b337]Port_InputResume@vmkernel#nover+0x146 stack: 0x41229f25bc18, 0x41001
2013-01-10T14:41:19.358Z cpu20:14281)0x41229f25bc18:[0x41801692cab2]Port_Input_Committed@vmkernel#nover+0x29 stack: 0x11221ead0, 0x41220
2013-01-10T14:41:19.359Z cpu20:14281)0x41229f25bc98:[0x41801696b541]Vmxnet3VMKDevTQDoTx@vmkernel#nover+0x2f8 stack: 0x41229f25bd28, 0x14
2013-01-10T14:41:19.359Z cpu20:14281)0x41229f25bce8:[0x41801696c968]Vmxnet3VMKDev_AsyncTx@vmkernel#nover+0xd7 stack: 0x41229f25bd28, 0x4
2013-01-10T14:41:19.360Z cpu20:14281)0x41229f25bd58:[0x4180169518a3]NetWorldletPerVMCB@vmkernel#nover+0xae stack: 0xe9, 0x41229f25bdf0,
2013-01-10T14:41:19.360Z cpu20:14281)0x41229f25bed8:[0x41801690af2b]WorldletProcessQueue@vmkernel#nover+0x486 stack: 0x41229f25bf18, 0xb
2013-01-10T14:41:19.361Z cpu20:14281)0x41229f25bf18:[0x41801690b5a5]WorldletBHHandler@vmkernel#nover+0x60 stack: 0x10041229f25bf48, 0x41
2013-01-10T14:41:19.361Z cpu20:14281)0x41229f25bf98:[0x4180168207fa]BH_Check@vmkernel#nover+0x185 stack: 0x4180169b84f9, 0x4100346ac188,
2013-01-10T14:41:19.362Z cpu20:14281)0x41229f25bfe8:[0x4180168f3763]VMMVMKCall_Call@vmkernel#nover+0x27a stack: 0x0, 0x0, 0x0, 0x0, 0x0
2013-01-10T14:41:19.362Z cpu20:14281)0x4180168c77d8:[0xfffffffffc223a12]__vmk_versionInfo_str@esx#nover+0xe4f024b1 stack: 0x0, 0x0, 0x0,
Here is the link to the patch:
With VMware vSphere 5.5 Update 2 out, you can now edit a virtual machine in the vsphere client that has a hardware level of 10. Previously you could only do this in the vsphere web client. Still, new functionality is only available in the Web Client, but if you need to make some quick changes via the vsphere client you can now do this successfully.
Why do you create load-sharing mirrors on the root volume of a vserver ? For 2 main reasons:
1. to protect the vserver root volume in case of a disaster and the root volume is lost
2. to load-balance client requests
In the case of a vserver root volume disaster, any of the load-sharing mirrored destinations can be promoted to the full read/write root volume
In the case of load-balancing client requests, you need to have a load-sharing mirror setup for each node in your cluster. If the client requests data from a volume on a particular node that does not hold the root volume, the client connection gets re-directed to the root volume of the node where the data resides.
For example:
If I have a 2 node-cluster – Node1 and Node2, Node1 holding the root volume with Node2 holding a load-sharing mirror of the root. I have data on a volume on Node2, without a load-sharing mirror the client will reference the root volume in Node1 in order to access data on Node2.
With a load-sharing mirror setup on Node2 the client can access data on Node2 without having to reference the root volume on Node1.
Scenario – Node1 and Node2. Root volume is in a vserver called Corporate and is named root
CLUSTER::> volume create -vserver corporate -volume root_m01 -aggregate aggr1node1 -size 1GB -type DP
CLUSTER::> volume create -vserver corporate -volume root_m02 -aggregate aggr1node2 -size 1GB -type DP
CLUSTER::> snapmirror create -source-path CLUSTER://corporate/root -destination-path CLUSTER://corporate/root_m01 -type LS
CLUSTER::> snapmirror create -source-path CLUSTER://corporate/root -destination-path CLUSTER://corporate/root_m02 -type LS
CLUSTER::> snapmirror initialize-ls-set -source-path CLUSTER://corporate/root
At this point we want to create a schedule so that the load-sharing mirror updates periodically
CLUSTER::> job schedule interval create -name 20mins -minutes 20mins
Now we will apply the schedule to the snapmirror job
CLUSTER::> snapmirror modify -source-path CLUSTER://corporate/root -destination-path * -schedule 20mins
Let’s say we add 2 more nodes to the cluster to make a 4 node cluster. We would create a volume on each node (steps 1,2), create the snapmirror relationship (steps 3,4) and initialize each relationship separately. So in step 5 we would replace the initialize-ls-set with:
CLUSTER::> snapmirror initialize -source-path CLUSTER://corporate/root -destination-path CLUSTER://corporate/root_m03 -type LS
CLUSTER::> snapmirror initialize -source-path CLUSTER://corporate/root -destination-path CLUSTER://corporate/root_m04 -type LS
and we would apply the job schedule of 20mins to these destinations by re-apply the command:
CLUSTER::> job schedule interval create -name 20mins -minutes 20mins
I have just completed 2 new tutorials with the latest Veeam Backup and Replication version 8. The 2 tutorials concentrate on backing up and restoring virtual machines on a VMware vSphere environment via the Veeam Backup and Replication version 8 console.
In the demonstration we backup a virtual machine which resides on a Netapp NFS datastore using the Veeam Backup and Replication console. The backup takes a consistent backup of the virtual machine and also triggers a Netapp snapshot on the source volume.
In part 2, we delete the source virtual machine and then use Veeam Backup and Replication version 8 to recover the virtual machine from a Netapp snapshot.
