Last Friday 11th of September, Netapp has announced the availability of Synergy version 6.2.1 General Availability.
Top features and additions:
Netapp Synergy can be access via the following URL https://synergy.netapp.com
On September 3rd, 2015, Netapp announced the General Availability of Clustered Data Ontap 8.3.1
It is a minor release in the 8.3 family and brings with it a list of new and changed features.
A complete list of these features can be found in the Release Notes:
Netapp Clustered Data Ontap 8.3.1 Release Notes
If you are thinking to upgrade your system remember to always generate an Upgrade Plan from Upgrade Advisor (which is available via My AutoSupport)
Unified Manager 6.3RC1, OnCommand Performance Manager 2.0RC1 and Workflow Automation 3.1RC1 are all compatible with Clustered Data Ontap 8.3.1 GA
The default caching action for volumes on a flash pool aggregate is to cache reads and writes, however you may have a use case to disable caching on a per volume basis.
To determine the caching policy of a volume, we have to jump into the node shell and set the diag privilege.
CLUSTER::> node run NODE1
NODE1> priv set diag
NODE1*> priority hybrid-cache show volume1
Volume: volume1
Status: enabled
Read Cache Policy: random-read
Write Cache Policy: random-write
This is the default Read Cache and Write Cache policy setting, to cache random-reads and random-writes.
Valid settings are:
Not sure what the Read and Write Policies mean ? You can check out the descriptions in the links below:
Read Cache Policy: http://community.netapp.com/t5/Technology/Flash-Pool-Read-Policies/ba-p/83308
Write Cache Policy: http://community.netapp.com/t5/Technology/Flash-Pool-Write-Policies/ba-p/83405
To disable read and write cache on our volume in the example above we type:
NODE1*> priority hybrid-cache set volume1 read-cache=none write-cache=none
Netapp SnapCenter is the new application consistent backup, verification, cloning and recovery too that unlike previous SnapManagers, allows you to perform all backup and recovery operations from a single graphical interface.
Within a SnapCenter implementation you would find the SnapCenter Server and various SnapCenter plug-ins which support a variety of applications, databases and hypervisors.
SnapCenter 1.0 includes the Microsoft SQL Plug-In and is considered the next generation of Snapmanager for SQL.
SnapCenter Plug-In for Windows is also the next generation of SnapDrive for Windows.
SnapCenter Plug-In for Microsoft SQL is only supported on Netapp Clustered Data Ontap.
SnapCenter Server is available for download via the following URL: http://mysupport.netapp.com/NOW/download/software/snapcenter/1.0/
SnapCenter Plug-In for Microsoft SQL is available for download via the following URL: http://mysupport.netapp.com/NOW/download/software/snapcenter_sql/1.0/
Currently as of this post there is no available Microsoft Exchange plug-in for SnapCenter
There is a nasty bug within certain versions of Netapp Data Ontap where if you have fpolicy enabled on any vserver within your cluster, the node can crash, including multiple HA pairs at the same time.
Netapp’s Bug ID is 800390 and it explains the following:
When performing certain SAN SCSI operations (Compare and Write, Write Same), a system outage might occur. This is most often seen with CIFS I/O with Vserver fpolicy enabled. However, other workloads might encounter this problem.
The Page Fault Panic string looks like this:
page fault (supervisor read, page not present) on VA 0 cs:rip = 0x20:0xffffffff84963aea rflags = 0x10202
Copying this page fault string into the Netapp Data Ontap Panic Message Analyzer points you directly to Bug ID 800390
The versions of Data Ontap that this has been fixed in are:
Data ONTAP 8.2.1P1 (First Fixed) – Fixed
Data ONTAP 8.2.3 (GA) – Fixed
Data ONTAP 8.3 (GA) – Fixed
Data ONTAP 8.3.1 (GA) – Fixed
However always log a support ticket with Netapp and get their advice on which is the best Data Ontap version to upgrade to.
The workaround until you get time to upgrade your nodes, is to disable fpolicy within the vserver, or even better record your settings and delete the policy.
A customer wanted to know if it was possible to use Veeam Replication between 2 different versions of VMware vSphere. The version being vSphere 5.5 and vSphere 6.
The answer is yes as long as the Virtual Machine Hardware is running at the highest level or lower than the oldest vSphere version.
For example if you would like to replicate between vSphere 5.5 which has a Virtual Machine level of 10 and vSphere 6 which has a virtual machine level of 11, then virtual machine level 10 will be the highest you will be able to use between the 2 hosts. If you attempt to use Virtual Machine Hardware 11 on vSphere 6 and then try to replicate this virtual machine to a vSphere 5.5 environment, it will fail upon powering on.
Here is a link to the VMware site which illustrates the Virtual Machine Hardware levels that are compatible between different versions of vSphere:
Here are a few photos from the VeeamOn event:
VeeamOn 2015 Welcome Party
VeeamOn 2015 Key Note
VeeamOn 2015 Netapp
VeeamOn 2015 VMware
VeeamOn 2015 Microsoft
VeeamOn 2015 Cisco
Veeam Backup and Replication setup with application aware processing requires a user account that has admin access to the virtual machines you wish to backup.
The user account must be in the format domainuser and not user@domain.local. If your user account is in the format user@domain.local you will be receiving this error:
Failed to prepare guest for hot backup. Details: Failed to check whether snapshot is in progress (network mode). RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine: [192.168.1.116]. RPC error:Access is denied. Code: 5
Failed to index guest file system. Veeam Guest Agent is not started
Changing the user account from user@domain.local to domainuser resolves this issue
It is also important to disable Windows UAC on each virtual machine that will be backed up with application aware processing or you will receive an Access Denied error within the backup job logs.
To disable Windows UAC follow my article:
If you have your vcloud director cells setup to use shared NFS storage and you do not have the correct permissions on the mount point folder, you will run into this error:
Error starting application: Transfer spooling area is not writable: /opt/vmware/vcloud-director/data/transfer
You can see this error message if you tail the cell.log during the startup of vcloud director services (#tail -f cell.log)
The location for cell.log is: /opt/vmware/vcloud-director/logs
Having a look at the owner and permissions of the transfer folder we can see that there is only d and the owner is root
Firstly we lets change the owner of the transfer folder from root to vcloud, we can do this by typing:
# chown -R vcloud:vcloud /opt/vmware/vcloud-director/data/transfer
If we examine the folder again we can see that the owner has changed from root to vcloud
Next, we will change the permissions of the folder so that vcloud can have full access to it. To change permissions we use the command:
# chmod -R 775 /opt/vmware/vcloud-director/data/transfer
Examining the transfer folder, we can see that this folder has full access applied to it.
Now lets restart our vcloud director services by typing: # service vmware-vcd restart
Once the services have started up again, let’s tail the cell.log file (# tail -f cell.log). We can now see that vcloud director has full access to the NFS directory via the cell.log entry:
Successfully verified transfer spooling area: /opt/vmware/vcloud-director/data/transfer
Netapp has now released Data Ontap 8.2.4 as GA. This version of Data Ontap is supported on 7-mode as well as C-mode
To check if your system is compatible with this release be sure to check out Netapp’s Hardware Universe
For a complete list of features within the 8.2 family check out the following release notes:
Clustered Data Ontap 8.2.4 Release Notes
Data Ontap 8.2.4 7-Mode Release Notes
Last week an issue was discovered within VMware ESXi 6 where by Change Block Tracking (CBT) can return incorrect changed blocks . This means you will end up with inconsistent incremental backups.
VMware have released a KB article (KB2136854) which illustrates the problem and further goes on to say that there is no current patch or fix, however there are some work arounds:
To me none of these work arounds are really an option for many organizations
The team over at Veeam have been testing the CBT issue with ESXi 6 and have come up with a fourth option which is to disable Change Block Tracking within the advanced settings of the backup job (full, incremental backups and replica jobs). Veeam go onto explain that disabling CBT in the backup job is essential or your backups could contain corruption.
The official Veeam forum thread for this issue can be found by clicking here – There’s a new CBT bug in ESXi 6
Today Netapp announced the release of Clustered Data Ontap 8.3.2 RC1
There have been a few enhancements and bug fixes included in this release, such as:
For a full list of 8.3.x features you can visit the release notes here:
https://mysupport.netapp.com/ecm/ecm_get_file/ECMLP2348067
For a full list of bug fixes between versions, be sure to check out Release Bug Comparison:
http://mysupport.netapp.com/NOW/cgi-bin/relcmp.on?
If planning any upgrade on a Netapp system, be sure to utilize the Upgrade Advisor within My Autosupport:
http://mysupport.netapp.com/myautosupport/home.html
Last week we came across an issue where Veeam Backup and Replication Update 3 was unable to find Netapp snapmirror or snapvault relationships for the specific job that was running. The specific warning within the job window was:
cannot find any relevant Netapp Snapvault and Snapmirror Relationships
The virtual machines for this backup job are located on an NFS volume named customer_os_data
Looking through the Veeam backup logs (C: ProgramData Veeam Backup) we noticed the following entries:
Veeam would go through and list out all the snapmirror and snapvault relationships and then look for a match on the volume name, customer_os_data. However no match was found. The backup log (C:ProgramDataVeeamBackup) produced this log entry against the specific snapvault relationship for this volume:
[20.11.2015 15:04:28] <01> Info [NetApp] Unable to find any LUN or NFS export for SnapProtection relationsip: ‘SouceLocation: ‘Vserver:customer_os_data’, SourceVolume: ‘customer_os_data’, SourceVserver: ‘Vserver’, SourceCluster: ‘CLUSTER1’, DestinationLocation: ‘VserverDR:customer_os_data_sv’, DestinationVolume: ‘customer_os_data_sv’, DestinationVServer: ‘VserverDR’, DestinationCluster: ‘CLUSTER2’RelationshipType: ‘Vault”
Investigating a little further and comparing between a working backup job and this job, we found that Veeam was actually looking for a volume with the path /vol/customer_os/_data <—- Notice the / inbetween os and _data:
[20.11.2015 15:04:26] <01> Info Path: ‘/vol/customer_os/_data‘
We logged a support ticket with Veeam and they were able to identify this as Bug 52968.
This required us to patch 2 files:
The patch is not readily available and is only available via a support ticket with Veeam.
If you run into the same issue you can log a support ticket to request the patch via the customer support portal: https://www.veeam.com/support.html
What is an RCF file ? An RCF file stands for Reference Configuration File and contains a basic certified configuration for the Netapp CN1610 intercluster switches
Currently the latest version of RCF file for Netapp CN1610 intercluster switches is 1.1, however we had 2 switches that were on version 1.0 and in need of an upgrade.
The process is quiet simple:
The final step is to apply the newly uploaded script file.
At this point, take a backup of your configuration. When you apply the script configuration file, it does not blow away your current running-config file, it actually appends changes to your current config.
To apply the newly uploaded script file:
When converting a system from 7-mode to Clustered Data Ontap, there are a few steps that need to be done. You can find the process in my article:
Netapp Data Ontap 7-Mode to Clustered Ontap Steps
I came across a system which was running Data Ontap 8.1.4P3 as 7-mode that had been decommissioned, there was a requirement to take this system and convert it to Clustered Data Ontap 8.2.3P6.
For a new system I enter into the boot menu – select install new data ontap image and go through the process of installing the 8.2.3P6 image. Upon rebooting the system, I then break the boot sequence so that I end up at the loader prompt. I enter all the variables to convert the system to Clustered Data Ontap as per my article above followed by the boot_ontap command. I press ctrl-c to enter into the boot menu and select maintenance mode, however when I press y to continue to boot the system, it throws up this error:
Warning: Data ONTAP has detected an attempt to switch the value of
bootarg.init.boot_clustered. Usage of this boot argument requires Technical
Support assistance. If this was an accidental change, ensure
bootarg.init.boot_clustered matches the root aggregate of the node.
So what has happened ? The system does not like that I don’t have any assigned disks to the controller.
What I had to do was boot back to the loader prompt change the environment variable bootarg.init.boot_clustered to false so that it would boot as 7-mode. Enter into the boot menu, select maintenance mode and assign disks to the controller. Once I did that, I booted back to the loader prompt, change the variable bootarg.init.boot_clustered back to true, entered back into the boot menu and selecteded option 4 Clean configuration and initialize all disks
It is always best to log a Netapp support ticket if you are not sure, unless you are 100% confident in making these types of changes and understand the consequences i.e. that all data will be destroyed upon initializing all disks
VMware have release a patch ESXi600-201511001 that resolves the vSphere 6 Change Block Tracking (CBT) issue which I wrote about in an earlier blog post
The Patch willl resolve the issue on ESXi 6 Update 1a
Reading a little more into this patch on the Veeam Community Forums, Gostev goes into explain that applying the patch is simply not enough. In every instance a CBT reset is mandatory. The reason for this is that existing CBT maps may already contain issues which result in inconsistent backups.
Veeam has posted the following article (KB1113) which lists the steps involved in performing a CBT reset
The process is quite simple however there are a few optional parameters you will want to use to ensure your disk is aligned correctly
First up add a new disk to your Linux virtual machine via editing the config of the virtual machine.
Once the disk is added we’ll jump into the CLI of the Linux virtual machine either via console or SSH
Find the disk that as been added to the Linux virtual machine, in my example it is /dev/sdb
fdisk /dev/sdb – Select the following options:
(n) creates a new partition
(p) creates a primary partition
(1) creates partition 1
accept the defaults to create the partition
still within fdisk select the following options:
(x) select expert mode
(b) to change the beginning sector
(1) select partition 1
(64) change the beginning sector to 64
(w) write changes
You can now exit fdisk
mkdir /datadrive (Creates a folder where we will be mounting our new drive)
We’ll now edit the /etc/fstab file to make the mount persistent across reboots
vi /etc/fstab
Enter a new line with the following details:
/dev/sdb1 /datadrive ext3 defaults 1 3
save the file and exit
mkfs -t ext3 -b 4096 /dev/sdb1
A new Day 0 Linux Grub Vulnerability has been identified which allows an attacker to bypass security and potentially take control of the system.
The attacker can by pass security by simply press backspace 28 times when grub asks you for your username, if the machine reboots or you get placed into the rescue shell, your system is affected.
Thanks to the Veeam Community Forum Digest for alerting us
For full details on the vulnerability you can visit this site: http://hmarco.org/bugs/CVE-2015-8370-Grub2-authentication-bypass.html
CVE ID: 2015-8370 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-8370
A quick post to wish everyone a Happy New Year 2016, hope you all had a great Christmas.
You have probably noticed by now the new look for the website. I have installed it today and have gone through and checked as much as I can. If you notice any components or parts not working correctly, it would be great if you could let me know.
All the best for 2016, I look forward to bringing you more fresh new content
Here is a quick post showing you how you can enable Remote Desktop via Windows Powershell
Below is a screen shot of the commands
