CoNetrix

Blog: backup

Many organizations are adopting Microsoft 365 (formerly Office 365) and businesses nationwide are seeing the benefits of improved productivity through its email and collaboration solution. Organizations of all sizes can benefit from a seamless user experience between mobile and on-premise environments.

While Microsoft 365 offers great flexibility, it mostly focuses on infrastructure management rather than data management. Meaning: You are responsible for your data.

Some businesses who have migrated their workloads to Microsoft 365 do not realize that the same reasons they had for backing up and protecting that data on-premises applies even in the cloud.

If you are still considering Microsoft 365 for office productivity and collaboration, this article may be for you: Microsoft 365: Is it the right choice for your business?

Without proper backup and recovery, your data is at risk, because Microsoft isn't providing complete protection. It's important to create a backup and recovery strategy to ensure you avoid permanently losing your critical data.

It's important to understand the difference in responsibilities of Microsoft and Microsoft 365 user organizations. Microsoft hosts the infrastructure, but you are responsible for your data.

What is Microsoft's Responsibility?

Cloud Infrastructure Uptime — Microsoft focuses on the infrastructure management rather than data management. By focusing on infrastructure, Microsoft ensures its cloud service is online and operational. Guaranteed uptime is based on your agreement level and outlined in the availability SLA (Service Level Agreement).

Basic Data Replication — Microsoft provides basic data replication with datacenter-to-datacenter geo redundancy, and limited retention for short-time data recovery.

Data Processing Compliance — Compliance and controls for data processing are limited to the processor, not the data itself. Microsoft ensures data privacy, regulatory controls, and industry certifications for compliance are in place and maintained for the infrastructure of its cloud service.

Physical Infrastructure Security — Security functions for Microsoft 365 are limited to physical infrastructure, not data. It includes app-level security, logical security, and access controls for users and administrators.

What is the Customer's Responsibility?

Business Data in Microsoft 365 — The customer is the owner of the data that resides in the Microsoft 365 data centers. As the owner, the customer controls the data and who can access the data. All responsibility of the data is on the user to ensure data security, privacy, and retention.

Enterprise-grade Backup and Long-Term Data Retention — Implementing an enterprise-grade backup solution for Microsoft 365 can give businesses confidence to recover from security breaches, compliance exposure, and data loss. With enterprise-grade backup, a copy of the data is stored outside the environment. In the event of an incident, it provides granular and point-in-time recovery options.

Data Owner Compliance — As the data owner, the customer has the ultimate responsibility of data for internal legal and compliance teams. The customer answers to the demands from corporate and industry regulations.

Security Functions to Protect Data — Protection of data is the responsibility of the user, not Microsoft. Security controls must be implemented to protect the data from internal threats, such as accidental deletion, insider threat, and disgruntled employees, and external threats, such as malware, ransomware, and rogue applications.

What happens when Microsoft 365 is used without backup?

Microsoft only provides basic and limited retention. If you don't implement a backup strategy outside of Microsoft's native capabilities, you are opening up your business for unnecessary risk. Lack of a Microsoft 365 backup plan is a risky data strategy.

Without proper backup and recovery, your organization can expose itself to the following risks:

  • Data loss from accidental deletions
  • Ransomware attacks and security breaches
  • Insufficient retention time for regulatory compliance policies
  • Lack of data control due to potential SaaS lock-in

Organizations investing in productivity and collaboration tools should also consider their backup and retention needs as a factor in efficiency and productivity. Considering a third-party backup solution is critical for data loss avoidance.

What is the best strategy for Microsoft 365 backup?

Your data is your business. By taking a data-driven approach to your backup strategy, you recognize the critical importance of your data for your business stability.

Make Microsoft 365 Backup a Key Priority

Backup for cloud services (SaaS), such as Microsoft 365, is imperative for security and data control. Full oversight and control of data is a boardroom priority. Without backup, organizations do not have an exit strategy or freedom from SaaS lock-in because they are not in complete control of their data. Backup should be part of the conversation when buying SaaS and not an afterthought.

Consider Enterprise-grade Data Protection

When investing in backup solutions, consider integration between the Microsoft 365 environment and your existing data protection environment. Evaluate automation, security, and integration between systems when comparing enterprise-grade data protection and recovery features. Integrating SaaS into enterprise data protection can help unify data management.

What to look for in a Microsoft 365 backup solution

1) Freedom to use existing on-premise capacity for Microsoft 365 backup, or the ability to leverage another cloud for cloud backup.
2) Basic features provided, such as incremental backups, granular recovery, automation, and policy-based retention capabilities.
3) A solution capable of managing and protecting hybrid deployments and the ability to ease the full adoption of SaaS.
4) Integration between Microsoft 365 and the customer's existing data protection environment.
5) Advanced security features such as access control, SaaS usage metrics, and multifactor authentication for additional security.
6) Ability to scale up or down as business and data demand changes and as SaaS is rolled out more widely within the company.


Investing in productivity tools and the corresponding backup is an exciting adventure. When you are ready for a guide, we are here to help. We can advise on and implement a solution that fits your business needs. Contact us today to schedule a consultation.

Financial Institutions, General, Networking, Security and Compliance, Backup, Recovery, Information Security Program, Microsoft, Exchange, Financial Institution, Credit Union, Microsoft Office, Cybersecurity, Microsoft Exchange, Microsoft 365, backup, Exchange Web Services,
 

A Windows 8 machine was being backed up with the Windows 7 backup. The backup completed all the file level backup but it failed backing up the system image. I found various articles indicating that the problem was in creating the shadow copy, and apparently it tries to create the shadow copy on the system partition instead of the larger “C” partition (in this case). In this case, the system partition (partition #1 on the physical disk 0) was 1GB and the C drive (partition #2 on the physical disk 0) is about 450GB. [more]

I used the partition program Mini Partition Home Edition V7.7 (downloaded from http://www.partitionwizard.com/download.html) to resize the C drive smaller, then shift it so the system partition can grow contiguously. I increased the system partition size to 2.5GB. Then, the Windows 7 backup program ran to completion and backed up the system image also.

In using the Mini Partition program, I had to remove all USB drives from the system. If USB drives are found, then the Partition Wizard will error out when it reboots to apply the partition changes. This problem is discussed in the FAQ’s for the Partition Wizard found here: http://www.partitionwizard.com/faq.html

Networking, Backup, Windows 7, Windows 8, backup, Mini Partition,
 

A system was running GUID partition tables (GPT) in place of MBR and UEFI instead of BIOS. After a restor from backup, when trying to enable BitLocker, I got an error saying, “Element not found”. This vague error message did not provide any helpful results on Google, so I tried running BitLocker from the command line. Running the command “manage-bde –on C: -tpmandpin” gave me an error code (0x80070490) to go with the vague message. A Google search for the error code yielded this link to TechNet that says this is a known issue when moving hard drives between systems using the UEFI boot firmware and that running “bcdboot %systemdrive%\Windows” command will fix it. The command did not fix the problem, but it pointed me in the right direction. Some more searching led me to this link that talks about how to manually delete the “bootmgfw.efi” file in the UEFI boot partition. After deleting the file and then running the “bcdboot” command from the TechNet article, BitLocker encrypted the drive.

Networking, Backup, GPT, UEFI, BitLocker, backup,
 

A monitoring service reported an Asigra DS-System running low on available disk space. Looking at our storage reports, I was unsure as how we had filled up so much space so quickly. The answer comes from the “Trash” that Asigra creates. When Asigra expires backup data either due to the max generations or retention policies being met, it doesn’t immediately remove the data from the DS-System. Instead, this data is moved to the Asigra  equivalent of the Windows Recycle Bin before being purged. By default, a scheduled job runs monthly to remove data from Trash that is older than 30 days.  This means that data could be in there for potentially 2 months before being removed permanently.  You can run a job to purge the data manually or you can alter the schedule to run more frequently if needed.

Networking, Asigra, backup,
 

Over the years, many people have asked me about backup for home machines.  Burning files to DVDs and carry them to a different location is problematic.  It's a lot of trouble to make frequent offsite backups.  I recently did some research and decided on using a program called Duplicati for backup and Amazon S3 (Simple Storage Service) for storage.  I think having the backup program and the storage separate is the best solution.  I can even back up to multiple providers in case one of them just goes away without warning.

Duplicati is free and open source and runs on Windows, Linux and MacOS.  It has a nice GUI interface plus a rich command line.  Duplicati has built-in AES-256 encryption, which means you hold the key and your backups are encrypted before leaving your network.  It creates normal zip files and then encrypts them with AES Crypt, so even if Duplicati breaks, you can still download, decrypt, and unzip your backups using other standard tools. [more]

Duplicati will back up to many different cloud providers (Amazon S3, Rackspace, Google Docs, SkyDrive, Tahoe-LAFS, WEBDAV, FTP, SSH) as well as file based locations.

I chose Amazon S3 for storage because of the history of reliability of Amazon.  The cost if not much either.  You get 5 GB free, and then it’s 12.5¢/GB/month after that.  So you can store 50GB for less than $6/month.  It is even cheaper if you choose the Reduced Redundancy Storage (RRS).

Get Duplicati here http://www.duplicati.com/.

Sign up for Amazon S3 storage here http://aws.amazon.com/.

General, Amazon, backup,
 

The other day I had an issue come up with a customer where VSS (Versative Storage Server) integrated file system backups stopped working for some unknown reason. Usually, a reboot fixes these types of issues, but backups continued to fail after a reboot. I started a support call with the backup vendor and after seeing the error logs, the support tech seemed fairly sure he knew what the problem was. This error is usually caused by a malformed path within the registry. So he had me run the following commands on the server and send him the output. [more]

vssadmin list writers >> c:\writers.txt
vssadmin list providers >> c:\providers.txt
vssadmin list volumes >> c:\volumes.txt

diskshadow /L c:\shadow.txt
list writers detailed

After reviewing the text files created, he found the malformed path:

- File List: Path = c:/windows\hpsum_1327455089, Filespec = hpsumserverw32.exe

To correct the issue, I searched the registry under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\ for “c:/windows\hpsum_1327455089” and corrected the path to “c:\windows\hpsum_1327455089” . After doing this, the backup ran fine. Further research uncovered the root cause. During the last maintenance window, HP System Update Manager was used to update the HP System Management Homepage on these servers. This malformed registry key was created by HPSUM during the upgrade.

Backup, VSS, HP, backup,
 

When setting up VaultLogix online backup, make sure the server is not configured to apply Windows Automatic Update and reboot during the backup window. The problem that occurs is Windows Automatic Update reboots the server in the middle of a backup, VSS shuts down, but can allowed enough time to commit a partial backup which can leave off drives that need backed up. On the next backup, the agent will think there is new data and commenced to reseed those drives on the subsequent backup. Which can cause a problem, if the network has a slow connection that had to be seeded with a mobile vault, because backups will never be able to catch up.

If this occurs you must stop the re-seeding, purge all of the impartial backups since the error occurred. Then resynchronize the Vault Logix DTA file. Then the agent will not try to reseed data that is already backed up and just do the deltas for the selected drives.

Backup, VaultLogix, backup,
 

The CommVault Exchange Mailbox iData agents do not backup mailboxes associated with disabled Windows user accounts. The backup job reports a "success" for the job, but when the details of the backup are explored, the backup set does not contain any data. Additionally, requesting a listing of all failed objects for the backup job results in a "no failures" status. According to CommVault, this behavior is by design as is the "successful" backup status. After all, the job did not technically fail if it is not designed to include mailboxes belonging to disabled user accounts. This is very strange given that, in general, CommVault iData agents have an "inclusive by default" behavior.  This can become a real problem if you try to restore data for a former employee whose Windows user account was disabled when they left the company.  The lesson here is that you should always test your backups. Even if the backup report and all job status notifications indicate you are good....test anyway.

Backup, Exchange, CommVault, backup,
 

One of our customers uses VMware VCB backups integrated with CommVault Simpana. The CommVault job simply calls a pre-backup script to snapshot the VM and copy all the VM files to the VCB proxy, backs up the files from the proxy to the CommVault media server, then a post-backup script commits the snapshot and purges the VM files from the VCB proxy.

Recently, we upgraded this customer from VMware VI3.5 to VMware vSphere v4 Update 2. For most of the VMs that are backed up with VCB, we had no issues at all. The backups ran the weekend following the upgrade with no issues. However, all of the VMs that had been secured with the Windows Security Configuration Wizard would not back up. These VMs are in the DMZ and are locked down very tight because they host externally available web applications. The issue is that each time a backup was initiated from CommVault, the VCB script would return a non-zero error due to a snapshot failure in VMware. VMware’s error was “Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.” This would happen when using VCB scripts, but I could create a snapshot without error from the VI client. [more]

After much research and testing, I determined that the problem was hold-over from the VMTools upgrade. In the new version of VMTools, a new service is installed called VMware Snapshot Provider is installed. This service gets installed when VMTools is upgraded. Its purpose is to help facilitate application consistent snapshots through the VMTools. On the servers that were getting the “quiesced snapshot error”, this service was not present at all, but VMTools had already been updated…very strange. Here is where the Security Configuration Wizard comes in. Part of our lockdown policy is to disable a service called COM+ System Application. This service manages the configuration and tracking of COM+ based components. Apparently, without this service enabled, VMTools upgrade will NOT install the VMware Snapshot Provider service. Without the service, no quiesced snapshots and you get errors when creating snapshots via the VCB integration modules.

So why could I create a snapshot from the Vi client? Well, VMware knows that you are using VCB to create snapshots for the purpose of backup. What good would the backup be if it wasn’t app consistent? The VI client, on the other hand, will first try to create an app consistent snapshot, but if it fails or times out, it will go ahead and create the snapshot “crash consistent” without error. VCB is not as forgiving. If the guest quiesce fails, the snapshot fails…end of story. The solution was to uninstall the VMTools, reboot, temporarily enable and start the COM+ System Application service, install VMTools, then disable the COM+ System Application service. After I did that, backups have been running fine since.

Networking, vSphere, VMTools, CommVault, VMware, backup,
 

I learned the reason that VMware suggests having service consoles for ESX hosts on at least two distinct networks last week. I was troubleshooting intermittent backup issues with Veeam on a customer network and couldn’t really find any pattern to the failures. Two or three backups in a row would run successfully, then 5 in row might fail. The behavior was very random. However, the failures were always on Virtual Machines associated with a specific ESX host. At first I thought the host was healthy, but after watching the VI client for an extended period of time, I noticed that the ESX host would drop offline (showing disconnected in the VI client) and then come back online again.  This indicated the problem wasn’t just affecting the management/backup server. [more]

In order to level set my troubleshooting efforts, I decided to reboot this ESX host. However, after the reboot, I could not connect to it with the VI client. I could ping the IP assigned to the service console, but couldn’t SSH or connect via the VI client. I logged in via iLO and found that an ifconfig at the command line returned IP = 0.0.0.0…..interesting. So what is responding to my pings. I checked the arp cache on one of the switches and found that a thin client had been plugged in that had the same IP as my LAN service console. What is really odd is the MAC address for the thin client was all zeros AND the IP I was using for the LAN service console is not even available to be distributed by DHCP. I was not able to connect to the thin client to see how it was configured, but I was able to connect to ESX host via a second service console port that I placed on the iSCSI network. The management/backup server has a connection to the iSCSI network to do backups to disk so I was able to change the LAN-facing service console IP to another IP and everything started working fine. The backup issue was obviously being caused by changes in the arp entries on the backup server between the thin client and the ESX host. So, be aware that at boot-time, if ESX determines that the IP it is using for a service console is already in use, it just rips it out of the configuration and continues to boot with NO WARNINGS or ERRORS on the console.

Networking, Veeam, iSCSI, iLO, VMware, ESX, backup,