CoNetrix

Blog: Networking

I was configuring a new Windows 10 PC for a customer and logged in under the local administrator account. I tried to open Edge but received a notification that Edge could not be opened by the built-in administrator account.

After some research, I discovered that Microsoft has become distrusting enough of the local administrator account that they prevent it from opening Microsoft Apps. This was actually introduced with Windows 8, but Windows 10 introduced Edge, which is potentially the first widespread use of Microsoft App.

The options to fix this are to either disable UAC, or adjust group policy to allow the local admin account to access Apps. The details of the issue and the options for a fix are included in this link: https://4sysops.com/archives/why-the-built-in-administrator-account-cant-open-edge-and-a-lesson-in-uac/

Networking, Windows 10,
 

At the beginning of 2018, news broke regarding "Meltdown" and "Spectre"; two vulnerabilities that took advantage of speculative execution in Intel CPUs to retrieve sensitive information. This quickly expanded some from the initial report as OS vendors would release patches for their respective systems, but the basic vulnerability remained the same.

Microsoft released an out-of-band patch to mitigate the vulnerability from the software side with a caveat; several antivirus vendors were taking advantage of kernel processing in ways that were not best practice. When the Microsoft patch is installed, the system would get a "blue screen of death" (BSOD) due to the antivirus software.

In response Microsoft implemented a check for a registry key before installing the patch - antivirus vendors would need to add this key to show they were compatible with the release. Vendors that did not add this (despite the compatibility) caused IT administrators to manually add the key in order to continue receiving patches following the January release.

Over the next several months, issues with CPU firmware caused software patches to be re-released, rolled back, and released again across a variety of vendors. Only recently have these firmware releases stabilized enough that software vendors can re-release and support their mitigation patches.

More recently, the March Monthly Rollup for Server 2008 R2 (KB 4088875/4088878) had an issue that affected many virtual servers with static IP addresses. Upon reboot, these servers would "lose" and "rediscover" the NIC, forcing administrators to delete the "disconnected" and hidden NIC driver and reconfigure the new NIC. Around a week after the initial release, Microsoft published workaround instructions for administrators to run some VBScript code, that would clear some registry settings, prior to installing this patch.

A few days later, information came out regarding "Total Meltdown" - a new vulnerability created from the patches of the original Meltdown/Spectre patches - that required an out-of-band kernel update in addition to the buggy March Monthly Rollup for Server 2008 R2.

Finally, a week before the April Patch Tuesday release, Microsoft released a patch that would execute the VBScript via Windows Update, and configured the metadata of the patches so that this patch should install prior to the buggy KB4088875 and the follow-up kernel update (KB4100480). As of the April Patch Tuesday, these patches appear to have been rolled up into the single Monthly Rollup release in order to take care of all the prerequisites automatically.

There are several other examples of patches in the past that require additional manual work following install. A few examples are below:

  • KB2871997 - Released October 2014, requires registry key to force clear leaked logon session credentials
  • KB3159706 - Released May 2016, requires post-installation command line for Server 2012 R2 WSUS to properly decrypt Windows 10 upgrades
  • KB4034879 - Released July 2017, requires registry keys to make LDAP authentication over SSL\TLS more secure

Needless to say, it is prudent that IT administrators remain on top of patching and vulnerabilities reported across your infrastructure. Many of these additional steps can easily slip through the cracks for someone who is blindly approving and installing patches - even though that appears to be the recommended best practice for Windows 10 / Windows Server 2016 going forward.

Networking, windows update,
 

I installed a new Synology NAS with two folder shares with encryption enabled.  The Synology is configured to check for updates, but not install them.  This is so we can control that this is performed when there is no backup activity going on. After installing the updates the NAS is automatically rebooted.

After the reboot when I went to make sure I could still access the file shares on the device, I discovered they appeared to be gone.  In a panic, I checked the disk volumes within the Synology Web UI and saw they still had quite a bit of used disk space, so the data was still there.

As it turns out, the shared folders need to be mounted after a reboot.  Select each shared folder and under the encryption dropdown will be an option to mount.  This will bring up a prompt for the encryption key password which you must provide.  After mounting the folders, I could access the file shares again over CIFS.

 

Networking, Synology,
 

Occasionally, I have the need to open a Visio diagram but don't have a need to create or modify them. So, the Visio viewer seemed to be an ideal option. However, after installing the viewer (I tried this with both 32-bit and 64-bit versions), I was still unable to open a Visio file.

The best I could get was Windows asking what I would like to use to open the file and Visio viewer wasn't an option. After drilling down to find the executable file, I found the viewer (VPREVIEW.EXE) would display a message saying "This program can only run from within another program” when I tried to execute it. I discovered the Visio viewer is designed to use ActiveX controls within Internet Explorer. Since I had disabled IE 11 on my system (using the "Turn Windows features on or off"), the viewer had nowhere to execute since Edge doesn't support ActiveX.

I found a Chrome plug-in in the Chrome web store that will allow me to view Visio files from inside Chrome. However, it requires me to click on a tag in the Chrome header and then drag the Visio file into the Chrome window.

So, the alternatives appear to be to enable IE 11 or use a Chrome plug-in. 

 

Networking, Windows 10, Visio, ActiveX,
 

Brocade (now owned by Ruckus) switches have two versions of code loaded by default to two different partitions. The primary partition contains a layer 2 image, while the secondary partition contains a layer 3 image. For the ICX-6610 switches, the layer 2 image is named FCXSxxxxx.bin and the layer 3 image is named FCXRxxxxx.bin. Notice the S and R to notate layer 2 (switching) and layer 3 (routing), respectively. In order to enable layer 3 features on an ICX-6610 switch, you must switch to the layer 3 image by updating the boot record. You can see the following link for more information: https://community.brocade.com/t5/Ethernet-Switches-Routers/FastIron-image-files-and-choosing-Layer-2-or-Layer-3-code/ta-p/2887

Also, you are able to update these switches using a manifest file, which allows it to do most of the update for you. Using the manifest file will only update the running code version. To update the non-running code, you can use TFTP.

Brocade has recently sold their product lines to several companies. The product line for larger equipment was purchased by Broadcom and the brocade.com website can still be used for information for these devices. The product line for smaller equipment (what most of our customers own) was sold to Ruckus and information/downloads can only be found on the ruckuswireless.com website.

 

 

Networking, Brocade,
 

Recently, quiesce snapshot jobs for some customers kept showing up as failed in Veeam with the error "msg.snapshot.error-quiescingerror".

After exhausting several research options I called VMware Support and we began sifting through event log files on the server as well as looking at the VSS writers and how VMware Tools was installed.

Looking at the log files on the ESX host the server was on led to this article:

https://kb.vmware.com/s/article/2039900

A folder named backupScripts.d gets created and references a path C:\Scripts\PrepostSnap\ which is empty. Therefore the job fails. The fix is found below:

  1. Log in to the Windows virtual machine that is experiencing the issue.
  2. Navigate to C:\Program Files\VMware\VMware Tools.
  3. Rename the backupscripts.d folder to backupscripts.d.old

If that folder is not present, and/or if the job still fails, the VMware services checked.

Networking, Veeam, VMware, ESX, snapshot,
 

I was working with a customer who called in a disk space issue. I ran SpaceSniffer and discovered  there were 92GB of files in a temp folder and nearly all were cab files.

My research discovered that on Windows 7 64bit and Server 2008 R2 the makecab.exe utility breaks whenever a log file is over 2GB. The problem is that the cabinet file format cannot store files larger than 2GB and it breaks the compression process as a result. Consequently all new logs created afterwards never get created properly and C:\Windows\Temp fills up with corrupt cabinet files; as much as 200MB+ a day. 

The only solution is to delete all of the corrupted cab files from the temp folder and the initially corrupted log file in the CBS folder. Here is a link to the article explaining this issue. 

https://serverfault.com/questions/746849/windows-temp-large-amounts-of-cab-xxxx-files

Networking, cab file, Windows,
 

I was working on a Windows Server 2016 system that I had already configured and put in production.  While I was monitoring the resources of the server when I noticed the processes for Windows Defender show up in the task manager as if it were scanning files.  This server already had a different AV solution installed.
 
As it turns out, you need to uninstall Windows Defender feature manually on Windows Server 2016 (https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-antivirus/windows-defender-antivirus-compatibility​).  Windows Server 2016 will not disable Windows Defender automatically.
 
"On Windows Server 2016, Windows Defender AV will not enter passive or disabled mode if you have also installed a third-party antivirus product. If you install a third-party antivirus product, you should uninstall Windows Defender AV on Windows Server 2016 to prevent problems caused by having multiple antivirus products installed on a machine." 
Networking, Windows Defender, Windows Server 2016,
 

Recently, I built a brand new Server 2016 system for one of our customer domains. This was the first Server 2016 server in this domain, but not the first one I had built – the procedures should have been fairly straightforward and similar to other build-outs in the past.

After the initial install, I went to the Settings app and tried to install the available Windows Updates. However every time that I tried to open Settings, I got this message saying the app had been blocked:

Strangely enough, I was the system administrator the message kindly passed the blame onto, and I don’t remember blocking the Settings app – especially on the first 2016 server in this network. As is typically the case in a lot of these random “issues” there was a Microsoft KB to the rescue explaining what was going on. https://support.microsoft.com/en-us/help/2750770/this-app-has-been-blocked-by-your-system-administrator-error-when-you 

This issue occurs when an administrator has deployed an application control policy (AppLocker) on the computer. By design, all Microsoft Store apps are blocked if an AppLocker policy is applied. Well, we do have AppLocker in use on this domain due to restricting access to applications for licensing purposes. Unfortunately, in order for me to even attempt to “unblock” the Settings app (which apparently qualifies as a Microsoft Store app), I needed to install the GPO tools on my new server. From there, I could set up the exception appropriately and get into Settings to install patches.

After the first reboot, I wanted to go into the users and groups and configure the access to allow the various groups of individuals to log in and do stuff. Clicking on the start menu does nothing. Hitting the windows key does nothing. Right click on the start menu works. Keyboard shortcuts (Win+R) works. Apparently the start menu qualifies as a Microsoft Store app as well. I loosened the restriction on this GPO and was able to get in and complete my setup.

Networking, Windows 2016, applocker,
 

The Meraki MX84 firewalls are subject to the Cisco Clock Signal Component issue that affects many firewalls and routers. With a typical Cisco device, we would copy the config from the old device and put it on the new device. Since Meraki devices are cloud managed, copying the config in this way is not possible. I called Meraki tech support and they said this could be accomplished by using a process called a “MX Cold Swap”. Meraki documents that process here: https://documentation.meraki.com/MX-Z/Other_Topics/MX_Cold_Swap_Replacing_an_Existing_MX_with_a_Different_MX
 
After performing the MX Cold Swap, everything seemed to be working, until we tested email. Inbound and outbound emails were not being delivered and email was queuing on the Exchange server. Since our visibility into the Meraki devices is limited to the web portal, I called Meraki tech support for assistance troubleshooting. After troubleshooting, we found that that main IP address was working, but the other public IP addresses that were NAT’d were not working. We tried rebooting the ISP’s equipment onsite to clear any MAC address tables, but that did not solve the issue. The Meraki tech support engineer found an article in their knowledge base that describes this issue: https://documentation.meraki.com/MX-Z/NAT_and_Port_Forwarding/1%3A1_NAT_Rules_not_working_properly_after_installing_MX.
 
The solution is to log into the local status page of the Meraki firewall and set the main IP to the NAT’d IP that is not working. This adds the NAT’d IP addresses to the ARP cache on the upstream routers. After a few seconds, you change the main IP address back to the correct address. Repeat for any other NAT’d IP addresses. This solution worked for the NAT’d IP addresses that originally were not working.
Networking, Meraki,