We had a customer report that all browser windows were closing for users and this was increasing in frequency. Most of the users reporting the issue were at the corporate office, which has about 150 users and is where the IT department is located. I performed a remote session with on the users and confirmed the issue. Internet Explorer, Chrome, and Firefox all would close, not crash, at the same time.
My first thought was that some remote assistance and IT management software they had recently installed was causing the issues. We uninstalled the software and the issues continued. My next thought was that something malicious was on the network and was killing the processes remotely. I moved the PC to the guest wireless network and the issues stopped. After moving the PC back to the internal network, the issues began again. After a while, the issues randomly stopped for this user. I moved on to looking at another user's PC. The IT department did not know of any new devices that had been brought onto the network.
Whatever was causing the issues was obviously powerful enough to kill processes. The browsers seemed to be closing at regular intervals, at the top of the hour and half after the hour. I started Process Monitor, Process Explorer, and WireShark, opened the browser, and waited. As expected, the issue occurred again. I started looking through the WireShark logs and did not see anything odd. I looked at the Process Monitor log and found several cmd.exe processes killing the browser applications. At about the same time I saw the cmd.exe commands that killed the browser processes, I saw nxclient.exe processes that called cmd.exe and ran taskkill commands.
I started searching online and found a blog on the NxFilter support group discussing the same issue. This customer has NxFilter for web filtering for several years. They were using version 5.0 of NxClient, which was before the version mentioned in the NxFilter support group. The NxFilter creator responded to that group and said that the client was doing so to force a refresh the user's session, but that this is not the correct behavior. There was a new version of NxClient that fixed this behavior. Version 9.1.3 of NxClient was current, so I updated the customer to use the newer version. That resolved the issues.
We recently moved a customer from a datacenter at one of their locations to a large datacenter in the Dallas/Ft. Worth area. One of the devices we moved was a Meraki MX84 being used as a VPN concentrator. A VPN concentrator works by extending the network the VPN concentrator is on to the access points. Basically, wireless clients at all locations get an IP address on the same layer two network. This is important for a few reasons. First, the VPN concentrator needs to be in it's own VLAN/DMZ. Second, something on the layer two network the VPN concentrator is connect to needs to be handing out DHCP addresses. In our case, we used a Fortigate UTM to run the DHCP server for that subnet. Third, traffic needs to be allowed outbound to the Internet from all clients on the VPN concentrator layer two network so clients can connect to the Internet. The traffic is tunneled from the access points to the VPN concentrator, so the traffic does not intermix with the normal network traffic.
One of the issues we had was that the access points would not create the tunnel back to the VPN concentrator. After talking to Meraki support, we found that the issue was that the access points and the VPN concentrator would not connect to each other if their public IP address was the same. This does not work because Meraki uses the same technology to build the VPN from the MX to the access points as they use to build a VPN mesh between MX devices. Our devices were both using the default overloaded outbound NAT rule, so they were coming from the same public IP address. The solution is to make the MX come from a different public IP address, which can be accomplished via an inbound and outbound NAT statement. After we made this change, the access points connected to the VPN tunnel and wireless began to work.
One other thing to note is that the access points will not broadcast SSIDs if the VPN to the concentrator is not up when configured to tunnel traffic through a VPN concentrator. This can be helpful when troubleshooting wireless when there are not clients at the location of the access points.
As a part of a recent data center move we had to reconfigure several APC management cards. The first thing that I did to each of these NMCs was to reset to factory defaults and update the firmware.
This is a fairly simply process normally. Connect the appropriate serial cable, connect to the comm port, press the reset button a couple of times, and log in with the default credentials (http://www.apc.com/us/en/faqs/FA156075/). Once logged in, you can use the command to factory_reset or format the card and bring it back to factory settings.
In one case, however, the card didn't survive the factory reset. In fact, it appeared the card had started to boot, but never finished the booting process. By changing the baud rate around in my settings, I was able to connect to the BootMonitor using 57600 baud, 8 data bits, no parity, no flow control. It was also at this point that I saw an error related to the checksum of the AOS firmware which was preventing the card from booting. We had another NMC that I could swap to if I needed to, but I wanted to see if I could simply reload the firmware onto the card and get everything working again. Fortunately, APC has an article for that: http://www.apc.com/us/en/faqs/FA293874/
Using TeraTerm and XMODEM, I was able to upload the bootmon, AOS, and application module firmware files (in that order) to the NMC. Once that had finally finished, simply rebooting the NMC brought everything back online.
This also had the positive side effect of updating the firmware on the card because I wasn't able to download old firmware files from APC's support site. From there, I could complete my setup process and bring everything back online.
Early this year the tech world was rocked with the announcement of two unprecedented vulnerabilities named Meltdown and Spectre.
These two vulnerabilities are a big deal because they are hardware vulnerabilities affecting any device with a silicon chip. This includes microprocessors on workstations and servers, mobile phones, tablets, cloud services, and other platforms.
Understandably there was a rush from three main industries, processor companies, operating system companies, and cloud providers to provide solutions. However, as a result of the urgent response, there were unanticipated update incompatibilities which crashed systems. This created a dilemma for IT professionals. "Do we install updates which may cause our systems to crash?" or "Do we sit-tight and remain vulnerable?"
Even in the weeks of uncertainty, there were calm voices of seasoned reasoning. Their message reminded us that basic security standards remain our first line of defense. No matter how bad an exploit may be, its impact can be limited if:
So how do you do achieve these standards? Here are some fundamental best practices:
One of our Technology customers recently migrated to a new AT&T WAN offering called AT&T Switched Ethernet – Network on Demand (ASE NoD). This is the most recent evolution of their metro Ethernet service with the addition of long-distance layer 2 connections.
What makes this "network on demand" is the ability to change bandwidth as needed through a web portal up to the physical Ethernet hand-off limit, typically 10Mb/s, 100mb/s, or 1Gb/s. The default rate for each of this customer's location was set to 20, 50, or 100Mb/s.
Since this is a relatively new product we had several Gotcha's in the implementation:
I have encountered issues on PCs that can't access CDs or flash drives that previously had removable media access restricted by either group policy or Symantec Endpoint device control. After the control restrictions were removed, trying to read from the CD or flash drive gave an "Access Denied" error.
The only way I've been able to resolve this issue is by going to the Device Manager, uninstalling the CD-ROM drive/flash drive, and then scanning for hardware changes to add it back.
My assumption is that some registry settings aren't being changed correctly when policies are removed, so re-adding the device recreates the registry settings for the hardware.
If you run the Windows 10 Upgrade Assistant and get an error code 0xc1900208. This means there is a compatibility issue with an application and the install is being blocked!
The only way to figure out what program is blocking it is to run a powershell script by Microsoft called Appraiser https://msdnshared.blob.core.windows.net/media/2018/03/AppRPS.zip
Recently, Microsoft released a production version of a new management interface called Windows Admin Center, formerly known as Project Honolulu. The purpose of this product is to provide a centralized or locally-deployed management interface that will (eventually, hopefully) replace Server Manager. It manages servers by using Remote PowerShell or WMI over WinRM and client systems through similar methods.
Simply add your machines in the list, assign credentials to connect with (they can be your own – it doesn't appear credentials would be shared between administrators), and connect. The main requirement is that the server you're connecting to for management has WMF 5.0 installed.
As you can see from the above screenshot, there are a TON of things that you can do through this interface – including: browsing files, managing local users, managing the registry, enabling RDP, installing roles & features, managing and installing Windows Updates. It's a really impressive application that I plan to start using more often. It's very responsive and even loads interfaces faster than the MMC equivalent in many cases. I can certainly see the difference in the Event Viewer, for example.
To learn more or to download the free product, check out https://docs.microsoft.com/en-us/windows-server/manage/windows-admin-center/overview
After installing each Windows 10 creator's update, I get the following error message when I try to click on any link in any email message or click on a table of contents link in a Word doc:
It's not an entirely bad thing to have email links require a copy and paste but it's a real problem with other links like the Table of Contents in a long Word document.
There is a KB article at https://support.microsoft.com/en-us/kb/310049 that discusses this issue. The solution for Windows 10 is to find a system that doesn't have the problem and export a registry key then import it into the offending system. The key it references gets deleted each time a new creator's update is installed.
HKEY_LOCAL_MACHINE\Software\Classes\htmlfile\shell\open\command
Then you export the subkey to a file, copy the file to the system having the problem and import it into that system's registry (either by double clicking the .reg file or importing it via regedit). There is a last verification step to verify the String (Default) value of "HKEY_CLASSES_ROOT \.html" key is "htmlfile".
That was several steps it took to make my system less secure. It's usually the other way around!