Blog: Exchange

I’ve been working on migrating an Exchange environment to 2010. This process includes an upgrade to the Unified Messaging role of Exchange to 2010 as well. We had purchased a UCC certificate to include all the Subject Alternate Names our Exchange environment would need and I had already applied it to the CAS server successfully. Since this certificate also included the FQDN of our UM server, I added the cert. and assigned the UM service to it so that Exchange could start processing voicemails through 2010 instead of 2007. [more]

After I had moved a couple of mailboxes (including mine) over for testing, I discovered that I could no longer receive voicemail. People were redirected to the Subscriber Attendant instead of my individual mailbox. There were events logged on the UM server saying the following:

Event ID: 1400 Source: MSExchange Unified Messaging
The following UM IP gateways did not respond as expected to a SIP OPTIONS request. Transport = TLS, Address = lync.ourdomain.com, Port = 5061, Response Code = 0, Message = This operation has timed out.

After considerable troubleshooting, I ran across a forum posting (http://social.msdn.microsoft.com/Forums/en-US/communicationsserversdk/thread/2733adf2-c91b-4b94-bf43-be93cdd2d5ef/) from someone who encountered the same issue and called for a support incident with Microsoft to figure out what was going on. In order for UM to work in a Lync phone system environment, the Subject Name of the certificate installed must be the FQDN of the UM server itself. It won’t work if it’s just included as a Subject Alternate Name; it must be the Subject Name.

I generated a new certificate from our internal CA with the UM server as the SN of the certificate, installed and assigned it to the UM roles (leaving our UCC cert running the remaining roles), and immediately started receiving voicemail notifications.

I found one other blog posting after the fact that backed this claim up even more (http://therealshrimp.blogspot.com/2012/03/multiple-exchange-um-servers-and.html).


 

I had upgraded a terminal server from Office XP to 2010 recently and users were getting pop up warnings about autodiscover.domain.com and remote.domain.com names not matching the certificate.  Exchange happened to be setup with both external and internal autodiscover URLs pointing to an external domain address. 

In order to resolve this issue completely, I had to change the internal URLs used by autodiscover in four places.  The URLs need to be configured using the Exchange Power Shell.  The commands I list basically get the value first, and then set the value.  [more]

This article helped me find most of the commands: http://www.msexchange.org/articles_tutorials/exchange-server-2007/management-administration/configuring-exchange-server-2007-web-services-urls.html

Get-AutoDiscoverVirtualDirectory | fl
Set-AutoDiscoverVirtualDirectory -internalUrl “https://internalname/Autodiscover/Autodiscover.xml” -identity “<IDENTITY>”  

Get-ClientAccessServer | fl
Set-ClientAccessServer -Identity "<IDENTITY>” -AutodiscoverServiceInternalURI "https://internalname/Autodiscover/Autodiscover.xml"  

Up to this point, this fixed the SCP URL and allowed the Autodiscover E-mail test to return data.

I had to change the following internal URLs on these services also:

Get-WebServicesVirtualDirectory | fl
Set-WebServicesVirtualDirectory -Identity "<IDENTITY>" -InternalUrl "https://internalname/EWS/Exchange.asmx"

Get-UMVirtualDirectory | fl
Set-UMVirtualDirectory -Identity "<IDENTITY>" -InternalUrl "https://internalname/UnifiedMessaging/Service.asmx"


 

I was performing a PST e-mail import task for a migration being done recently.  The user’s PST files were larger than the mailbox quota limits set at 200 MB.  Once the import reached 200 MB of data, it stopped and gave me an error in powershell. 

After examining the quota limit on the mailbox, I increased the size and tried the import again.  It kept failing immediately and the logs showed that the quota limit had been reached. 

After some searching, I found out that there is a waiting period between changing the quota limit and it actually taking effect.  To make the change happen immediately, I found that you can restart the Microsoft Exchange Information Store service, and it will update the quota limits on the mailbox.


 

We had a user recently that was reporting excessive amounts of spam in her inbox. This company uses Postini as their filtering service, so naturally, this didn’t seem quite right. After some research, I determined that it was non-account blocking (a Postini feature) that was causing the problem. In this example, let’s assume the user is Jane Smith. Her email address is jane.smith@company.com. The spam was coming into jsmith@company.com, an alternate SMTP address in Exchange.

Non-account blocking in Postini bounces all email that comes to addresses not registered in the Postini user database. If this feature is not enabled (as was the case here), Postini does not filter the email according to the spam filters and, instead, passes it through untouched. The jsmith@company.com address was not added into the user database as either a user or an alias to a user. When Postini received email on this address, it passed it straight through to their exchange server. The exchange server recognized the recipient as a legitimate user and delivered the mail as expected.

The fix here was to enable non-account blocking and add these secondary SMTP addresses as aliases in Postini. Jane has not received any spam since then.


 

Log Parser 2.2 is a free command line tool available from Microsoft.  It provides universal query access to text-based data such as log files, XML files, and CSV files.  It also can query Windows system data sources such as the Event Log, the Registry, the file system, Active Directory, and NetMon captures.  You can pick the information you want returned in the results and those results can be sent to a text file, SQL Server, or SYSLOG.  This tool basically reads your log files and lets you query them as if they were in a SQL Server database.  It is also light weight at only 1.4 MB download.

The possible uses for the Log Parser are endless, but I use is specifically for querying IIS logs when trouble shooting problems.  For example, using this tool makes it easy to find all the requests made by a specific signed in user.  Since this application is ran at the command like it can take a little time to get your query right, but after you get it working you can add the commands to a .bat file for future reference or scheduled tasks.  Here are some examples: [more]

Search IIS Logs for User Requests
Here is an example batch file that when run searches a directory of IIS log files for all requests made by users signed in with a username ending in “@example.com” and saves the results to a text file:
cd "C:\Program Files\Log Parser 2.2\"
logparser.exe "select logrow, date, time, c-ip, cs-username, cs-method, cs-uri-stem, cs-uri-query from ‘< your log directory path>\*.*’ where cs-username like '%%@example.com%%' order by date, time, logrow" -i:IISW3C -rtp:-1 > c:\temp\example-requests.txt

Search IIS Logs for Most Download Files
cd "C:\Program Files\Log Parser 2.2\"
logparser.exe " SELECT TOP 10 cs-uri-stem, count(*) as Downloads FROM ' from <your log directory path>\*.*' GROUP BY cs-uri-stem ORDER BY Downloads DESC" -i:IISW3C > c:\temp\most-downloaded.txt

Find 10 Largest Files in a Directory or Subdirectory
cd "C:\Program Files\Log Parser 2.2\"
logparser.exe " SELECT TOP 10 Path, Name, Size, Attributes FROM 'C:\Program Files\*.*' ORDER BY Size DESC"  -i:FS –Recurse:-1 > c:\temp\10-largest-program-files.txt

Get Number of Outbound Emails from Exchange
logparser.exe "SELECT connector-id, client-hostname, COUNT(*) AS Total INTO c:\temp\outbound-email-totals.csv FROM '<log file directory>\MSG*.log,<another log file directory>\MSG*.log' WHERE connector-id LIKE '%outbound' OR connector-id LIKE '%to Internet' GROUP BY client-hostname,connector-id WITH Rollup"  -i:CSV -nSkipLines:4 -o:csv

This is a very flexible tool.  There are tons of parameters that control how the application functions and the number of different queries you could write is only limited by your imagination.  I’ve found the best way to get started using it is to look at examples and there is a “Samples” folder included in the install directory that is helpful.

Related Links
Home Page (http://www.iis.net/community/default.aspx?tabid=34&g=6&i=1976)
Log Parser 2.2 Documentation (http://www.iis.net/community/default.aspx?tabid=34&g=6&i=1976)
Download (http://www.microsoft.com/downloads/en/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en)
TechNet Article (http://technet.microsoft.com/en-us/library/ee692937.aspx)
Other examples of IIS log queries (http://blogs.iis.net/carlosag/archive/2010/03/25/analyze-your-iis-log-files-favorite-log-parser-queries.aspx)
Log Parser Forums (http://forums.iis.net/default.aspx?GroupID=51)
Graphing Ping Results (http://www.adopenstatic.com/cs/blogs/ken/archive/2005/05/30/22.aspx)
Query Windows Event Log (http://oreilly.com/pub/a/windows/2005/07/12/logparser.html)


 

I had a problem with my iPhone. It was getting hot to the touch. I then discovered that it was chewing up download data... about 5MB every 15 minutes. This was discovered when AT&T sent me a message that my consumption of my monthly allotment was at 90%.

After many hours of work, I discovered that it was the Exchange server “push”   that was causing it to chew through data. Specifically, it was “push” on the Contacts folder. I ended up extracting my contacts folder to a PST file, and re-importing the file and this seemed to fix the issue of chewing through the Cellular Network Data. [more]

At this point, I realized that I had a problem syncing all my contacts. The contacts would just not all load onto my phone. This was not related to the issue above with Cellular Network Data, but the contacts download would just stop before synchronizing all the contacts. I had noticed this problem forever, but had not researched. It turns out that there were two contacts in my address book that were causing the problem. These contacts have been in my list for years.  After removing these two contacts ( I discovered which ones they were by dividing my list in halves  - binary search- until I isolated the culprits) everything works fine. I have not yet discovered the cause as to why these particular contacts will not sync. I sent one of the contacts to a coworker, and it will not sync with his phone (not an iPhone) either …


 

The CommVault Exchange Mailbox iData agents do not backup mailboxes associated with disabled Windows user accounts. The backup job reports a "success" for the job, but when the details of the backup are explored, the backup set does not contain any data. Additionally, requesting a listing of all failed objects for the backup job results in a "no failures" status. According to CommVault, this behavior is by design as is the "successful" backup status. After all, the job did not technically fail if it is not designed to include mailboxes belonging to disabled user accounts. This is very strange given that, in general, CommVault iData agents have an "inclusive by default" behavior.  This can become a real problem if you try to restore data for a former employee whose Windows user account was disabled when they left the company.  The lesson here is that you should always test your backups. Even if the backup report and all job status notifications indicate you are good....test anyway.


 

During the recent move of a customer’s servers to our network, we had to change the IP address to match our addressing scheme. This ended up breaking many of the applications on the server (including OWA) that we needed to go fix. I opened up IIS and changed the connection address from their previous address to the current address of the network. After running iisreset, OWA still did not work. I couldn’t get the websites to even start up. It was as if the server still wasn’t listening on the correct address.

Well, sure enough, that was the case. The command “httpcfg query iplisten” will show you the IP addresses that the server is listening for. In my case, I saw something similar to the following:

 IP : 127.0.0.1
-------------------------------------
 IP : 192.168.1.10
-------------------------------------

Where 192.168.1.10 is the wrong address. For the sake of this example, our “correct” address will be defined as 10.1.1.10. [more]

Now, there are two ways you can resolve this, the first is running “httpcfg delete 192.168.1.10” followed by “httpcfg set 10.192.0.10” which should resolve the problem. In addition, I found a knowledge base article (http://support.microsoft.com/default.aspx?scid=kb;en-us;890015) that explained how to reconfigure the IP addresses from the registry. After running through the instructions, followed by another iisreset, I got the following from my “httpcfg query iplisten” command:

 IP : 127.0.0.1
-------------------------------------
 IP : 10.1.1.10
-------------------------------------

Problem solved.


 

I installed Exchange 2010 on a new Windows 2008 R2 server for a customer. I was attempting to do a test move on a mailbox from the old Exchange 2003 server and it failed. I found that the Microsoft Exchange Mailbox Replication service was stopped and it would not start. I did some online research and was unable to find a solution.  After further investigation it was discovered that the VaultLogix Classic Agent used for the online backup was using the same port as the Mailbox Replication service. I spoke to a VaultLogix support technician who showed me a registry key that would change the default port from 808 for the agent.

I change “HKEY_LOCAL_MACHINE\SOFTWARE\EVault\InfoStage\Agent\AgentPortNumber” to port 807 and was then able to start the Mailbox Replication service. [more]

During the installation of the backup agent it will not allow you to change the port. However another method to change the port number once it is installed, is by opening the Classic CentralControl application right click on the server name and choose “Properties. Then change the port number to an available port.


 

A couple weeks ago, one of our customers had their Exchange SCR copy fail due to a corrupt log file. At first we assumed that the log file was corrupted during transit to the DR site, but after recopying the log file over multiple times and attempting to restart replication, we realized the log file was actually corrupted on the source server which is a virtual machine. I had never seen this happen before and was a little surprised that the corrupt log file had not taken the mailbox database offline. With nothing to attribute the corruption to, I decided it must have been a fluke and started a database reseed the following weekend. After 3 days, the database seeding finished, but 4 hours after the reseed completed, the SCR copied failed again…another corrupt log file. [more]

I decided there must be a bigger issue. I reviewed the logs and found numerous eventid 7 errors (bad block on disk) and a few pvscsi warnings. It seemed logical that maybe the paravirtualized SCSI adapter that was being used on this virtual machine may be causing an issue…maybe it was a weird PVSCSI / Windows 2008 server problem. I had to take a break from this issue to troubleshoot another server issue for the same customer. In doing so, I had an idea…what if the physical disk is going bad, but hadn’t completely failed. Could that cause the underlying VMware VMFS partition to look fine but cause problems with virtual disk files attached to VMs. I used iLO to check out the hardware status and sure enough one of the disks had encountered numerous SMART errors and was marked “impending failure”. The array was not degraded yet because the disk had not completely failed. I have replaced the disk and will reseed the database soon, but since replacement there have been no bad block on disk errors on this VM so it looks promising.