Exchange 2007 SP1 – Rollup 5 installation Issue (OWA)

There have been quite a few incidents reported out there regarding installation issues with application of Rollup 5 (and other versions) to Exchange 2007.  Granted many incidents have been reported with the Rollup 5 install for Exchange 2007 RTM; however there are also incidents with Exchange 2007 SP1 as well.  Below I will provide a brief overview of a incident I have experienced and action taken to remediate.

Scenario:

   Server State Pre-Installation of Exchange 2007 SP1 Rollup 5: 

• Exchange 2007 SP1 (w/ Rollup 3 applied). 
• No interim hotfixes for Exchange 2007 applied.
• Antivirus services stopped
• Backend servers have applied the rollup successfully

Problem(s):

• Installation of Rollup 5 takes LONG time (30-60 min) – Hangs on .NET assemblies portion of installation
• Exchange related services are left in a “DISABLED” state following installation as well as Reboot.
• Exchange related service will not start up.
• Users accessing OWA images are not displayed properly and pageNotFound for reading pane, etc…..

Symptoms:

• Visible Symptoms:
  • MSExchangeOWAAppPool does not exist for Rollup 5 version.  (MSExchangeOWAAppPool > OWA 8.1.336.0)
    Problem Server
    
    
    Good Server
    
    
    
    
• OWA does not display correctly when accessed.  Images are replaced with a “X”.
• *exe.config (C:\<drive>\Microsoft\Exchange Servers\Bin files have not been “recreated/updated” (reflecting Rollup application date)
  

Solution:

The installation of the rollup issue was not consistent across the board with the Client Access Servers.  Some servers installed successfully and other did not.  So the root cause was not determined; however the fix was the same across the board for problem servers.  

1. If stopped > Start IISAdmin
2. Modify the properties of the OWA 8.1.336 virtual Directory (Default Web Site > OWA > OWA 8.1.336) and assign Application Name and Application Pool (MSExchangeOWAAppPool)
3. Start Exchange Related Services
4. If services do not start, Verify *exe.config file contents for the following files.
5. Start Exchange Related Services
6. Verify OWA Authentication and set accordingly
7. Verify OWA Connectivity

Start IISAdmin (if stopped/disabled)

1.  Open command Exchange Management Shell.

If Disabled Type:  Set-Service “IISAdmin” –startupType Automatic
If Stopped Type:  Start-Service “IISAdmin”

Note:  Verify W3SVC and HTTPSSL service as well.

Verify MSExchangeOWAAppPool virtual Directory for Rollup 5 (owa 8.1.336)

1.  Open IIS Manager > ServerName > Application Pools >

MSExchangeOWAAppPool > owa/<RollupVersion>   (ex.  <default web site>/owa/8.1.336)

2. If version does not exist that matches the rollup version (Rollup 5 for Exchange 2007 SP1 = 8.1.336), navigate down to the Web Sites > Default Web Site > OWA > <rollup version>. 

3.  Right click the virtual directory that matches the rollup version, select Properties > Application Name click “Create”.  Enter “owa” text.  and then for Application Pool select MSExchangeOWAAppPool.

4.  Select OK and verify that the MSExchangeOWAAppPool > owa/<rollupVersion> exists.  (Procedure in Step 1.)

5. From command prompt, run:    IISRESET /NoForce

If Exchange Services do not startup normally or hang.   Verify the Exchange   *.exe.config File Content of the following files:

Reference Article:  http://msexchangeteam.com/archive/2008/07/08/449159.aspx

**Advise following the reference article for procedure.  However the gist of it is listed below.

1.  Check to see if the configuration files within the <Drive>\Microsoft\Exchange Server\bin directory contain the following key entry.  If not backup the existing files and either create or modify existing files.  List of files is below as well.

<configuration>

  <runtime>


          <generatePublisherEvidence enabled="false"/>


  </runtime>


</configuration>

*.exe.config files:  (exe.config versions of these files; so EdgeTransport.exe will have a EdgeTransport.exe.config file)

Bin\EdgeTransport.exe Bin\ExBPA.exe Bin\ExBPACmd.exe Bin\ExTRA.exe Bin\Microsoft.Exchange.Cluster.ReplayService.exe Bin\Microsoft.Exchange.EdgeSyncSvc.exe Bin\Microsoft.Exchange.Monitoring.exe Bin\Microsoft.Exchange.Search.ExSearch.exe Bin\Microsoft.Exchange.ServiceHost.exe Bin\MSExchangeMailboxAssistants.exe Bin\MSExchangeMailSubmission.exe Bin\MSExchangeTransportLogSearch.exe ClientAccess\PopImap\Microsoft.Exchange.Imap4.Exe ClientAccess\PopImap\Microsoft.Exchange.Pop3.Exe

2.  After the configuration files have been modified.  Try to startup the Exchange services.

Verify OWA Authentication:

1.  First check to see if the authentication method is set correctly for the OWA virtual directory.

get-owavirtualdirectoy -identity "msgexsv23004\owa (default web site)" | fl *authenti*

InternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated}
BasicAuthentication           : False
DigestAuthentication          : False
WindowsAuthentication         : False
FormsAuthentication           : False

2.  If authentication is not set correctly then set it according to your standards.  Below is an example of enabling WIndows Integrated and Basic Authentication.

Set OWA Virtual Directory authentication (Integrated and Basic):

set-owavirtualdirectoy -identity "msgexsv23004\owa (default web site)"  -WindowsAuthentication $true -BasicAuthentication $true

3. Next you will want to reset IIS.  Open a command prompt and run the following command.


 IISRESET /NOFORCE

4. Finally, verify authentication settings and open a browser and test OWA access/connectivit. (Internal and External URLS).   The command below is the same verification command issued in step 1.

get-owavirtualdirectoy -identity "msgexsv23004\owa (default web site)" | fl *authenti*

Exchange 2007 – UM – Terminology

For the telephony challenged Exchange 2007 professional…  Below are some common concepts and terminology that may stumble across during your Unified Messaging deployment.

Telephony Concepts
Terminology	Description
Circuit Switching (Telephone Networks)	→ Network in which there is a dedicated connection. → Analog and digital circuit switched networks        ▪ Analog - deisgned for voice transmission        ▪ Analog transmission signal must be encoded or converted into a digital format         before it enters the telephony WAN
Connectivity Concepts	-->T1 and E1 lines      ▪ E1 line can handle more simultaneous calls      ▪ T1 is used mostly in North America and Japan Trunk and tie lines       ▪ Trunk line - connects corporate PBX back to the phone company's PBX system       ▪ Tie line - typically a T1 or E1 line used to connect two or more corporate PBX systems Time Division Multiplexing (TDM)     ▪ Transmits separate voice signals simultaneously over a single physical, high bandwidth phone line
Packet Switching (Data Networks)
Voice over IP (VoIP)	Voice over IP (VoIP) is a technology that contains hardware and software that enabled people to use an IP-based network as a transmission medium for telephone calls. Protocols: SIP ; RTP ; T.38 (fax)
Session Initiated Protocol (SIP) and Real-Time Transport Protocol (RTP)	SIP  Only used for setting up and tearing down voice or video calls RTP:  All voice and video communications occur over RTP
Common Existing Enterprise Scenarios
System	Description
Central Office Telephone Exchange (Centrex)	A business phone service offered by a local telephone company from the phone company's central office (CO)
Key Telephone System	Is a multiple line telephone system typically employed by small organizations.
Private Branch eXchange (PBX)	aka Telephone switch or telephone switching device… Connects office telephones in a business to the public telephone network.
PBX Types
Type	Description
Analog PBX	Voice and signaling remain analog
Digital PBX	A code converts audio to a digital signal of a particualar format and vice versa.  Digitized voice is routed over circuit-switched channels
IP PBX	The IP phone is a node on the data network and can digitize voice ; Voice traffic will be packet switched and routed over LAN and WAN data networks.
Hybrid PBX	A combination of IP and digital capbilities in a singe PBX
PBX Concepts
Concept/Term	Relationship to Unified Messaging
Direct Inward Dialing (DID)	Only requires DID for targeted custom auto attendants, such as one to go to "Sales".
Dial Plan	Dial plan +extension is unique identifier
Coverage Path	Used on specific extensions to route calls from the PBX to Unified Messaging
Hunt group	Direct calls to VoIP gateway and/or Unified Messaging
Pilot number	Used for Call answering, Outlook Voice Access, and Auto Attendants. (Address or Label Group)
Unsupervised Transfer	Transfer method used by Unified Messaging
VoIP Gateways
Concept
Their function-----	Translate TDM or Telephone circuit switched based protocols from PBX system to protocols based on VoIP or IP   (SIP; RTP ; T.38)
Types of VoIP Gateways	IP/PBX  - Single device capable of communicating directly with Unified Messaging server using supported VoIP protocols.
	PBX (legacy) and IP/VoIP gateways - Two separate components are deployed ; the legacy PBX and the VoIP gateway performing translation to supported VoIP protocols

Exchange 2007 – Services By Role

Below is a table which breaks down the Exchange 2007 related services and the Exchange role associated with that service.

Service	Description	MBX	CAS	HUB	EDGE	UM
Microsoft Exchange Active Directory Topology	Provide Active Directory topology information to Exchange services.	x	x	x		x
Microsoft Exchange ADAM	Provides ADAM directory service function to the Edge Transport server				x
Microsoft Exchange Credential Service	Manages credentials that the Hub Transport Server uses to authenticate to ADAM for a subscribed Edge Transport Server				x
Microsoft Exchange EdgeSync	Connects to ADAM instance on subscribed Edge Transport servers over secure Lightweight Directory Access Protocol (LDAP) channel to synchronize data between a Hub Transport server and an Edge Transport server			x
Microsoft Exchange File Distribution	Used to distribute offline address book and custom Unified Messaging prompts		x			x
Microsoft Exchange Anti-spam Update	Used to automatically download anti-spam filter updates from Microsoft Update			x	x
Microsoft Exchange IMAP4	Provides IMAP4 services to IMAP clients		x
Microsoft Exchange Information Store	Manages Exchange Server databases. Provides data storage for messaging clients	x
Microsoft Exchange Mail Submission Service	Used to notify a Hub Transport server located in the Mailbox server's Active Directory site that messages are ready for retrieval from a sender's outbox	x
Microsoft Exchange Mailbox Assistants	Provides functionality for Calendar Attendant, Resource Booking Attendant, Out of Office Assistant, and Managed Folder Mailbox Assistant	x
Microsoft Exchange Monitoring	Provides a remote procedure call (RPC) server that can be used to invoke diagnostic cmdlets	x	x	x	x	x
Microsoft Exchange POP3	Provides POP3 services to POP3 clients		x
Microsoft Exchange Replication Service	Provides log shipping functionality for local continuous replication (LCR) and cluster continuous replication (CCR)	x
Microsoft Exchange Search Indexer	Provides content to the Microsoft Search (Exchange Server) service for indexing	x
Microsoft Exchange Service Host	Configures the RPC virtual directory in Internet Information Services (IIS), and registry data for ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere	x	x
Microsoft Exchange Speech Engine	Provides speech processing services for Unified Messaging					x
Microsoft Exchange System Attendant	Provides monitoring, maintenance, and directory lookup services for Exchange Server	x
Microsoft Exchange Transport	Provides Simple Message Transfer Protocol (SMTP) server and transport stack			x	x
Microsoft Exchange Transport Log Search	Provides message tracking and transport log searching	x		x	x
Microsoft Exchange Unified Messaging	Provides Unified Messaging features, such as the storing of inbound faxes and voice mail messages in a user's mailbox, and access to that mailbox via Outlook Voice Access					x
Microsoft Exchange (Exchange Server)	Provides full-text indexing of mailbox data content. This is a Microsoft Exchange-customized version of Microsoft Search	x

Exchange 2007 – Mail Disabling Objects

 Mail Disabling Objects

This section will detail procedures for mail disabling mail objects. The mail disabling process essentially strips the mail objects of its Exchange mail attributes. All Mail Specific attributes will be removed from the Active Directory object. The “Windows Email Address” (aka “mail”) attribute is the only attribute that will remain in tact, eventhough this attribute exists the object is not considered a mail object.

Mail Disabling Mail Users

How To Mail Disable a Mail User (Mail-Enabled user).

Command Shell
Note: The –identity can be any of the following attribute values for the mail user you are going to disable.

Identity Values

ADObjectID

GUID

distinguished name

Domain\SamAccountName

user principal name (UPN)

LegacyExchangeDN

E-mail Address

User alias

1. Open Exchange Management Shell
2. Have the Identity value on hand.
3. Enter the following command line pull the correct user object.

Get-MailUser –identity “<idValue>”

4. Verify that you have the correct Mail User based on the properties displayed.


5. If correct user has been chosen enter the following command to disable the mail user.


Tip: You can “Up” arrow to pull the command just entered above.

Get-MailUser –identity “<idValue>” | Disable-MailUser

6. Confirm action to perform. Press Enter.

Management Console (GUI)

1. Locate object within Exchange management console


2. Right Click and select Disable.


3. Follow the wizard and confirm successful disabling.

Mail Disabling Mailboxes

How To Mailbox Disable a Mailbox (Mailbox-Enabled user).

Command Shell

Note: The –identity can be any of the following attribute values for the mailbox you are going to disable.

Identity Values

GUID

Distinguished name

Domain\Account

User principal name

LegacyExchangeDN

SmtpAddress

Alias

1. Open Exchange Management Shell

2. Have the Identity value on hand.


3. Enter the following command line pull the correct mailbox object.

Get-Mailbox –identity “<idValue>”

4. Verify that you have the correct Mailbox based on the properties displayed.


5. If correct user has been chosen enter the following command to disable the mailbox.


Tip: You can “Up” arrow to pull the command just entered above.

Get-Mailbox–identity “<idValue>” | Disable-Mailbox

6. Confirm action to perform. Press Enter.

 Management Console (GUI)

1. Locate object within Exchange management console

2. Right Click and select Disable.


3. Follow the wizard and confirm successful disabling.

Mail Disabling Distribution Groups

How to Mail Disable a Distribution Group.

Command Shell

Note: The –identity can be any of the following attribute values for the distribution group you are going to disable.

Identity Values

GUID

DN

LegacyExchangeDN

Domain\Account Name

Alias

1. Open Exchange Management Shell

2. Have the Identity value on hand.


3. Enter the following command line pull the correct distribution group object.

Get-DistributionGroup –identity “<idValue>”

4. Verify that you have the correct Distribution Group based on the properties displayed.


5. If correct user has been chosen enter the following command to disable the distribution group


Tip: You can “Up” arrow to pull the command just entered above.

Get-DistributionGroup–identity “<idValue>” | Disable-DistributionGroup

6. Confirm action to perform. Press Enter.

Management Console (GUI)

1. Locate object within Exchange management console


2. Right Click and select Disable.


3. Follow the wizard and confirm successful disabling.

Mail Disabling Contacts

How to Mail Disable a Mail Contact object.

Command Shell

Note: The –identity can be any of the following attribute values for the mail contact you are going to disable.

Identity Values

ADObject

Distinguished name

GUID

Alias

2. Have the Identity value on hand.

3. Enter the following command line pull the correct distribution mail contact object

Get-MailContact –identity “<idValue>”

4. Verify that you have the correct Distribution Group based on the properties displayed.

5. If correct user has been chosen enter the following command to disable the Mail Contact.


Tip: You can “Up” arrow to pull the command just entered above.

Get-MailContact–identity “<idValue>” | Disable-MailContact

6. Confirm action to perform. Press Enter.

Management Console (GUI)

1. Locate object within Exchange management console



2. Right Click and select Disable.



3. Follow the wizard and confirm successful disabling.

Exchange 2007 – Message Tracking and Logging

Message Tracking and Logging

How to verify message tracking settings.

To check current message tracking parameters set for an exchange mail server run the following commands. By default message tracking is enabled on exchange mailbox, Hub Transport and Edge Transport servers.

1. Start the Exchange Management Shell

2. Type Get-mailboxserver “servername” |fl *tracking*

3. The resulting output will show current tracking settings.

How to change the default Log file path

By default Exchange stores message tracking log files in the C:\Program files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking.

To change this location use the following procedures. “C:\exchange logs” should be replaced with the directory of your choosing.

1. Start the Exchange Management Shell

2. Type Set-TransportServer “servername” –MessgaeTrackingLogPath “C:\Exchange Logs”

Note: The log file path must be local to the server. If the new target directory does not exist, the command will create it.

All log files that were generated in the original directory will not automatically be moved to the new directory.

How to change the default log directory size limits

By default the message tracking log directory is will over write the oldest log files once the maximum directory size of 250MB has been reached. The following procedures will change the max directory size.

1. Start the Exchange Management shell

2. Type Set-TransportServer servername –MessageTrackingLogMaxDirectorySize 1GB

Note: Sizes are factored in bytes (B), Kilobytes (KB) Megabytes (MB) gigabytes (GB) and terabytes (TB)

How to set the Max log file age

By default Exchange will retain message tracking log files for a period of 30 days before overwriting them. The following procedures will change this retention time.

1. Start the Exchange Management shell

2. Type Set-TransportServer servername –MessageTrackingLogMaxAge DD.HH:MM:SS

How to Track messages

Exchange 2007 provides a message tracking tool available through the Exchange Trouble shooting assistant in the Exchange management console as well as through Exchange Command Shell.

 

Management Shell – Message Tracking

Use the following procedure to track a message in Exchange Management Shell

1. From Hub Transport server (DCA-APP-CSHU01) open the Exchange Management Shell

2. Enter the following command line with the correct parameters.

Get-MessageTrackingLog <SearchFilters>

Filter Table:

Search filter	Corresponding field in the message tracking log
End	date-time
EventId	event-id
InternalMessageId	internal-message-id
MessageId	message-id
MessageSubject	message-subject
Recipients	recipient-address
Reference	reference
ResultSize	None. This parameter limits the number of results that are displayed by the search.
Sender	sender-address
Start	date-time

 

Console - Message Tracking

Use the following procedures to track a message in Exchange Console.

1. Start the Exchange Management Console.

2. In the left hand pane select Toolbox.

3. In the middle pane select Message tracking.

4. With Message tracking highlighted select Open tool from the right hand pane. This will open the Exchange Troubleshooting assistant (Extra).

Within Extra, the following search criteria are provided to track messages.

• i. Recipients.



• ii. Sender.



• iii. Server sent from.



• iv. Event ID



• v. Message ID



• vi. Internal Message ID



• vii. Subject.



• viii. Reference.ix. Start and End dates for the search.

5. Once the search criteria has been defined click next to begin the search.

6. The search results will display the following data fields.

a. Time stamp b. Event ID c. Source d. Source Context e. Message ID f. Message Subject g. Message Sender h. Recipients i. Internal Message j. Client IP

a. Client Hostname b. Server IP c. Server Hostname d. Connector ID e. Recipient status f. Total Byte size g. Recipient count h. Related recipients i. Reference j. Return path k. Message Info

 

Message Tracking Example

This section will provide details for resolving a real world message routing issue.

Scenario: Sender Joe.Smith@company.com received NDR when sending to recipient:

Info	Details
Sender	Joe.smith@company.com
Recipient	Another.User@company.com
Date Time Message Delivered	10/07 12:01am – 2:00AM

NOTE: This is a Domino to Exchange mail routing scenario. All mail sent from Notes to Exchange targets the following Hub Transport Server (DCA-EM-CSHU01.amat.com)

Exchange Management Shell – Message Tracking

Below is the process for tracking the cause for the NDR for recipient Another.User@company.com

1. From Hub Transport server (HT01) open the Exchange Management Shell.

2. Enter the following command line based on information in table above.

get-messagetrackinglog -Recipients:Another.User@company.com -Start "10/7/2008 12:01:00 AM" -End "10/7/2008 2:00:00 AM"

3. Data will be displayed for the messages that meet the filter criteria.

4. Key in on the eventID column. “Fail” is key indicator. To view the that particular message details enter the following command.

get-messagetrackinglog –eventID “Fail” -Recipients:Another.User@company.com -Start "10/7/2008 12:01:00 AM" -End "10/7/2008 2:00:00 AM"

5. This will narrow down the search criteria to failed messages. Now view the message in Format-List view to display the message details.

get-messagetrackinglog –eventID “Fail” -Recipients:Another.User@company.com -Start "10/7/2008 12:01:00 AM" -End "10/7/2008 2:00:00 AM" | format-list.

6. The message status states the RecipientStatus of 550 5.1.4 ambiguous address. Now this can be a result of duplicate email adddresses or duplicate “legacyExchangeDN” addresses.

7. To verify, perform a number of searches for duplicates within the console to determine the problem.

a. To locate duplicate email addresses run the following command:

i. Get-Recipient –id Another.User@company.com  .

ii. This command only came up with one recipient using that “email Address”. So duplicate email address is not the cause of the problem. Next step is to discover whether or not there are duplicate legacyExchangeDNs.

b. To locate duplicate legacyExchangeDNs.

Get-user –id “Another.User@company.com” 

ii. This command comes up with two users. Now to compare legacyExchangeDNs. Run the format-list option for each user to get the “RecipientType” in order to display and inspect the “legacyExchangeDN” value.


Note: Another indicator of duplicate legacyExchangeDns is duplicate “Alias” attributes (aka mailNickname).

iii. If “RecipientType” value equals “MailUser”. Then run the following command.

get-mailUser –id “Another.User@company.com” | fl 

iv. Inspect the legacyExchangeDN for each user. In this case (although not shown) each user had the same LegacyExchangeDN value. This was due to the HR contractor to perm employee conversion process.

v. To resolve the mail routing issue for legacyExchangeDN duplicates. The incorrect account must be mail disabled (stripping of Exchange Attributes).

Exchange Console– Message Tracking

Below is the process for tracking the cause for the NDR for recipient Another.User@company.com

Use the following procedures to track a message in Exchange Console.

1. Start the Exchange Management Console.

2. In the left hand pane select Toolbox.

3. In the middle pane select Message tracking.

a. Enter the following parameters for:

b. Recipients.

c. Start and End dates for the search.

4. Press “Next”

5. The next screen will disply the filtered list based on your pararmeters.

6. Notice the “EventID” which indicates a FAIL. The same information is available within this screen as it is within the command shell.

7. Note the Recipient Status column. This will provide the delivery issue.

8. This particular error indicates that there are duplicate email addresses or legacyExchangeDN duplicates. The easiest method to discover the root cause is through the command shell. Below is the command shell process used to resolve this particular recipient delivery issue.

9. To verify, perform a number of searches for duplicates within the console to determine the problem.

a. To locate duplicate email addresses run the following command:

i. Get-Recipient –id Another.User@company.com  .

ii. This command only came up with one recipient using that “email Address”. So duplicate email address is not the cause of the problem. Next step is to discover whether or not there are duplicate legacyExchangeDNs.

b. To locate duplicate legacyExchangeDNs.

Get-user –id “Anja_Niederbremer@amat.com” 

ii. This command comes up with two users. Now to compare legacyExchangeDNs. Run the format-list option for each user to get the “RecipientType” in order to display and inspect the “legacyExchangeDN” value.


Note: Another indicator of duplicate legacyExchangeDns is duplicate “Alias” attributes (aka mailNickname).

iii. If “RecipientType” value equals “MailUser”. Then run the following command.

get-mailUser –id “Anaj_Niederbremer” | fl .

iv. Inspect the legacyExchangeDN for each user. In this case (although not shown) each user had the same LegacyExchangeDN value. This was due to the HR contractor to perm employee conversion process.

v. To resolve the mail routing issue for legacyExchangeDN duplicates. The incorrect account must be mail disabled (stripping of Exchange Attributes). Once stripped only one account will contain that legacyExchangeDN and therefore fix the mail delivery issue.