Remote Infrastructure Management

If studied closely, RIM makes more sense not in terms of a measure to cut costs during recessions but more as a business sense. How? Let’s see.

Today enterprises leverage the power of IT to streamline their key business processes and gain a competitive advantage. Even with the investments already made, the key issues relating to application performance, scalability, security, management and effective utilization are mostly unresolved. Apart from helping to reduce the overall IT Infrastructure Management costs, a well structured and managed RIM relationship can bring significant benefits.

Flexibility of Service
Customers hiring service providers have an advantage here as they can align the business needs with the services provided by the Service Provider. Services can be flexed to meet peak times and loads, mission-critical and non-mission critical or a combination of both. This type of flexible pricing model and service delivery helps in apportioning the infrastructure management costs.

Domain Expertise
In this highly volatile IT industry it is hard to keep up with the latest technology trends and it is almost impossible to have on pay roll a staff that can match the ever demanding and changing face of IT technology across all segments. A RIM service provider can add value here as they have diverse IT staffs which are at a client’s disposal, working 24×7. Moreover, these specialists are constantly updated with the latest technology trends via internal training programs, technical training programs from vendors, etc.

This basically helps companies to gain a competitive advantage as it helps to reduce the time to market. Companies can take the bold steps of adopting and switching to new technologies to gain a competitive advantage. Serious service providers in the RIM industry would normally have alliances in place with leading technology suppliers and as a result clients can also enjoy a single window of service from them with an assurance that the team working on their projects would be capable of handling the technology and related needs.

Visibility and Control
Companies retain full control of its strategic components such as physical IT assets, technology refresh, policy and architecture which provides visibility into the availability, performance and utilization of each component. Many of the established RIM suppliers have embraced the ITIL framework which lends itself to RIM very effectively. Putting the collected data in to this type of best practices framework provides lot of insight and visibility. This visibility further enables companies to take control of the situation, make informed decisions on future investments in IT and improve current service levels.

Reduced Costs
Remote Infrastructure Management not only assists in reducing the manpower cost of managing the infrastructure, but also brings down the costs related to future infrastructure investments. It has been reported and endorsed by agencies like McKinsey and NASSCOM that outsourcing the infrastructure management services can save dramatically reduce the labour related costs by 30% to 40%. On the other hand, companies can also reduce costs associated with future infrastructure investments using expertise of the service provider by way of infrastructure optimization, standardization using industry best practises and consolidation using virtualization technologies.

Proactive Services
Most of the time RIM services engagements are driven and governed by SLA’s (Service Level Agreements). Service providers, in order to maintain the SLA, invest in to software and technology that enable them to gain an insight on failing components in the IT infrastructure, usage patterns, thresholds, etc. These tools enable a service provider to provide 24×7 proactive monitoring and management via a Remote NOC (Network Operations Centre). This helps to resolve issues in a faster and structured manner.

So, in today’s fast paced and ever changing IT world, RIMs makes more business sense and CIOs have to step back and ask themselves what they’re trying to accomplish with infrastructure outsourcing. Cost savings, optimization, transformation or efficiency? CIO’s decisions on outsourcing infrastructure management are going to be driven by more than just cost savings and as a result it is also going to create a challenge for the service providers to tweak their service models to show more benefits and not only cost savings.

In the first step we have to configure a Server for monitoring and select/configure the resources to monitor. To do so, open the Exchange System Manager Snap-In and select the appropriate Server object in the Servers container.

Right-click the Server object and open the Properties for it.

Once the Properties dialog box is open, select the Monitoring tab as shown above.

There are many resources available for monitoring. We can add the following resources:

• Available Virtual Memory
• CPU Utilization
• Free disk space
• SMTP queue growth
• Windows 2003 Service
• X.400 queue growth

Many other 3rd party monitoring tools like Nagios can do most of the monitoring and so we will only add the two most critical items which are not monitored by Nagios. These are:

• Available Virtual Memory
• SMTP queue growth 

In the first step, we will add the Available Virtual Memory instance. Click on the Add button under Monitoring tab. A dialog box similar to the one shown below will open.

Select the Available Virtual Memory from the list and click on OK.

Configuring virtual memory monitoring is very similar to configuring CPU utilization monitoring. We have skipped the CPU Utilization Monitoring. Set the duration to 5 minutes, the warning threshold to 15% and the critical threshold to 10% for Available Virtual Memory. Once the values are set, click on OK.

Follow the process for adding another instance and select SMTP Queue Growth this time from the list.

If SMTP queues start backing up, it often indicates that there is a major problem. It shows that message transport has failed, the Internet connection has failed, or someone is using the server to send spam. In any event, we need to know that there is a problem.

We must input a warning and a critical state threshold value, but these values are entered in minutes. The idea is that we must tell Exchange how many minutes the queue should be constantly growing for before we generate a warning or a critical state message. We can use any values that we want, but I recommend setting the warning threshold at 10 minutes and the critical state threshold at 15 minutes.

Close the dialog boxes and the Properties once done. The next step is to configure an E-Mail notification for the recipients of the “Warning” and “Critical” state conditions. To do so, open the Notifications object under Tools.

Click on New and select E-mail Notification.

In this Property dialog box we can specify the monitoring Server and the Servers / Connectors to monitor. We also have to select the State (Critical and warning) for which we configure E-Mail notification.

First we will configure the Warning State notification.

In the “To” field select a recipient for the notifications. 

Change the Email server value to some other server in your organization which runs SMTP. If you do not have any, I suggest setting up a basic MS SMTP service in IIS. This is because when the monitored Server is the same Server as the monitoring Server we can run into trouble. The Server cannot send us an E-Mail notification when it is in an critical state and is unable to send E-Mails.

Next step is to configure E-mail Notifications for Critical State. Follow the same steps that we went through earlier for Warning state notification except for the fact that this time we will select Critical from the drop down.

Once done, close all the property pages and dialog boxes. We are done with the setup.

Now when one or more of the configured resources run into a “Critical” or “Warning” state we receive a E-Mail from an account “WMI@SERVERNAME” with a error message with the server name in the subject line and the condition of the resource / service in the E-Mail body.

Technorati Profile

The free tools that I mention come with Microsoft’s Support Tool Kit and while writing this post I’ve ran the tools on Windows 2003 R2 Server. Download the Support Tool Kit. Once downloaded, install the Support Tools on the server.

Now before we move on to the health monitoring tools, it is always a good idea to make a list of all the Domain Controllers in your organization along with their IP address and trust relationships if any. Assuming that you have completed this step, let’s move on to using the tools.

Since many of these tests generate logs, let’s start by creating a log folder to store the diagnostic information. I’m going to use C:\AD-Health-Logs on my server.

Let’s start by verifying the health of the Domain. For this we are going to use NSLOOKUP which is a command line tool for testing and troubleshooting DNS. We all know that DNS is the heart of Active Directory and so that’s the first thing that we will check.

For replication to happen seamlessly between multiple Domain Controllers, name resolution should work fine. Each DC should be able to resolve the _msdcs record for other DC. Open the DNS mmc and look for the _msdcs record.

Once you have the _msdcs record for the DC object, we can use NSLOOKUP to check the name resolution. Open Command Prompt and type:

NSLOOKUP <GUID._msdcs.DnsForestName> replace the GUID string in <> brackets with your actual GUID.

If the NSLOOKUP resolves the correct IP address of the replication partner than DNS is working fine. Do check that the DNS server returned in the NSLOOKUP query is returning the correct records for the replication partner.

Since DNS is working fine now, let’s move on to checking the Replication functions and topology. To verify this, we are going to use REPLMON. This tool is included in the Windows Support Tools kit and offers a GUI based interface to check a low-level status of AD replication, force replication or even monitor the status of replication. You should find REPLMON utility from the Support Tools group under Administrative Tools.

Once the GUI is opened, right-click on the Monitored Servers and click on Add Monitored Server.

This will invoke the Add Monitored Server wizard. Here either you can search the AD for the server that you want to monitor or you can add it explicitly by name. Since I’ve already documented my list of Domain Controllers and their corresponding IP address before I started this exercise, I’ll add the server explicitly by name. Once the server is added, click on Action, navigate to Server and choose Generate Status Report.

When prompted for the location where we should store the log file, choose the folder that you created earlier. In my case, I’m saving this to C:\AD-Health-Logs. Choose all the reporting options on the next screen and click OK.  Once the report has been written to the Log file, you’ll see a message that the Report is complete. You can now navigate to the folder where you saved the log and open it in Notepad, Word, etc. to read the contents. The report basically gives you lot of in-depth in to the replication status and topology. Play with the other options available in the REPLMON tool like Show Domain Controllers in Domain, Show Global Catalog Servers in Enterprise, etc.

My REPLMON log files did not show any problems with replication and so we will now move forward to perform the DC health check. Open Command Prompt once again and type the following command:

DCDIAG /s: <ServerName> /v /c > C:\AD-Health_logs\<servername-DCDIAG-DD-MM-YY.log>

Replace ServerName with the actual DC name and the log path with the actual values. Using the /v switch will provide us a verbose result with extended information while the /c switch will provide us with a comprehensive report by running all tests. If you get any errors you can also use the DCDIAG /fix option here to perform some soft fixes. Run this for all the DC’s in your organization by replacing the ServerName and the log file prefix. You can run this from the same machine and does not require you to log individually to machines that you are checking.

Moving ahead, we will check the network connectivity health using the NETDIAG tool. The NETDIAG tool helps in isolating network connectivity problems by performing a series of tests. Open the Command Prompt once again and type NETDIAG /v > C:\AD-Health_logs\<servername-NETDIAG-DD-MM-YY.log>.

Again replace the appropriate variables with the actual values and wait for the diagnostics to complete. Open the log file once the process has completed and remediate any errors that you get. Note that you may see errors related to WINS in this. If you are not using WINS in your organization these can be safely ignored.

That’s it to the basic checks for monitoring the health of your Active Directory. I’ve not covered NetDom in this series since most of the functionality can be accessed from the Active Directory mmc’s that are available in the Administrative Tools.