Application Log Monitoring

In the multi-server (multi-cluster) environment on-premise or on cloud, there is always requirement to monitor it for any issues encountered or about to come.

In a SAAS environment, we never wanted any downtime because of any issue with the application or with the server environment. We need to have some monitoring system in-place to monitor the health of application as well as servers on which it is hosted. Monitoring system will not only help in pro-active prevention of any upcoming issue/s or doing troubleshooting when application faces issue/s, it also helps us in monitoring the security of the application. For example, if some unauthorized person tries to get hold of the application, application will write this access request with details like person ID, IP address and time of access to the log. One can then investigate it and pro-actively acts on it.

Logs generated by application can be written in the eventlog or anywhere in the application specific log file.

There are many ways to monitor the logs. It not only requires the capturing of log data from eventlog or any other source, it also requires strong analytics tool to produce sensible information from the log files. Note that we can have Gigs/TBs of log data. Manually, it will be very difficult to analyze it.

Here in this article, I have discussed about two approaches –

1. Automating it using PowerShell.
2. Using industry standard tools like SPLUNK.

    

Let’s discuss both approaches here –

Automation using PowerShell –

Application is writing logs in the eventlog. Approach will be to write a PowerShell script that will read the eventlog for any error or warning messages regularly and will then either take some action like restarting a service or just send the email with log as attachment to the application administrator.

Here is the script that can be used for the same –

 

In the above PowerShell, I have used “Get-WinEvent” cmdlet (Gets events from event logs and event tracing log files on local and remote computers.). I added filters to get the “error” and “warning” data for the last 48 hours. It is taking the below actions –

1. It is analyzing each row to find some specific keywords and based on it is taking some action. In the above script it is restarting a windows service.
2. It is exporting the data into csv file and sending it as attachment to the application administrator using “Send-MailMessage” cmdlet.

 

We can now add this PowerShell script to the Windows scheduler to run it regularly. Above script will run as a job and you can get its status using the cmdlet ‘Get-Job’.

Automation using tools like SPLUNK –

Suppose we have n number of servers (say 15) in cluster on which we have deployed the application. Application will be running on all these servers using load balancer. While application is running on these servers, it or web server may encounter some issues. How to troubleshoot these issues efficiently? We may use PowerShell way as discussed above but it needs complex script to visualize data out of tons of logs generated. Tools like SPLUNK provides automated ways to collect the log data from all the server in real time and can be queried and visualized using its analytics.

Splunk have many components that needs to be setup for making it work. One of the important component is “forwarder” which needs to be installed on all the servers. This part can easily be done using SCCM based deployment. It has the indexer that will index the data for efficient querying of the data.

Now suppose an event has been triggered in 3 of the servers in cluster and application/system administrators are not aware of. Splunk will get the logs and analyze the collected logs for different keywords using the defined queries. It will then send the alert to administrators using email. With it you will be able to proactively address any upcoming issues.

One more good example is, suppose the InfoSec (Information security) team has mandated the requirement that only special type of user accounts should be added to the servers. Suppose someone added user account which is not supposed to be added in the server. As soon as it is added, system will add a log in the log about the addition of user in the administrator group. Splunk will collect that log and the predefined query will run and immediately it will find the non-compliance and will send the email notification to the Infosec team.

—–End of Article—-

How to Find a Hostname for an IP Address

Sometimes we need to find the name of server using its IP address. Below are the steps that can help us finding the same:

How to Find a Hostname for an IP Address:

Step 1: Open the "Command Prompt" as Administrator.

Step 2: Type "nbtstat -A %ipaddress%" at the command prompt, substituting the IP address for "%ipaddress%."

Example: nbtstat –A 125.125.126.127

Step 3: Review the results and find the NETBIOS table. Locate a row where the type is "UNIQUE" and find the hostname of the computer in the "Name" column for that row as shown below:

Getting Windows services information using VBScript

Many times, we need to check if particular service is running or not on some particular server. Then we need to log into that server to check it.

To simplify it, I wrote a VBScript that will get the information for you from that server without login to the server.

Important NOTE: To get the service information, you need to run this VBScript with the account that has admin rights on the server for which you need the service information.

VBScript will point to below screen: Enter server name and click Ok.

It will create Service_Info.txt at the root location from where you are running the script.

Format of output log file:

Service Name::- Application Experience Lookup Service

Description::- Processes application compatibility lookup requests for applications as they are launched.

Status::- Running

Startup Type ::- Auto

Log on as ::- LocalSystem

VBScript Content:

'##############################################################

'#

'# Language     VBscript

'# 

'# Author       Mohd Aslam

'# Date         28.06.2011

'# Description  This script does following actions:

'#              1) it asks user to input the name of server 

'#                 whose service info you want

'#              2) It then creates Service_Info.txt file in 

'#                 the directory of the script file.

'#

'#############################################################


Dim oFSO              'File Scripting Object

Dim sScriptLog        'output log filename

Dim oScriptLog        'Script output file object

Dim oWshShell         'Windows Scripting Shell object


sScriptLog =    ".\Service_Info.txt"


Set oWshShell = WScript.CreateObject("WScript.Shell")

Set oFSO = CreateObject("Scripting.FileSystemObject")


Set oScriptLog =  oFSO.CreateTextFile (sScriptLog,True)

oScriptLog.Close()

Set oScriptLog = oFSO.OpenTextFile(sScriptLog,8)


InputServerName = InputBox("Enter the name of server whose service information you want:","Service Information") 


strComputer = InputServerName

Set objWMIService = GetObject("winmgmts:" _

    & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")

Set colRunningServices = objWMIService.ExecQuery _

    ("Select * from Win32_Service")    

    

For Each objService in colRunningServices          

     oScriptLog.WriteLine "Service Name::- " & objService.DisplayName  & VbCrLf 

     oScriptLog.WriteLine "Description::- " & objService.Description  & VbCrLf 

     oScriptLog.WriteLine "Status::- " & objService.State & VbCrLf

     oScriptLog.WriteLine "Startup Type ::- " & objService.startmode & VbCrLf

     oScriptLog.WriteLine "Log on as ::- " & objService.startname & VbCrLf     

     oScriptLog.WriteLine "-------------------------------------------" & VbCrLf     

Next
                               –End of Article–
			

Blade Servers

Many times we came across this word “Blade Server”….

 What are they?

 A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. The blades are literally servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports.

 Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. According to a SearchWinSystems.com article on server technology, enterprises moving to blade servers can experience as much as an 85% reduction in cabling for blade installations over conventional 1U or tower servers. With so much less cabling, IT administrators can spend less time managing the infrastructure and more time ensuring high availability.

 Each blade typically comes with one or two local ATA or SCSI drives. For additional storage, blade servers can connect to a storage pool facilitated by a network-attached storage (NAS), Fiber Channel, or iSCSI storage-area network (SAN). The advantage of blade servers comes not only from the consolidation benefits of housing several servers in a single chassis, but also from the consolidation of associated resources (like storage and networking equipment) into a smaller architecture that can be managed through a single interface.

 A blade server is sometimes referred to as a high-density server and is typically used in a clustering of servers that are dedicated to a single task, such as:

 File sharing

Web page serving and caching

SSL encrypting of Web communication

The transcoding of Web page content for smaller displays Streaming audio and video content Like most clustering applications, blade servers can also be managed to include load balancing and failover capabilities.