Home     RSSRSS

Archives: Web Server

Linux vmstat command

April 4, 2019 by kiranbadi1991 | Comments Off on Linux vmstat command | Filed in Database, Development, Environment, Memory, Performance Engineering, Process, Web Server

I have been spending  bit of my time on EC2 Amazon linux. So thought of just making a note of some of the commands I frequently use.

It helps me to look directly at my site for information on this command rather than googling and spending time for the information in the internet for this command.(All I need is what each column stands for)

vmstat gives information about processes, memory, paging, block I/O, traps, and CPU activity. It displays either average data or actual samples. The sampling mode can be enabled by providing vmstat with a sampling frequency and a sampling duration.

vmstat

The columns in the output are as follows:

Process (procs)  r: The number of processes waiting for runtime

                             b: The number of processes in uninterruptable sleep

Memory      swpd: The amount of virtual memory used (KB)

                     free: The amount of idle memory (KB)

                     buff: The amount of memory used as buffers (KB)

                     cache: The amount of memory used as cache (KB)

Swap                   si: Amount of memory swapped from the disk (KBps)

                            so: Amount of memory swapped to the disk (KBps)

IO                         bi: Blocks sent to a block device (blocks/s)

                             bo: Blocks received from a block device (blocks/s)

System                in: The number of interrupts per second, including the clock

                             cs: The number of context switches per second

CPU (% of total CPU time)

                            us: Time spent running non-kernel code (user time, including nice time).
                             sy: Time spent running kernel code (system time).
                             id: Time spent idle. Prior to Linux 2.5.41, this included I/O-wait time.
                            wa: Time spent waiting for IO.

Some additional flags for vmstat are

-m   -  displays the memory utilization of the kernel (slabs)
-a    – provides information about active and inactive memory pages
-n   – displays only one header line, useful if running vmstat in sampling mode and piping the output to a file. (eg.root#vmstat –n 2 10 generates vmstat 10 times with a sampling rate of two seconds.)
          When used with the –p {partition} flag, vmstat also provides I/O statistics

Tags: , ,

Oops Error and Java Error/Exception Handling

April 4, 2014 by kiranbadi1991 | Comments Off on Oops Error and Java Error/Exception Handling | Filed in Development, Environment, Performance Engineering, Project Management, Web Server

Very Often during the testing for java based application, I come across the generic Oops error message which looks something like below screenshot,

image

Lot many developers who are building the java based application  or are new to development use these techniques to display error or exception message. It’s a good technique and even I have this technique in my application which I have coded.The code to display Oops error which I have in my application is something like below,

<%@page contentType="text/html" pageEncoding="UTF-8"%>
<!DOCTYPE html>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@page isErrorPage="true" %>
<html>
    <head>
        <title>Show Error Page</title>
    </head>
    <body>
        <h1>Oops...</h1>
        <table width="100%" border="1">
            <tr valign="top">
                <td width="40%"><b>Error:</b></td>
                <td>${pageContext.exception}</td>
            </tr>
            <tr valign="top">
                <td><b>URI:</b></td>
                <td>${pageContext.errorData.requestURI}</td>
            </tr>
            <tr valign="top">
                <td><b>Status code:</b></td>
                <td>${pageContext.errorData.statusCode}</td>
            </tr>
            <tr valign="top">
                <td><b>Stack trace:</b></td>
                <td>
                    <c:forEach var="trace" 
                               items="${pageContext.exception.stackTrace}">
                        <p>${trace}</p>
                    </c:forEach>
                </td>
            </tr>
        </table>
    </body>
</html>

The code above uses the JSTL tag and EL to display the error page.It works perfectly fine and it does display most of the exception which is thrown by the application.

However drawback of this approach  is that it shows error or exception directly to the end user and it means that we are leaking information to the outside world about our code base. Its not good. Some of the reasons as why its not good can be found here.

Most of the sites display catchy error screens when something goes wrong with site. They look so good and in fact  appeal the users. Great number of examples  can be found here.

Now changing from that Generic Oops page to some thing catchy and stylish is also very easy. It involves creating one static page with funny image in it and steps for the user to go to other section of the site or report the error message to site administrator or something which you want your users do when they see error page. However to implement this change some knowledge as how java handles error/exception is required. Below links are probably worth the read  to understand the science of java errors or exception,

http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html

http://www.artima.com/intv/solidP.html

http://www.onjava.com/pub/a/onjava/2003/11/19/exceptions.html?page=2

http://docs.oracle.com/javase/6/docs/technotes/guides/logging/overview.html

Now changing from Generic Oops page to some catchy page is also quite easy. Build the static JSP or HTML page and edit the web xml error page section. It looks something like below in Netbean editor,sorry I use netbean for development so I have screenshot based on it. Web xml can also edited in notepad or any other IDE.Editing via IDE is recommended since its heart of the application.

image

We can also write the exception handler servlet and build the page dynamically using the throwable class. The important thing to be considered here is that we are not leaking information and at the same time we are also informing users that something went wrong and they can follow some other path to their work. Once you have implemented the change, we can immediately test this change by triggering the error condition to see that in fact we are redirecting to error page correctly.

If you are QA  and if you see Oops Page, then probably you want to log the defect for this irrespective of the condition/data as what triggered the error.However you also need to make sure that exception stack trace is correctly logged in the logs and it has all the information that is required to debug the condition. If you are using any logging framework like Log4j ,it should not take more time to redirect the error to appender and verify the completeness of the message.

Tags: , ,

Automated way of logging IIS Logs to MSSQL Database using ODBC

March 15, 2014 by kiranbadi1991 | Comments Off on Automated way of logging IIS Logs to MSSQL Database using ODBC | Filed in Development, Environment, Others, Web Server

Sometimes for few of the internal applications it helps to automate the application logging process to the database so that operation’s team can measure and benchmark the operation efficiencies of the systems.So in order to achieve these objectives, it helps to log the information directly to the database and use the facilities offered by database. So in this post I will share some of my experience in automating these processes and few of the risks elements we need to take into consideration..

The first step of this entire process is to create the database table which will hold your IIS log information.IIS 6,IIS 7 and IIS 7.5 in fact provides you the script to create the table. The sql script for creating table is located in below path,

%windir%\system32\inetsrv\logtemp.sql

However if you are on IIS 8 or IIS 8.5, then probably you might not find it in that location. So you can use the below script and create the table,

USE [IISLOGS]
GO

/****** Object:  Table [dbo].[InternetLog]    Script Date: 3/16/2014 1:23:42 PM ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[InternetLog](
    [ClientHost] [varchar](255) NULL,
    [username] [varchar](255) NULL,
    [LogTime] [datetime] NULL,
    [service] [varchar](255) NULL,
    [machine] [varchar](255) NULL,
    [serverip] [varchar](50) NULL,
    [processingtime] [int] NULL,
    [bytesrecvd] [int] NULL,
    [bytessent] [int] NULL,
    [servicestatus] [int] NULL,
    [win32status] [int] NULL,
    [operation] [varchar](255) NULL,
    [target] [varchar](255) NULL,
    [parameters] [varchar](255) NULL
) ON [PRIMARY]

GO

SET ANSI_PADDING OFF
GO

For each of the table columns, Microsoft has below explanation, its always safe to go with Microsoft suggestions since having incorrect column types means that at some point we might see data truncation issues.

FieldName: ClientHost
Data Source/Type: Varchar(255)
Explanations: Client IP address.

FieldName: Username
Data Source/Type: Varchar(255)
Explanations: User name for the client. If the page is not password-protected, this is always the anonymous user name.

FieldName: LogTime
Data Source/Type: Datetime
Explanations: Date and time that the log entry was created.

FieldName: Service
Data Source/Type: Varchar(255)
Explanations: Name of the service. This can be WWW, FTP, or some other name.

FieldName: Machine
Data Source/Type: Varchar(255)
Explanations: Server name.

FieldName: ServerIP
Data Source/Type: Varchar(255)
Explanations: Server IP address.

FieldName: ProcessingTime
Data Source/Type: Int
Explanations: Time spent on request processing (in milliseconds).

FieldName: BytesRecvd
Data Source/Type: Int
Explanations: Number of bytes received.

FieldName: BytesSent
Data Source/Type: Int
Explanations: Number of bytes sent.

FieldName: ServiceStatus
Data Source/Type: Int
Explanations: Service status, such as 200.

FieldName: Win32Status
Data Source/Type: Long Integer
Explanations: Windows NT status code. 0 typically indicates success.

FieldName: Operation
Data Source/Type: Varchar(255)
Explanations: Type of the operation or command. For example, this may be USER for FTP or GET for WWW.

FieldName: Target
Data Source/Type: Varchar(255)
Explanations: Target of the operation. For example, this may be Default.htm.

FieldName: Parameters
Data Source/Type: Varchar(255)
Explanations: Any parameters for the operation. This can be either name/value pairs for invoking CGI or an ISAPI extension. It is a user name for the FTP command USER. 

Once the table is created, the next step is to configure ODBC System DSN on the machine which hosts the IIS Server.If you are 32 Bit systems , then you need to Configure 32 bit System ODBC and for 64 bit systems you need to enable 64 Bit System ODBC.Below steps should help you to configure ODBC DSN,

  1. On the IIS Server, open Control Panel, double-click the ODBC data source, click the System DSN tab, and then click Add.When the Create New Data Source window appears, click to select SQL Server, and then click Finish
  2. image[8]

  3. Click Add and in the Name box, type HTTPLOG(Name of DSN), type a description, click to select the SQL server that you want to connect to, and then click Next. If the SQL server is on the same computer, select (local).

image

    3. In the creation wizard, make sure that you click to select With Windows NT  authentication using the network login ID for the computer that is running SQL Server. Examine the client configuration, and use the default Named Pipe setting. Make sure that the SQL server name is correct, and then click OK.Click Next.

image

4. Map the IIS Logs database to the database where the original created table resides, and then click Next.If you want to, you can click to select Save long running queries to the log file and Log ODBC driver statistics to the log file in the wizard.Saving these queries will help you to debug any issues you find in operational or during set up phase.Else its not required to save queries.

Click Finish.At the end of the wizard, click Test Data Source. Make sure that you have successfully connected to the computer that is running SQL Server, and then click OK to exit. You need to ensure that Data Source test is successful.

image

With the above steps , you have now successfully done the setup for System DSN for logging IIS Logs into SQL Database.Microsoft provides you the DSN to log information to My SQL , Oracle and MS SQL Database. So if your database is different than these ,like Postgre or Derby then you need to install the relevant drivers for the same before they can be used.

Now the final step is  configuring the IIS to use the System DSN and start logging the application logs into the database table which we created earlier,On IIS 7, below steps can be followed to configure IIS to do ODBC logging,

  1. In the ISM MMC, right-click the Web site, and then click Properties.
  2. Click the Web Site tab.You can use configure the logging based on per site basis.
  3. In the Active log format list, click to select ODBC Logging. You can ignore the user name and password on the ODBC Logging Properties page if you selected Windows NT Integrated authentication when you set up the system DSN that is mapped to the computer that is running SQL Server.
  4. Click Apply, and then click OK.NOTE: If an account is specified on the ODBC Logging Properties page, the Username field in the SQL Server table is blank or contains a dash (-). If a domain account is used, the account name appears in the SQL Server logging table.
  5. Stop and Restart the Website.

Now IIS will direct all logging information to the table.However there are some risks for using this approach and they definitely needs to be taken into consideration,

  1. If your IIS Site is very busy , then logging using ODBC might consume lot of system resources which might impact the performance of the server.
  2. Its recommended that database holding the logging information should never be shared with live application else there exists substantial risks that database performance also might get impacted and slow down the application.

Few of the situation where I feel we can use these types of automation process is when your application is internal in nature and does not deal with very heavy loads.

For all heavy load application, I would recommend that we log IIS logs from the file to the database using  this approach as we can also automate that process on daily, nightly or weekly basis to load logs into the database.

 

Tags: , , , ,